当前位置：首页 > news >正文

Pandas JSON学习

news 2024/11/2 4:29:25

1.JSON简介

JSON（JavaScript Object Notation，JavaScript 对象表示法），是存储和交换文本信息的语法，类似 XML。JSON 比 XML 更小、更快，更易解析，Pandas 可以很方便的处理 JSON 数据。

[{"id": "A001","name": "百度","url": "www.baidu.com","likes": 61},{"id": "A002","name": "Google","url": "www.google.com","likes": 124},{"id": "A003","name": "淘宝","url": "www.taobao.com","likes": 45}
]

可以直接用to_string()处理 JSON 字符串。

import pandas as pddf = pd.read_json('sites.json')print(df.to_string())

import pandas as pddata =[{"id": "A001","name": "百度","url": "www.baidu.com","likes": 61},{"id": "A002","name": "Google","url": "www.google.com","likes": 124},{"id": "A003","name": "淘宝","url": "www.taobao.com","likes": 45}
]
df = pd.DataFrame(data)print(df)

2.可以直接将 Python 字典转化为 DataFrame 数据

JSON 对象与 Python 字典具有相同的格式。

import pandas as pd# 字典格式的 JSON                                                                                             
s = {"col1":{"row1":1,"row2":2,"row3":3},"col2":{"row1":"x","row2":"y","row3":"z"}
}# 读取 JSON 转为 DataFrame                                                                                          
df = pd.DataFrame(s)print(df)

3.假设有一组内嵌的 JSON 数据文件 nested_list.json

{"school_name": "ABC primary school","class": "Year 1","students": [{"id": "A001","name": "Tom","math": 60,"physics": 66,"chemistry": 61},{"id": "A002","name": "James","math": 89,"physics": 76,"chemistry": 51},{"id": "A003","name": "Jenny","math": 79,"physics": 90,"chemistry": 78}]
}

import pandas as pddf = pd.read_json('nested_list.json')print(df)

4.使用 json_normalize() 方法将内嵌的数据完整解析

import pandas as pd
import json# 使用 Python JSON 模块载入数据
with open('nested_list.json','r') as f:data = json.loads(f.read())# 展平数据
df_nested_list = pd.json_normalize(data, record_path =['students'])
print(df_nested_list)

data = json.loads(f.read()) 使用 Python JSON 模块载入数据，json_normalize() 使用了参数 record_path 并设置为 ['students'] 用于展开内嵌的 JSON 数据 students。

5.使用 meta 参数显示元数据

import pandas as pd
import json# 使用 Python JSON 模块载入数据
with open('nested_list.json','r') as f:data = json.loads(f.read())# 展平数据
df_nested_list = pd.json_normalize(data,record_path =['students'],meta=['school_name', 'class']
)
print(df_nested_list)

6.假设数据文件 nested_mix.json嵌套了列表和字典

{"school_name": "local primary school","class": "Year 1","info": {"president": "John Kasich","address": "ABC road, London, UK","contacts": {"email": "admin@e.com","tel": "123456789"}},"students": [{"id": "A001","name": "Tom","math": 60,"physics": 66,"chemistry": 61},{"id": "A002","name": "James","math": 89,"physics": 76,"chemistry": 51},{"id": "A003","name": "Jenny","math": 79,"physics": 90,"chemistry": 78}]
}

7.文件转换为 DataFrame

import pandas as pd
import json# 使用 Python JSON 模块载入数据
with open('nested_mix.json','r') as f:data = json.loads(f.read())df = pd.json_normalize(data,record_path =['students'],meta=['class',['info', 'president'],['info', 'contacts', 'tel']]
)print(df)

8.假设存在nested_deep.json文件

{"school_name": "local primary school","class": "Year 1","students": [{"id": "A001","name": "Tom","grade": {"math": 60,"physics": 66,"chemistry": 61}},{"id": "A002","name": "James","grade": {"math": 89,"physics": 76,"chemistry": 51}     },{"id": "A003","name": "Jenny","grade": {"math": 79,"physics": 90,"chemistry": 78}}]
}

9.使用glom 模块来处理数据套嵌

glom 模块允许使用 . 来访问内嵌对象的属性。第一次使用需要安装 glom。

!pip install glom

import pandas as pd
from glom import glomdf = pd.read_json('nested_deep.json')data = df['students'].apply(lambda row: glom(row, 'grade.math'))
print(data)