將各種字段列表的JSON響應轉換爲Pandas數據框

我正在使用python的Requests庫從BestBuy Products API下載一些數據，並且我想將它們存儲到pandas數據框中。將各種字段列表的JSON響應轉換爲Pandas數據框

類似的東西：

results = requests.get(url1, 
        params={'paramStuff'}, 
        headers={'User-Agent': ua}) 
products = json.loads(results.text)

一個得到了很多的服務信息各領域的，因此我只瞄準，我想在JSON特定領域：

products['products']

我：

[{'details':[{'name': 'Name of Feature', 'value':'Value Of Feature'}, 
      {'name': 'Name of Other Feature', 'value':'Value Of Other 
       Feature'}, ...], 
    'ProductId': 'Id Of Product 1', 
    'Some Other Field': 'Some Other Field Value'}, 
{same structure as above for other product}, {etc}]

因此，當你看到它就像是一個字典列表，其中又包含詞典列表自己。要突出顯示 - 細節詞典可以有各種名稱組合的名稱：值（名稱在產品中也不同）。

對如何處理這樣的結構，以獲得與這種格式的數據幀的任何想法：

+-----------+-------------------+-------------------+-------------------+------------------+ 
| ProductID | Name of Feature 1 | Name of Feature 2 | Name Of Feature 3 | Some Other Field | 
+-----------+-------------------+-------------------+-------------------+------------------+ 
| Product 1 | Value    | NULL    | Value    | Value   | 
| Product 2 | NULL    | Value    | Value    | Value   | 
+-----------+-------------------+-------------------+-------------------+------------------+

到目前爲止，我只設法得到這樣的：

+-----------+-----------------------------------------------------------------------------------------------------------------------------------+------------------+ 
| ProductID |                Details                | Some Other Field | 
+-----------+-----------------------------------------------------------------------------------------------------------------------------------+------------------+ 
| Product 1 | [{'name': 'Name of Feature', 'value':'Value Of Feature'},{'name': 'Name of Other Feature', 'value':'Value Of Other Feature'},...] | Value 1   | 
| Product 2 | [{'name': 'Name of Feature', 'value':'Value Of Feature'},{'name': 'Name of Other Feature', 'value':'Value Of Other Feature'},...] | Value 2   | 
+-----------+-----------------------------------------------------------------------------------------------------------------------------------+------------------+

來源

2017-06-21 Maxim Haytovich

您是否嘗試過使用'pandas.read_json'？ – jhamman

@jhamman是的，可悲的是它無法處理沉重的嵌套。我最終通過編寫手動解析器來解決這個問題，如果沒有更好的方法，我將在這裏作爲答案張貼 –

好吧，我結束開發一種手動解析嵌套字段的方法。沒有弄清楚是否有任何簡單的方法。僅供參考，它用於解析BestBuy Products API的響應，以防有人發現它有用。

#first build the pandas DF shown in question 
df = pd.io.json.json_normalize(products) 

#fields which are not nested and not require parsing 
fields = ['sku', 'name', 'regularPrice', 'manufacturer'] 
#nested field is called 'details', as mentioned can have a lot of different subfields 
featureFields = [] 


#first build a list which will have all potential features from the nested field 
for i in range(0,len(df)): 
    row = df.iloc[i] 
    for detail in row['details']: 
     featureFields.append(detail['name'].split('>', 1)[-1]) 

#make a list unique 
featureFields = set(featureFields)  
fields = set(fields) 

#now we go over each record in dataframe and parse nested field to a dict 
records = [] 

for i in range(0,len(df)): 
    row = df.iloc[i] 
    record = dict.fromkeys(fields) 
    record['name'] = row['name'] 
    record['regularPrice'] = row['regularPrice'] 
    record['manufacturer'] = row['manufacturer'] 
    record['sku'] = row['sku'] 
    for detail in row['details']: 
     record[detail['name'].split('>', 1)[-1]] = detail['value'].split('>', 1)[-1] 
    records.append(record) 

#finally we have not nested list of dictionaries with records 
dfFinal = pd.DataFrame.from_dict(records)

來源

2017-06-22 02:18:10

將各種字段列表的JSON響應轉換爲Pandas數據框

回答

相關問題