從csv構建熊貓數據框，它包含多個字典列

Question

我有一個csv，其中包含填充了單個字典的多個列。有成千上萬的行。我想把這些命令拉出來，並將列從列表中移出，然後用它們的值填充單元格，填入缺少值的NaN。所以說：

id       attributes 
0 255RSSSTCHL-QLTDGLZD-BLK  {"color": "Black", "hardware": "Goldtone"} 
1 C3ACCRDNFLP-QLTDS-S-BLK  {"size": "Small", "color": "Black"} 

變爲：

id       size color hardware 
0 255RSSSTCHL-QLTDGLZD-BLK  NaN Black Goldtone 
1 C3ACCRDNFLP-QLTDS-S-BLK  Small Black NaN 

有幾列像「身份證」，我想在生成的數據幀，以保持原封不動，有幾列像「屬性」那充滿有我想吹入列的字典。爲了說明，我將它們截斷爲上面的例子。

Answer 1

來源DF：

In [172]: df 
Out[172]: 
         id        attributes      attr2 
0 255RSSSTCHL-QLTDGLZD-BLK {"color":"Black","hardware":"Goldtone"} {"aaa":"aaa", "bbb":"bbb"} 
1 C3ACCRDNFLP-QLTDS-S-BLK   {"size":"Small","color":"Black"}    {"ccc":"ccc"} 

解決方案1：

import ast 

attr_cols = ['attributes','attr2'] 

def f(df, attr_col): 
    return df.join(df.pop(attr_col) \ 
      .apply(lambda x: pd.Series(ast.literal_eval(x)))) 


for col in attr_cols: 
    df = f(df, col) 

解決方案2：由於@DYZ for the hint：

import json 

attr_cols = ['attributes','attr2'] 

def f(df, attr_col): 
    return df.join(df.pop(attr_col) \ 
      .apply(lambda x: pd.Series(json.loads(x)))) 

for col in attr_cols: 
    df = f(df, col) 

結果：

In [175]: df 
Out[175]: 
         id color hardware size aaa bbb ccc 
0 255RSSSTCHL-QLTDGLZD-BLK Black Goldtone NaN aaa bbb NaN 
1 C3ACCRDNFLP-QLTDS-S-BLK Black  NaN Small NaN NaN ccc 

時間：爲20.000行DF：

In [198]: df = pd.concat([df] * 10**4, ignore_index=True) 

In [199]: df.shape 
Out[199]: (20000, 3) 

In [201]: %paste 
def f_ast(df, attr_col): 
    return df.join(df.pop(attr_col) \ 
      .apply(lambda x: pd.Series(ast.literal_eval(x)))) 

def f_json(df, attr_col): 
    return df.join(df.pop(attr_col) \ 
      .apply(lambda x: pd.Series(json.loads(x)))) 
## -- End pasted text -- 

In [202]: %%timeit 
    ...: for col in attr_cols: 
    ...:  f_ast(df.copy(), col) 
    ...: 
1 loop, best of 3: 33.1 s per loop 

In [203]: 

In [203]: %%timeit 
    ...: for col in attr_cols: 
    ...:  f_json(df.copy(), col) 
    ...: 
1 loop, best of 3: 30 s per loop 

In [204]: df.shape 
Out[204]: (20000, 3) 

Answer 2

您可以嵌入使用converters選項

import pandas as pd 
from io import StringIO 
from cytoolz.dicttoolz import merge as dmerge 
from json import loads 

txt = """id|attributes|attr2 
255RSSSTCHL-QLTDGLZD-BLK|{"color":"Black","hardware":"Goldtone"}|{"aaa":"aaa", "bbb":"bbb"} 
C3ACCRDNFLP-QLTDS-S-BLK|{"size":"Small","color":"Black"}|{"ccc":"ccc"}""" 

converters = dict(attributes=loads, attr2=loads) 

df = pd.read_csv(StringIO(txt), sep='|', index_col='id', converters=converters) 
df 

在pd.read_csv調用字符串解析

然後我們可以merge橫跨每一行的字典並轉換爲pd.DataFrame。我將使用上面導入的cytoolz.dicttoolz.merge作爲dmerge。

pd.DataFrame(df.apply(dmerge, 1).values.tolist(), df.index).reset_index() 

         id aaa bbb ccc color hardware size 
0 255RSSSTCHL-QLTDGLZD-BLK aaa bbb NaN Black Goldtone NaN 
1 C3ACCRDNFLP-QLTDS-S-BLK NaN NaN ccc Black  NaN Small