大熊貓檢查平等太慢使用

Question

我有一個檢查從一個DataFrame更改爲另一個的記錄的要求。它必須匹配全部列。

一個是excel文件（new_df），一個是SQL查詢（sql_df）。形狀是〜20,000行×39列。我認爲這將是df.equals(other_df)

目前我使用下面的工作：

import pandas as pd 
import numpy as np 
new_df = pd.DataFrame({'ID' : [0 ,1, 2, 3, 4, 5, 6, 7, 8, 9], 
        'B' : [1,0,3,5,0,0,np.NaN,9,0,0], 
        'C' : [10,0,30,50,0,0,4,10,1,3], 
        'D' : [1,0,3,4,0,0,7,8,0,1], 
        'E' : ['Universtiy of New York','New Hampshire University','JMU','Oklahoma State','Penn State', 
          'New Mexico Univ','Rutgers','Indiana State','JMU','University of South Carolina']}) 

sql_df= pd.DataFrame({'ID' : [0 ,1, 2, 3, 4, 5, 6, 7, 8, 9], 
        'B' : [1,0,3,5,0,0,np.NaN,9,0,0], 
        'C' : [10,0,30,50,0,0,4,10,1,0], 
        'D' : [5,0,3,4,0,0,7,8,0,1], 
        'E' : ['Universtiy of New York','New Hampshire University','NYU','Oklahoma State','Penn State', 
          'New Mexico Univ','Rutgers','Indiana State','NYU','University of South Carolina']}) 

# creates an empty list to append to 
differences = [] 
# for all the IDs in the dataframe that should not change check if this record is the same in the database 
# must use reset_index() so the equals() will work as I expect it to 
# if it is not the same, append to a list which has the Aspn ID that is failing, along with the columns that changed 
for unique_id in new_df['ID'].tolist(): 
# get the id from the list, and filter both sql and new dfs to this record 
    if new_df.loc[new_df['ID'] == unique_id].reset_index(drop=True).equals(sql_df.loc[sql_df['ID'] == unique_id].reset_index(drop=True)) is False: 
     bad_columns = [] 
     for column in new_df.columns.tolist(): 
     # if not the same above, check which column using the same logic 
      if new_df.loc[new_df['ID'] == unique_id][column].reset_index(drop=True).equals(sql_df.loc[sql_df['ID'] == unique_id][column].reset_index(drop=True)) is False: 
       bad_columns.append(column)        
     differences.append([unique_id, bad_columns]) 

後來我拿differences和bad_columns和執行其它任務與他們。

有很多循環，我希望避免...因爲這可能是我的性能問題的原因。目前，20,000個記錄需要5分鐘以上（硬件上有所不同），這是糟糕的表現。我想加入/連接所有列成一個長字符串來比較，但這似乎是另一種低效率的方式。解決這個問題的更好方法是什麼？我怎樣才能避免將這種混亂添加到空列表解決方案中？

Answer 1

In [26]: new_df.ne(sql_df) 
Out[26]: 
     B  C  D  E  ID 
0 False False True False False 
1 False False False False False 
2 False False False True False 
3 False False False False False 
4 False False False False False 
5 False False False False False 
6 True False False False False 
7 False False False False False 
8 False False False True False 
9 False True False False False 

顯示相異列：

In [27]: new_df.ne(sql_df).any(axis=0) 
Out[27]: 
B  True 
C  True 
D  True 
E  True 
ID False 
dtype: bool 

顯示不同行數：

In [28]: new_df.ne(sql_df).any(axis=1) 
Out[28]: 
0  True 
1 False 
2  True 
3 False 
4 False 
5 False 
6  True 
7 False 
8  True 
9  True 
dtype: bool 

UPDATE：

表示異種細胞：

In [86]: x = new_df.ne(sql_df) 

In [87]: new_df[x].loc[x.any(1)] 
Out[87]: 
    B C D E ID 
0 NaN NaN 1.0 NaN NaN 
2 NaN NaN NaN JMU NaN 
6 NaN NaN NaN NaN NaN 
8 NaN NaN NaN JMU NaN 
9 NaN 3.0 NaN NaN NaN 

In [88]: sql_df[x].loc[x.any(1)] 
Out[88]: 
    B C D E ID 
0 NaN NaN 5.0 NaN NaN 
2 NaN NaN NaN NYU NaN 
6 NaN NaN NaN NaN NaN 
8 NaN NaN NaN NYU NaN 
9 NaN 0.0 NaN NaN NaN 

Answer 2

獲取過濾數據框中顯示有區別僅行：

result_df = new_df[new_df != sql_df].dropna(how='all') 

>>> result_df 
Out[]: 
    B C D E ID 
0 NaN NaN 1.0 NaN NaN 
2 NaN NaN NaN JMU NaN 
8 NaN NaN NaN JMU NaN 
9 NaN 3.0 NaN NaN NaN 

獲取的ID和列名其中是有區別的，這是你在哪裏試圖產生輸出元組。 即使您在同一個ID上有多個不同的列，也應該可以工作。

result_df.set_axis(labels=new_df.ID[result_df.index], axis=0) 

>>> result_df.apply(lambda x: (x.name, result_df.columns[x.notnull()]), axis=1) 
Out[]: 
ID 
0 (0, [D]) 
2 (2, [E]) 
8 (8, [E]) 
9 (9, [C]) 
dtype: object 

請注意：apply是接近for循環，所以第二部分可能會花費更多的時間比第一。