Python PANDAS df.duplicated和df.drop_duplicated找不到所有重複項

Question

我無法使用duplicateated或drop_duplicates從數據框中查找/刪除所有重複項。

我的數據看起來像這樣，但是，我的數據是52k行長。

data = {'inventory number':['WL-SMART-INWALL', 
         'WL-NMDISH-22', 
         'WL-MPS546-MESH', 
         'WAS-WG-500P', 
         'UKS/99757/69975', 
         'UKS/99757/69975', 
         'UKS/99750/S26361F2293L10', 
         'UKS/99750/S26361F2293L10', 
         'UKS/99733/69973', 
         'UKS/99733/69973', 
         'UKS/99727/AHD6502TU3CBK', 
         'UKS/99727/AHD6502TU3CBK', 
         'UKS/99725/PMK01', 
         'UKS/99725/PMK01', 
         'UKS/99294/A3L791R15MS', 
         'UKS/99294/A3L791R15MS', 
         'UKS/98865/58018251', 
         'UKS/98865/58018251', 
         'UKS/98509/90Q653AN1N0N2UA0', 
         'UKS/98509/90Q653AN1N0N2UA0', 
         'UKS/97771/FIBLCSC2', 
         'UKS/97771/FIBLCSC2', 
         'UKS/97627/FIBLCLC1', 
         'UKS/97627/FIBLCLC1'], 
     'minimum price': ['36.85', 
         '55.45', 
         '361.29', 
         '265.0', 
         '22.46', 
         '22.46', 
         '15.0', 
         '15.0', 
         '26.71', 
         '26.71', 
         '104.0', 
         '104.0', 
         '32.3', 
         '32.3', 
         '22.51', 
         '22.51', 
         '13.0', 
         '13.0', 
         '9.59', 
         '9.59', 
         '15.0', 
         '15.0', 
         '15.0', 
         '15.0'], 
    'cost':['26.11', 
         '39.23', 
         '254.99', 
         '187.09', 
         '16.0', 
         '16.0', 
         '10.7', 
         '10.7', 
         '19.0', 
         '19.0', 
         '73.46', 
         '73.46', 
         '23.0', 
         '23.0', 
         '16.0', 
         '16.0', 
         '9.29', 
         '9.29', 
         '7.0', 
         '7.0', 
         '10.7', 
         '10.7', 
         '10.7', 
         '10.7'] 
    } 
df = pd.DataFrame(data=data) 

我的最後幾周目錄追加到本週的底部產生的數據幀我。我只想做一些與已更改的庫存號有關的事情，或者我想要三角洲。我想我可以追加這兩個，確保他們是相同的數據類型，重新索引和刪除重複項，但是當我寫入CSV到QA時，仍有數千個重複項。

這裏是我的代碼：

_import['inventory number'] = _import['inventory number'].str.encode('utf-8') 
ts_data['inventory number'] = ts_data['inventory number'].str.encode('utf-8') 
overlap = overlap.append(ts_data, ignore_index=True) 
overlap_dedupe = overlap[overlap.duplicated(['inventory number','minimum price','cost'],keep=False)==False] 

我也試過：

overlap_dedupe = overlap.drop_duplicates(keep=False) 

所以，我知道我遇到了某種編碼的問題，因爲現在我沒有得到重複。

combined.head(50).duplicated() 

回報：

42736 False 
32567 False 
43033 False 
33212 False 
46592 False 
46023 False 
32568 False 
33520 False 
32756 False 
26741 False 
46133 False 
42737 False 
42480 False 
40227 False 
40562 False 
49623 False 
27712 False 
31848 False 
49794 False 
27296 False 
38198 False 
35674 False 
27907 False 
22210 False 
40563 False 
18025 False 
49624 False 
18138 False 
19357 False 
43698 False 
24398 False 
50566 False 
22276 False 
38382 False 
20507 False 
43550 False 
18150 False 
29968 False 
19247 False 
47706 False 
19248 False 
43955 False 
20731 False 
38199 False 
44168 False 
17580 False 
15944 False 
44891 False 
28327 False 
16027 False 
dtype: bool 

Answer 1

drop_duplicates

---

duplicated

---

這些是姐妹功能，可以很好地協同工作。

使用df

df = pd.read_json(
    ''.join(
     ['[[26.11,"WL-SMART-INWALL",36.85],[39.23,"WL-NMDISH-22",55.45', 
     '],[73.46,"UKS\\/99727\\/AHD6502TU3CBK",104.0],[73.46,"UKS\\/997', 
     '27\\/AHD6502TU3CBK",104.0],[23.0,"UKS\\/99725\\/PMK01",32.3],[2', 
     '3.0,"UKS\\/99725\\/PMK01",32.3],[16.0,"UKS\\/99294\\/A3L791R15MS', 
     '",22.51],[16.0,"UKS\\/99294\\/A3L791R15MS",22.51],[9.29,"UKS\\/', 
     '98865\\/58018251",13.0],[9.29,"UKS\\/98865\\/58018251",13.0],[7', 
     '.0,"UKS\\/98509\\/90Q653AN1N0N2UA0",9.59],[7.0,"UKS\\/98509\\/90', 
     'Q653AN1N0N2UA0",9.59],[254.99,"WL-MPS546-MESH",361.29],[10.7', 
     ',"UKS\\/97771\\/FIBLCSC2",15.0],[10.7,"UKS\\/97771\\/FIBLCSC2",1', 
     '5.0],[10.7,"UKS\\/97627\\/FIBLCLC1",15.0],[10.7,"UKS\\/97627\\/F', 
     'IBLCLC1",15.0],[187.09,"WAS-WG-500P",265.0],[16.0,"UKS\\/9975', 
     '7\\/69975",22.46],[16.0,"UKS\\/99757\\/69975",22.46],[10.7,"UKS', 
     '\\/99750\\/S26361F2293L10",15.0],[10.7,"UKS\\/99750\\/S26361F229', 
     '3L10",15.0],[19.0,"UKS\\/99733\\/69973",26.71],[19.0,"UKS\\/997', 
     '33\\/69973",26.71]]'] 
    ) 
) 

我們可以清楚地看到，有與

df.duplicated() 

0  False 
1  False 
2  False 
3  True 
4  False 
5  True 
6  False 
7  True 
8  False 
9  True 
10 False 
11  True 
12 False 
13 False 
14  True 
15 False 
16  True 
17 False 
18 False 
19  True 
20 False 
21  True 
22 False 
23  True 
dtype: bool 

重複因爲我們沒有通過keep參數，我們採用默認值是keep='first'。這意味着該系列中的每個True表示與其上面的另一行的副本相同的行，其狀態爲False。

我們可以剪短這和剛剛回來的是否不存在重複與

df.duplicated().any() 

True 

答案，我們可以驗證drop_duplicates通過鏈接我們方便的試驗重複做任何事情援引drop_duplicates

後

df.drop_duplicates().duplicated().any() 

False 

太棒了！有效。
這可以用

df = df.drop_duplicates() 
df 

     0       1  2 
0 26.11    WL-SMART-INWALL 36.85 
1 39.23    WL-NMDISH-22 55.45 
2 73.46  UKS/99727/AHD6502TU3CBK 104.00 
4 23.00    UKS/99725/PMK01 32.30 
6 16.00  UKS/99294/A3L791R15MS 22.51 
8  9.29   UKS/98865/58018251 13.00 
10 7.00 UKS/98509/90Q653AN1N0N2UA0 9.59 
12 254.99    WL-MPS546-MESH 361.29 
13 10.70   UKS/97771/FIBLCSC2 15.00 
15 10.70   UKS/97627/FIBLCLC1 15.00 
17 187.09     WAS-WG-500P 265.00 
18 16.00    UKS/99757/69975 22.46 
20 10.70 UKS/99750/S26361F2293L10 15.00 
22 19.00    UKS/99733/69973 26.71 

保留只是爲了確保

df.duplicated().any() 

False 

---

結論
它的工作對我罰款。希望這個演示能夠幫助你清除任何對你來說不對的問題。