如何使熊貓數據框str.contains搜索更快

Question

我正在搜索400萬行數據框中的子字符串或多個子字符串。

df[df.col.str.contains('Donald',case=True,na=False)] 

或

df[df.col.str.contains('Donald|Trump|Dump',case=True,na=False)] 

數據幀（DF）看起來像以下（400萬個行）

df = pd.DataFrame({'col': ["very definition of the American success story, continually setting the standards of excellence in business, real estate and entertainment.", 
         "The myriad vulgarities of Donald Trump—examples of which are retailed daily on Web sites and front pages these days—are not news to those of us who have", 
         "While a fearful nation watched the terrorists attack again, striking the cafés of Paris and the conference rooms of San Bernardino"]}) 

是否有任何提示，使這個字符串搜索速度更快？ 例如，首先排序數據框，以某種方式建立索引，將列名更改爲數字，從查詢中刪除「na = False」等。即使毫秒級的速度增加也會非常有幫助！

Answer 1

如果子串的數量很少，則一次搜索一個子查詢可能會更快，因爲您可以將regex=False參數傳遞給contains，從而加快速度。

在我用兩個樣本子串測試的大約6000行的示例DataFrame上，blah.contains("foo", regex=False) | blah.contains("bar", regex=False)大約是blah.contains("foo|bar")的兩倍。你必須用你的數據來測試它，看它是如何擴展的。

Answer 2

您可以將其轉換爲列表。似乎在一個列表中搜索而不是將字符串方法應用於一個系列的速度要快得多。

樣品的編號：

import timeit 
df = pd.DataFrame({'col': ["very definition of the American success story, continually setting the standards of excellence in business, real estate and entertainment.", 
         "The myriad vulgarities of Donald Trump—examples of which are retailed daily on Web sites and front pages these days—are not news to those of us who have", 
         "While a fearful nation watched the terrorists attack again, striking the cafés of Paris and the conference rooms of San Bernardino"]}) 



def first_way(): 
    df["new"] = pd.Series(df["col"].str.contains('Donald',case=True,na=False)) 
    return None 
print "First_way: " 
%timeit for x in range(10): first_way() 
print df 

df = pd.DataFrame({'col': ["very definition of the American success story, continually setting the standards of excellence in business, real estate and entertainment.", 
         "The myriad vulgarities of Donald Trump—examples of which are retailed daily on Web sites and front pages these days—are not news to those of us who have", 
         "While a fearful nation watched the terrorists attack again, striking the cafés of Paris and the conference rooms of San Bernardino"]}) 


def second_way(): 
    listed = df["col"].tolist() 
    df["new"] = ["Donald" in n for n in listed] 
    return None 

print "Second way: " 
%timeit for x in range(10): second_way() 
print df 

結果：

First_way: 
100 loops, best of 3: 2.77 ms per loop 
               col new 
0 very definition of the American success story,... False 
1 The myriad vulgarities of Donald Trump—example... True 
2 While a fearful nation watched the terrorists ... False 
Second way: 
1000 loops, best of 3: 1.79 ms per loop 
               col new 
0 very definition of the American success story,... False 
1 The myriad vulgarities of Donald Trump—example... True 
2 While a fearful nation watched the terrorists ... False