python熊貓與numpy數組的搜索性能

Question

我想在熊貓列中搜索一個字符串。我已經讀過，它應該是最快排序的第一列和搜索字符串使用搜索排序值。我發現這比在同一個numpy數組上搜索蠻力（array == string）要慢。要知道爲什麼，我已經進行了以下試驗：

import timeit 

setup4 = ''' 
import numpy as np, string, random 

d =  np.array([ 
      u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)) 
      for _ in range(20000) 
      ],dtype=np.object) 
''' 



setup5 = ''' 
import numpy as np, pandas as pd, string, random 

header = [ 
        u'A', 
        u'B', 
        u'C', 
        u'D', 
        u'E', 
        u'F', 
        u'G', 
        u'H', 
        u'I', 
        u'J', 
        u'K', 
        u'L', 
        u'M', 
        u'N' 
        ] 


data =  [[pd.to_datetime('20140505'), 
       u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)), 
       u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)), 
       u'sfgweorfjdfl', 
       u'dsiofqjwel;dmfv', 
       u'e3ruiwefjvgoiubg', 
       u'3124oirjrg;klhbas', 
       u';3rhfgfbnvsad3r', 
       pd.to_datetime('20140505'), 
       u'1234irtjurgbfas', 
       u'12;rhfd;hb;oasere', 
       u'124urgfdnv.,sadfg', 
       u'1rfnhsdjk.dhafgsrdew', 
       u'safeklrjh2nerfgsd.' 
       ] for _ in range(20000)] 

df = pd.DataFrame(data,columns=header) 
df_sorted = df.sort(['B','C']) 
e = df_sorted['B'].values 
''' 

setup6 = ''' 
import numpy as np, pandas as pd, string, random 

header = [ 
        u'A', 
        u'B', 
        u'C', 
        u'D', 
        u'E', 
        u'F', 
        u'G', 
        u'H', 
        u'I', 
        u'J', 
        u'K', 
        u'L', 
        u'M', 
        u'N' 
        ] 


data =  [[pd.to_datetime('20140505'), 
       u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)), 
       u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)), 
       u'sfgweorfjdfl', 
       u'dsiofqjwel;dmfv', 
       u'e3ruiwefjvgoiubg', 
       u'3124oirjrg;klhbas', 
       u';3rhfgfbnvsad3r', 
       pd.to_datetime('20140505'), 
       u'1234irtjurgbfas', 
       u'12;rhfd;hb;oasere', 
       u'124urgfdnv.,sadfg', 
       u'1rfnhsdjk.dhafgsrdew', 
       u'safeklrjh2nerfgsd.' 
       ] for _ in range(20000)] 

df = pd.DataFrame(data,columns=header) 
f = df['B'].values 
''' 

print(timeit.timeit("index = d == u'ASDASD123ASADKHX'", setup=setup4,number=1000)) 
print(timeit.timeit("index = e == u'ASDASD123ASADKHX'", setup=setup5,number=1000)) 
print(timeit.timeit("index = f == u'ASDASD123ASADKHX'", setup=setup6,number=1000)) 

結果如下：

print(timeit.timeit("index = d == u'ASDASD123ASADKHX'", setup=setup4,number=1000)) 
0.808505267014 

print(timeit.timeit("index = e == u'ASDASD123ASADKHX'", setup=setup5,number=1000)) 

3.06733738226 

print(timeit.timeit("index = f == u'ASDASD123ASADKHX'", setup=setup6,number=1000)) 
1.64207848896 

我的問題在這裏：爲什麼是純numpy的陣列上的表現好多了？我如何使用從熊貓表中提取的數據實現相同的性能？

非常感謝。

Answer 1

DataFrame中的每個字符串都是一個對象，您從df['B'].values得到的是對象數組。但是，當您通過np.array()創建字符串數組時，它會返回一個數組，每個字符串使用相同的字節數。

下面是一個示例，a是S10 dtype的數組，b是一個帶有對象dtype的數組。

import numpy as np 
import random 
import string 
words = ["".join(random.choice(string.ascii_uppercase) for _ in range(10)) for _ in range(10000)] 
a = np.array(words) 
b = a.astype("O") 
%timeit a == "123" 
%timeit b == "123" 

輸出：

10000 loops, best of 3: 122 µs per loop 
10000 loops, best of 3: 164 µs per loop 

Answer 2

我測試你的代碼在IPython中，並得到了幾乎所有相同的性能未排序的數據幀變種分開：

In [85]: 

import numpy as np, string, random 

d =  np.array([ 
      u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)) 
      for _ in range(20000) 
      ],dtype=np.object) 

header = [ 
        u'A', 
        u'B', 
        u'C', 
        u'D', 
        u'E', 
        u'F', 
        u'G', 
        u'H', 
        u'I', 
        u'J', 
        u'K', 
        u'L', 
        u'M', 
        u'N' 
        ] 


data =  [[pd.to_datetime('20140505'), 
       u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)), 
       u''.join(random.choice(string.ascii_uppercase + string.digits) for _ in range(16)), 
       u'sfgweorfjdfl', 
       u'dsiofqjwel;dmfv', 
       u'e3ruiwefjvgoiubg', 
       u'3124oirjrg;klhbas', 
       u';3rhfgfbnvsad3r', 
       pd.to_datetime('20140505'), 
       u'1234irtjurgbfas', 
       u'12;rhfd;hb;oasere', 
       u'124urgfdnv.,sadfg', 
       u'1rfnhsdjk.dhafgsrdew', 
       u'safeklrjh2nerfgsd.' 
       ] for _ in range(20000)] 

df = pd.DataFrame(data,columns=header) 
df_sorted = df.sort(['B','C']) 
e = df_sorted['B'].values 
f = df['B'].values 
%timeit index = d == u'ASDASD123ASADKHX' 
%timeit index = e == u'ASDASD123ASADKHX' 
%timeit index = f == u'ASDASD123ASADKHX' 
1000 loops, best of 3: 536 µs per loop 
1000 loops, best of 3: 568 µs per loop 
1000 loops, best of 3: 538 µs per loop