優化大熊貓GROUPBY許多小團體

Question

我有一個數據幀的大熊貓有許多小團體：

In [84]: n=10000 

In [85]: df=pd.DataFrame({'group':sorted(range(n)*4),'val':np.random.randint(6,size=4*n)}).sort(['group','val']).reset_index(drop=True) 

In [86]: df.head(9) 
Out[86]: 
    group val 
0  0 0 
1  0 0 
2  0 1 
3  0 2 
4  1 1 
5  1 2 
6  1 2 
7  1 4 
8  2 0 

我想要做的團體一些特別的東西其中val == 1次出現但不VAL == 0。例如。僅當val == 0在該組中時，纔將組1中的1替換爲99。

但是這個尺寸呢DataFrames是相當緩慢：

In [87]: def f(s): 
    ....: if (0 not in s) and (1 in s): s[s==1]=99 
    ....: return s 
    ....: 

In [88]: %timeit df.groupby('group')['val'].transform(f) 
1 loops, best of 3: 11.2 s per loop 

通過數據幀循環更噁心，但要快得多：

In [89]: %paste 

def g(df): 
    df.sort(['group','val'],inplace=True) 
    last_g=-1 
    for i in xrange(len(df)): 
     if df.group.iloc[i]!=last_g: 
      has_zero=False 
     if df.val.iloc[i]==0: 
      has_zero=True 
     elif has_zero and df.val.iloc[i]==1: 
      df.val.iloc[i]=99 
    return df 
## -- End pasted text -- 

In [90]: %timeit g(df) 
1 loops, best of 3: 2.53 s per loop 

但我想如果進一步優化它可能。

任何想法如何做？

感謝

---

基於傑夫的回答，我得到了一個解決方案，這是非常快的。我把它在這裏，如果別人覺得有用：

In [122]: def do_fast(df): 
    .....: has_zero_mask=df.group.isin(df[df.val==0].group.unique()) 
    .....: df.val[(df.val==1) & has_zero_mask]=99 
    .....: return df 
    .....: 

In [123]: %timeit do_fast(df) 
100 loops, best of 3: 11.2 ms per loop 

Answer 1

不是100％肯定這是你要的東西，但應該是簡單的有一個不同的過濾/設置標準

In [37]: pd.set_option('max_rows',10) 

In [38]: np.random.seed(1234) 

In [39]: def f(): 

      # create the frame 
      df=pd.DataFrame({'group':sorted(range(n)*4), 
           'val':np.random.randint(6,size=4*n)}).sort(['group','val']).reset_index(drop=True) 


      df['result'] = np.nan 

      # Create a count per group 
      df['counter'] = df.groupby('group').cumcount() 

      # select which values you want, returning the indexes of those 
      mask = df[df.val==1].groupby('group').grouper.group_info[0] 

      # set em 
      df.loc[df.index.isin(mask) & df['counter'] == 1,'result'] = 99 


In [40]: %timeit f() 
10 loops, best of 3: 95 ms per loop 

In [41]: df 
Out[41]: 
     group val result counter 
0   0 3  NaN  0 
1   0 4  99  1 
2   0 4  NaN  2 
3   0 5  99  3 
4   1 0  NaN  0 
...  ... ...  ...  ... 
39995 9998 4  NaN  3 
39996 9999 0  NaN  0 
39997 9999 0  NaN  1 
39998 9999 2  NaN  2 
39999 9999 3  NaN  3 

[40000 rows x 4 columns]