熊貓數據框 - 刪除異常值

Question

給定一個熊貓數據框，我想根據其中一列排除對應於異常值的行（Z值= 3）。

數據框看起來是這樣的：

df.dtypes 
_id     object 
_index    object 
_score    object 
_source.address  object 
_source.district  object 
_source.price  float64 
_source.roomCount float64 
_source.size   float64 
_type     object 
sort     object 
priceSquareMeter  float64 
dtype: object 

對於行：

dff=df[(np.abs(stats.zscore(df)) < 3).all(axis='_source.price')] 

以下異常引發：

-------------------------------------------------------------------------  
TypeError         Traceback (most recent call last) 
<ipython-input-68-02fb15620e33> in <module>() 
----> 1 dff=df[(np.abs(stats.zscore(df)) < 3).all(axis='_source.price')] 

/opt/anaconda3/lib/python3.6/site-packages/scipy/stats/stats.py in zscore(a, axis, ddof) 
    2239  """ 
    2240  a = np.asanyarray(a) 
-> 2241  mns = a.mean(axis=axis) 
    2242  sstd = a.std(axis=axis, ddof=ddof) 
    2243  if axis and mns.ndim < a.ndim: 

/opt/anaconda3/lib/python3.6/site-packages/numpy/core/_methods.py in _mean(a, axis, dtype, out, keepdims) 
    68    is_float16_result = True 
    69 
---> 70  ret = umr_sum(arr, axis, dtype, out, keepdims) 
    71  if isinstance(ret, mu.ndarray): 
    72   ret = um.true_divide(

TypeError: unsupported operand type(s) for +: 'NoneType' and 'NoneType' 

和

返回值10

是

True 

爲什麼我得到上面的例外，我怎麼能排除異常值？

Answer 1

使用此布爾，只要你有這類問題：

df=pd.DataFrame({'Data':np.random.normal(size=200)}) #example 
df[np.abs(df.Data-df.Data.mean())<=(3*df.Data.std())] #keep only the ones that are within +3 to -3 standard deviations in the column 'Data'. 
df[~(np.abs(df.Data-df.Data.mean())>(3*df.Data.std()))] #or the other way around 

Answer 2

我相信你可以創建一個異常值的布爾過濾器，然後選擇它的oposite。

outliers = stats.zscore(df['_source.price']).apply(lambda x: np.abs(x) == 3) 
df_without_outliers = df[~outliers]