PyTables讀取隨機子集

Question

是否可以從HDF5讀取隨機子集（通過pyTables或最好是熊貓）？我有一個包含數百萬行的非常大的數據集，但只需要幾千個樣本進行分析。那麼讀取壓縮的HDF文件呢？

Answer 1

使用HDFStore文檔是here，壓縮文檔是通過構造指數here

隨機訪問以0.13

In [26]: df = DataFrame(np.random.randn(100,2),columns=['A','B']) 

In [27]: df.to_hdf('test.h5','df',mode='w',format='table') 

In [28]: store = pd.HDFStore('test.h5') 

In [29]: nrows = store.get_storer('df').nrows 

In [30]: nrows 
Out[30]: 100 

In [32]: r = np.random.randint(0,nrows,size=10) 

In [33]: r 
Out[33]: array([69, 28, 8, 2, 14, 51, 92, 25, 82, 64]) 

In [34]: pd.read_hdf('test.h5','df',where=pd.Index(r)) 
Out[34]: 
      A   B 
69 -0.370739 -0.325433 
28 0.155775 0.961421 
8 0.101041 -0.047499 
2 0.204417 0.470805 
14 0.599348 1.174012 
51 0.634044 -0.769770 
92 0.240077 -0.154110 
25 0.367211 -1.027087 
82 -0.698825 -0.084713 
64 -1.029897 -0.796999 

[10 rows x 2 columns] 

支持要包括一個附加條件，你會做這樣的：

# make sure that we have indexable columns 
df.to_hdf('test.h5','df',mode='w',format='table',data_columns=True) 

# select where the index (an integer index) matches r and A > 0 
In [14]: r 
Out[14]: array([33, 51, 33, 95, 69, 21, 43, 58, 58, 58]) 

In [13]: pd.read_hdf('test.h5','df',where='index=r & A>0') 
Out[13]: 
      A   B 
21 1.456244 0.173443 
43 0.174464 -0.444029 

[2 rows x 2 columns]