scikit-learn SelectPercentile TFIDF數據特徵縮減

我在scikit-learn中使用各種機制來創建訓練數據集的tf-idf表示和由文本特徵組成的測試集。這兩個數據集都經過預處理以使用相同的詞彙表，因此特徵和特徵數量相同。我可以在訓練數據上創建一個模型並評估其在測試數據上的表現。我想知道是否使用SelectPercentile來減少轉換後訓練集中的特徵數量，怎樣才能確定測試集中的相同特徵以用於預測？scikit-learn SelectPercentile TFIDF數據特徵縮減

trainDenseData = trainTransformedData.toarray() 
testDenseData = testTransformedData.toarray() 

if (useFeatureReduction== True): 
    reducedTrainData = SelectPercentile(f_regression,percentile=10).fit_transform(trainDenseData,trainYarray) 

clf.fit(reducedTrainData, trainYarray) 


# apply feature reduction to the test data

來源

2015-04-01 Niall

查看下面的代碼和評論。

import numpy as np 

from sklearn.datasets import make_classification 
from sklearn import feature_selection 

# Build a classification task using 3 informative features 
X, y = make_classification(n_samples=1000, 
          n_features=10, 
          n_informative=3, 
          n_redundant=0, 
          n_repeated=0, 
          n_classes=2, 
          random_state=0, 
          shuffle=False) 

sp = feature_selection.SelectPercentile(feature_selection.f_regression, percentile=30) 
sp.fit_transform(X[:-1], y[:-1]) #here, training are the first 9 data vectors, and the last one is the test set 
idx = np.arange(0, X.shape[1]) #create an index array 
features_to_keep = idx[sp.get_support() == True] #get index positions of kept features 

x_fs = X[:,features_to_keep] #prune X data vectors 
x_test_fs = x_fs[-1] #take your last data vector (the test set) pruned values 
print x_test_fs #these are your pruned test set values

來源

2015-04-01 15:18:12 Ryan

你應該存儲SelectPercentile對象，並用它來transform測試數據：

select = SelectPercentile(f_regression,percentile=10) 
reducedTrainData = select.fit_transform(trainDenseData,trainYarray) 
reducedTestData = select.transform(testDenseData)

來源

2015-04-01 19:58:50

scikit-learn SelectPercentile TFIDF數據特徵縮減

回答

相關問題