與分類配合和partial_fit

Question

內存錯誤，我嘗試使用scikit學習，用於對輸入文本string.I預測值現在用HashingVectorizer數據量化和PassiveAggressiveClassifier使用partial_fit學習（參閱下面的代碼）：

from sklearn.feature_extraction.text import TfidfVectorizer 
import numpy as np 
from sklearn.naive_bayes import GaussianNB 
from sklearn.multiclass import OneVsRestClassifier 
from sklearn.svm import LinearSVC 
from sklearn import metrics 
from sklearn.metrics import zero_one_loss 
from sklearn.tree import DecisionTreeClassifier 
from sklearn.svm import SVC 
from sklearn.neighbors import KNeighborsClassifier 
from sklearn.linear_model import PassiveAggressiveClassifier, SGDClassifier, Perceptron 
from sklearn.pipeline import make_pipeline 
from sklearn.externals import joblib 
import pickle 

a,r = [],[] 

vectorizer = TfidfVectorizer() 

with open('val', 'rb') as f: 
    r = pickle.load(f) 

with open('text', 'rb') as f: 
    a = pickle.load(f) 

L = (vectorizer.fit_transform(a)) 

training_set = L[:3250] 
testing_set = L[3250:] 

M = np.array(r) 

training_result = M[:3250] 
testing_result = M[3250:] 

cls = np.unique(r) 

model = PassiveAggressiveClassifier() 

model.partial_fit(training_set, training_result, classes=cls) 
print(model) 
predicted = model.predict(testing_set) 

print testing_result 
print predicted 

錯誤日誌：

File "try.py", line 89, in <module> 
    model.partial_fit(training_set, training_result, classes=cls) 
    File "/usr/local/lib/python2.7/dist-packages/sklearn/linear_model/passive_aggressive.py", line 115, in partial_fit 
    coef_init=None, intercept_init=None) 
    File "/usr/local/lib/python2.7/dist-packages/sklearn/linear_model/stochastic_gradient.py", line 374, in _partial_fit 
    coef_init, intercept_init) 
    File "/usr/local/lib/python2.7/dist-packages/sklearn/linear_model/stochastic_gradient.py", line 167, in _allocate_parameter_mem 
    dtype=np.float64, order="C") 
MemoryError 

我以前使用CountVectorizer和邏輯迴歸分類和工作沒有問題。 但我的學習數據是約。數百萬行，我想執行增量學習使用上述腳本，這是導致內存錯誤每次執行。

UPDATE：

在環施加局部學習之後，partial_fit函數返回不匹配的號碼特徵的錯誤（ValueError: Number of features 8897 does not match previous data 9190.） 另外，即使我設置最大特徵屬性，那麼所產生的預測是不正確的。 有沒有什麼方法可以使partial_fit方法獲得可變數量的特徵？

執行輸出：

(400, 8481) 
(400, 9277) 
Traceback (most recent call last): 
    File "f9.py", line 65, in <module> 
    training_set, training_result, classes=cls) 
    File "/usr/local/lib/python2.7/dist-packages/sklearn/linear_model/passive_aggressive.py", line 115, in partial_fit 
    coef_init=None, intercept_init=None) 
    File "/usr/local/lib/python2.7/dist-packages/sklearn/linear_model/stochastic_gradient.py", line 379, in _partial_fit 
    % (n_features, self.coef_.shape[-1])) 
ValueError: Number of features 9277 does not match previous data 8481. 

任何幫助將不勝感激。

謝謝！

Answer 1

內存錯誤來自內存中數據太多的事實。當您加載數據時，您的數量等於N，那麼當您partial_fit時，根據算法的不同，它會存儲更多的數據，可能接近於N.

您不需要存儲兩次數據。儘量減少你的初始數據塊的大小。把它分成幾個部分，你會給partial_fit方法。

你應該閱讀你的行文件中的行來創建數據的塊，然後貼合該塊，並刷新內存，然後再

with open(path, "r", encoding='utf-8') as f: 
    i = 0 
    for line in f: 
     % Create chunk of X line 
     i ++ 
     arr.add(line) 

     % Learn with partial_fit 
     if (i == X): 
      model.partial_fit() 
      % Flush the last chunk 
      arr = []