如何在scikit-learn中計算correclty交叉驗證分數？

Question

我正在做一個分類任務。不過，我得到的結果稍有不同：

#First Approach 
kf = KFold(n=len(y), n_folds=10, shuffle=True, random_state=False) 
pipe= make_pipeline(SVC()) 
for train_index, test_index in kf: 
    X_train, X_test = X[train_index], X[test_index] 
    y_train, y_test = y[train_index], y[test_index] 

print ('Precision',np.mean(cross_val_score(pipe, X_train, y_train, scoring='precision'))) 



#Second Approach 
clf.fit(X_train,y_train) 
y_pred = clf.predict(X_test) 
print ('Precision:', precision_score(y_test, y_pred,average='binary')) 

#Third approach 
pipe= make_pipeline(SCV()) 
print('Precision',np.mean(cross_val_score(pipe, X, y, cv=kf, scoring='precision'))) 

#Fourth approach 

pipe= make_pipeline(SVC()) 
print('Precision',np.mean(cross_val_score(pipe, X_train, y_train, cv=kf, scoring='precision'))) 

日期：

Precision: 0.780422106837 
Precision: 0.782051282051 
Precision: 0.801544091998 

/usr/local/lib/python3.5/site-packages/sklearn/cross_validation.py in cross_val_score(estimator, X, y, scoring, cv, n_jobs, verbose, fit_params, pre_dispatch) 
    1431            train, test, verbose, None, 
    1432            fit_params) 
-> 1433      for train, test in cv) 
    1434  return np.array(scores)[:, 0] 
    1435 

/usr/local/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py in __call__(self, iterable) 
    798    # was dispatched. In particular this covers the edge 
    799    # case of Parallel used with an exhausted iterator. 
--> 800    while self.dispatch_one_batch(iterator): 
    801     self._iterating = True 
    802    else: 

/usr/local/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py in dispatch_one_batch(self, iterator) 
    656     return False 
    657    else: 
--> 658     self._dispatch(tasks) 
    659     return True 
    660 

/usr/local/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py in _dispatch(self, batch) 
    564 
    565   if self._pool is None: 
--> 566    job = ImmediateComputeBatch(batch) 
    567    self._jobs.append(job) 
    568    self.n_dispatched_batches += 1 

/usr/local/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py in __init__(self, batch) 
    178   # Don't delay the application, to avoid keeping the input 
    179   # arguments in memory 
--> 180   self.results = batch() 
    181 
    182  def get(self): 

/usr/local/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py in __call__(self) 
    70 
    71  def __call__(self): 
---> 72   return [func(*args, **kwargs) for func, args, kwargs in self.items] 
    73 
    74  def __len__(self): 

/usr/local/lib/python3.5/site-packages/sklearn/externals/joblib/parallel.py in <listcomp>(.0) 
    70 
    71  def __call__(self): 
---> 72   return [func(*args, **kwargs) for func, args, kwargs in self.items] 
    73 
    74  def __len__(self): 

/usr/local/lib/python3.5/site-packages/sklearn/cross_validation.py in _fit_and_score(estimator, X, y, scorer, train, test, verbose, parameters, fit_params, return_train_score, return_parameters, error_score) 
    1522  start_time = time.time() 
    1523 
-> 1524  X_train, y_train = _safe_split(estimator, X, y, train) 
    1525  X_test, y_test = _safe_split(estimator, X, y, test, train) 
    1526 

/usr/local/lib/python3.5/site-packages/sklearn/cross_validation.py in _safe_split(estimator, X, y, indices, train_indices) 
    1589     X_subset = X[np.ix_(indices, train_indices)] 
    1590   else: 
-> 1591    X_subset = safe_indexing(X, indices) 
    1592 
    1593  if y is not None: 

/usr/local/lib/python3.5/site-packages/sklearn/utils/__init__.py in safe_indexing(X, indices) 
    161         indices.dtype.kind == 'i'): 
    162    # This is often substantially faster than X[indices] 
--> 163    return X.take(indices, axis=0) 
    164   else: 
    165    return X[indices] 

IndexError: index 900 is out of bounds for size 900 

所以，我的問題是上述方法是正確的計算cross validated metrics？我相信我的分數是被污染的，因爲我對什麼時候執行交叉驗證感到困惑。因此，關於如何正確執行交叉驗證分數的任何想法？

UPDATE

在訓練步驟評估？

X_train, X_test, y_train, y_test = train_test_split(X, y, random_state = False) 
clf = make_pipeline(SVC()) 
# However, fot clf, you can use whatever estimator you like 
kf = StratifiedKFold(y = y_train, n_folds=10, shuffle=True, random_state=False) 
scores = cross_val_score(clf, X_train, y_train, cv = kf, scoring='precision') 
print('Mean score : ', np.mean(scores)) 
print('Score variance : ', np.var(scores)) 

Answer 1

對於任何分類任務，它總是很好用StratifiedKFold交叉驗證拆分。在分層KFold中，您的分類問題中每個類的樣本數量相同。

那就要看你的類型分類問題。它總是很高興看到精確度和召回分數。在傾斜的二元分類的情況下，人們傾向於使用ROC AUC分數：

from sklearn import metrics 
metrics.roc_auc_score(ytest, ypred) 

讓我們看一下你的解決方案：

import numpy as np 
from sklearn.cross_validation import cross_val_score 
from sklearn.metrics import precision_score 
from sklearn.cross_validation import KFold 
from sklearn.pipeline import make_pipeline 
from sklearn.svm import SVC 

np.random.seed(1337) 

X = np.random.rand(1000,5) 

y = np.random.randint(0,2,1000) 

kf = KFold(n=len(y), n_folds=10, shuffle=True, random_state=42) 
pipe= make_pipeline(SVC(random_state=42)) 
for train_index, test_index in kf: 
    X_train, X_test = X[train_index], X[test_index] 
    y_train, y_test = y[train_index], y[test_index] 

print ('Precision',np.mean(cross_val_score(pipe, X_train, y_train, scoring='precision'))) 
# Here you are evaluating precision score on X_train. 

#Second Approach 
clf = SVC(random_state=42) 
clf.fit(X_train,y_train) 
y_pred = clf.predict(X_test) 
print ('Precision:', precision_score(y_test, y_pred, average='binary')) 

# here you are evaluating precision score on X_test 

#Third approach 
pipe= make_pipeline(SVC()) 
print('Precision',np.mean(cross_val_score(pipe, X, y, cv=kf, scoring='precision'))) 

# Here you are splitting the data again and evaluating mean on each fold 

因此，結果是不同的

Answer 2

首先，解釋在documentation中顯示，並在examples中顯示，scikit-learn交叉驗證cross_val_score執行以下操作：

1. 將您的數據集X拆分N倍（根據參數cv）。它相應地拆分標籤y。
2. 使用估計器（參數estimator）在N-1個以前的摺疊上進行訓練。
3. 使用估計器來預測最後一個摺疊的標籤。
4. 通過比較預測值和真值，返回得分（參數scoring）
5. 通過更改測試倍數重複步驟2至步驟4。因此，你最終得到一組N分數。

讓我們來看看您的每種方法。

第一種方法：

你爲什麼會分裂訓練​​cross_validation之前作爲scikit學習功能，它會爲你設置？因此，你在訓練較少的數據模型，具有值得驗證結束比分

第二種方法：

在這裏，你用你的數據的另一個指標比cross_validation_sore。因此，您無法將其與其他驗證分數進行比較 - 因爲它們是兩個不同的東西。一個是經典的錯誤百分比，而precision是用於校準二進制分類器的指標（true或false）。儘管這是一個很好的指標（您可以檢查ROC曲線，精確度和召回指標），但只比較這些指標。

第三種方法：

這一個是更自然的。這個分數是好的一（我的意思是如果你想比較它與其他分類器/估計）。不過，我會警告你直接採取平均的結果。因爲有兩件事可以比較：平均值和方差。陣列的每個得分是與其他不同，你可能想知道多少，比其他估計（你一定希望你的方差越小越好）

第四種方法：

似乎有要與不相關的cross_val_score

最後Kfold一個問題：

只能使用第二OR比較估計量的第三種方法。但他們絕對不會估計同樣的事情 - 精度與錯誤率。

clf = make_pipeline(SVC()) 
# However, fot clf, you can use whatever estimator you like 
scores = cross_val_score(clf, X, y, cv = 10, scoring='precision') 
print('Mean score : ', np.mean(scores)) 
print('Score variance : ', np.var(scores)) 

通過改變clf到另一個估計（或將其集成到一個循環），您將能夠爲每個eastimator分數，並比較他們