Spark ML中的尺寸不匹配錯誤

我對ML和Spark ML都很新穎，我試圖用Spark ML創建一個使用神經網絡的預測模型，但是當我在我的電話上調用.transform方法時出現此錯誤學習模型。這個問題是由於使用OneHotEncoder引起的，因爲沒有它，一切正常。我已經嘗試將OneHotEncoder從管道中取出。Spark ML中的尺寸不匹配錯誤

我的問題是：如何使用OneHotEncoder並且不會出現此錯誤？

java.lang.IllegalArgumentException: requirement failed: A & B Dimension mismatch! 
at scala.Predef$.require(Predef.scala:224)  at org.apache.spark.ml.ann.BreezeUtil$.dgemm(BreezeUtil.scala:41) at 
org.apache.spark.ml.ann.AffineLayerModel.eval(Layer.scala:163)  at 
org.apache.spark.ml.ann.FeedForwardModel.forward(Layer.scala:482) at 
org.apache.spark.ml.ann.FeedForwardModel.predict(Layer.scala:529)

我的代碼：

test_pandas_df = pd.read_csv(
    '/home/piotrek/ml/adults/adult.test', names=header, skipinitialspace=True) 
train_pandas_df = pd.read_csv(
    '/home/piotrek/ml/adults/adult.data', names=header, skipinitialspace=True) 
train_df = sqlContext.createDataFrame(train_pandas_df) 
test_df = sqlContext.createDataFrame(test_pandas_df) 

joined = train_df.union(test_df) 

assembler = VectorAssembler().setInputCols(features).setOutputCol("features") 

label_indexer = StringIndexer().setInputCol(
    "label").setOutputCol("label_index") 

label_indexer_fit = [label_indexer.fit(joined)] 

string_indexers = [StringIndexer().setInputCol(
    name).setOutputCol(name + "_index").fit(joined) for name in categorical_feats] 

one_hot_pipeline = Pipeline().setStages([OneHotEncoder().setInputCol(
    name + '_index').setOutputCol(name + '_one_hot') for name in categorical_feats]) 

mlp = MultilayerPerceptronClassifier().setLabelCol(label_indexer.getOutputCol()).setFeaturesCol(
    assembler.getOutputCol()).setLayers([len(features), 20, 10, 2]).setSeed(42L).setBlockSize(1000).setMaxIter(500) 
pipeline = Pipeline().setStages(label_indexer_fit 
           + string_indexers + [one_hot_pipeline] + [assembler, mlp]) 

model = pipeline.fit(train_df) 

# compute accuracy on the test set 
result = model.transform(test_df) 

## FAILS ON RESULT 

predictionAndLabels = result.select("prediction", "label_index") 

evaluator = MulticlassClassificationEvaluator(labelCol="label_index") 
print "-------------------------------" 
print("Test set accuracy = " + str(evaluator.evaluate(predictionAndLabels))) 
print "-------------------------------"

謝謝！

來源

2017-02-17 piotrm50

layers在模型Param不正確：

setLayers([len(features), 20, 10, 2])

第一層應反映的輸入特徵，其一般不會是相同的原料列的編碼之前的數目的數目。

如果您不知道預先設置的功能的總數，例如可以分離特徵提取和模型訓練。僞代碼：

feature_pipeline_model = (Pipeline() 
    .setStages(...) # Only feature extraction 
    .fit(train_df)) 

train_df_features = feature_pipeline_model.transform(train_df) 
layers = [ 
    train_df_features.schema["features"].metadata["ml_attr"]["num_attrs"], 
    20, 10, 2 
]

來源

2017-02-17 15:32:08 user6910411

Spark ML中的尺寸不匹配錯誤

回答

相關問題