Spark 2.1.0 - SparkML要求失敗

我正在玩Spark 2.1.0 Kmeans - 聚類算法。Spark 2.1.0 - SparkML要求失敗

public class ClusteringTest { 
    public static void main(String[] args) { 
     SparkSession session = SparkSession.builder() 
       .appName("Clustering Test") 
       .config("spark.master", "local") 
       .getOrCreate(); 
     session.sparkContext().setLogLevel("ERROR"); 

     List<Row> rawDataTraining = Arrays.asList(
       RowFactory.create(1.0,Vectors.dense(1.0, 1.0, 1.0).toSparse()), 
       RowFactory.create(1.0,Vectors.dense(2.0, 2.0, 2.0).toSparse()), 
       RowFactory.create(1.0,Vectors.dense(3.0, 3.0, 3.0).toSparse()), 

       RowFactory.create(2.0,Vectors.dense(6.0, 6.0, 6.0).toSparse()), 
       RowFactory.create(2.0,Vectors.dense(7.0, 7.0, 7.0).toSparse()), 
       RowFactory.create(2.0,Vectors.dense(8.0, 8.0,8.0).toSparse()), 
//... 
     StructType schema = new StructType(new StructField[]{ 

       new StructField("label", DataTypes.DoubleType, false, Metadata.empty()), 
       new StructField("features", new VectorUDT(), false, Metadata.empty()) 
     }); 

     Dataset<Row> myRawData = session.createDataFrame(rawDataTraining, schema); 
     Dataset<Row>[] splits = myRawData.randomSplit(new double[]{0.75, 0.25}); 
     Dataset<Row> trainingData = splits[0]; 
     Dataset<Row> testData = splits[1]; 

     //Train Kmeans 
     KMeans kMeans = new KMeans().setK(3).setSeed(100); 
     KMeansModel kMeansModel = kMeans.fit(trainingData); 
     Dataset<Row> predictions = kMeansModel.transform(testData); 
     predictions.show(false); 
     MulticlassClassificationEvaluator evaluator = new MulticlassClassificationEvaluator().setLabelCol("label") 
       .setPredictionCol("prediction") 
       .setMetricName("accuracy"); 
     double accuracy = evaluator.evaluate(predictions); 
     System.out.println("accuracy" + accuracy); 
    } 
}

控制檯輸出有：

+-----+----------------------------+----------+ 
|label|features     |prediction| 
+-----+----------------------------+----------+ 
|2.0 |(3,[0,1,2],[7.0,7.0,7.0]) |2   | 
|3.0 |(3,[0,1,2],[11.0,11.0,11.0])|2   | 
|3.0 |(3,[0,1,2],[12.0,12.0,12.0])|1   | 
|3.0 |(3,[0,1,2],[13.0,13.0,13.0])|1   | 
+-----+----------------------------+----------+ 

Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: Column prediction must be of type DoubleType but was actually IntegerType. 
    at scala.Predef$.require(Predef.scala:233) 
    at org.apache.spark.ml.util.SchemaUtils$.checkColumnType(SchemaUtils.scala:42) 
    at org.apache.spark.ml.evaluation.MulticlassClassificationEvaluator.evaluate(MulticlassClassificationEvaluator.scala:75) 
    at ClusteringTest.main(ClusteringTest.java:84) 

Process finished with exit code 1

如u可以看到，預測結果是整數。但要使用MulticlassClassificationEvalutor，我需要將這些預測結果轉換爲Double。我該怎麼做？

來源

2017-07-29 Java Padawan

TL; DR這不是要走的路。

KMeans是無監督方法，您獲得的羣集標識符是任意的（羣集的ID可以置換）並且與label列無關。因此使用MulticlassClassificationEvaluator來比較現有標籤和KMeans的輸出沒有任何意義。

您應該使用一些有監督的分類器，如多項邏輯迴歸或樸素貝葉斯。

如果您想堅持使用KMeans，請使用適當的質量指標，如computeCost所返回的質量指標，但這會完全忽略標籤信息。

來源

2017-07-29 18:16:39 user8371915

Spark 2.1.0 - SparkML要求失敗

回答

相關問題