Spark 2.0.2在rdds /嵌套rdds或數據框或數據集中嵌套K-means

我試圖並行運行大量k-means。我有一個房間和大量的數據，我想計算每個房間的集羣。所以，我有Spark 2.0.2在rdds /嵌套rdds或數據框或數據集中嵌套K-means

roomsSignals[(room:String, signals:List[org.apache.spark.mllib.linalg.Vector]] 

roomsSignals.map{l=> 
val data=sc.parallelize(l.signals) 
val clusterCenters=2 
val model = KMeans.train(data, clusterCenters, 5) 
    model.clusterCenters.map { r =>r.toJson.toString}.mkString(",") 

}.collect.foreach(println)

這給我的錯誤：

org.apache.spark.SparkException: Job aborted due to stage failure: Task 33 in stage 18.0 failed 4 times, most recent failure: Lost task 33.3 in stage 18.0 (TID 1284, 192.168.181.122):  java.lang.NullPointerException 
at $anonfun$1.apply(<console>:77) 
at $anonfun$1.apply(<console>:76) 
at scala.collection.Iter ator$$anon$11.next(Iter ator.scala:409) 
at scala.collection.Iter ator$class.foreach(Iter ator.scala:893) 
at scala.collection.AbstractIter ator.foreach(Iter ator.scala:1336) 
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59) 
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104) 
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48) 
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310) 
at scala.collection.AbstractIter ator.to(Iter ator.scala:1336) 
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302) 
at scala.collection.AbstractIter ator.toBuffer(Iter ator.scala:1336) 
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289) 
at scala.collection.AbstractIter ator.toArray(Iter ator.scala:1336) 
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:912) 
at org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$13.apply(RDD.scala:912) 
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1916) 
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1916) 
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70) 
at org.apache.spark.scheduler.Task.run(Task.scala:86) 
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274) 
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745) 
Driver stacktrace: 
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1454) 
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1442) 
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1441) 
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)

來源

2016-12-13 Dima

請粘貼完整的堆棧跟蹤幷包含驅動程序堆棧跟蹤！ –

我貼了一些它不能把所有因爲限制 – Dima

然後只是看看自己的錯誤是哪裏！它會在驅動程序的堆棧軌跡中爲您提供確切的線路 –

遺憾的是不可能的。 Spark根本不支持任何類型的嵌套。

任一列車獨立分佈式模型遍歷roomsSignals.collect或使用本地庫選擇在分佈式結構中構建模型。

來源

2016-12-13 17:04:30

我知道它在spark 1.3中是不可能的，我會得到不可串行化的spark上下文的異常，於是我認爲spark已經進步了。 – Dima

它沒有，它不會。如果不明顯限制Spark功能，則無法執行此類任何操作。 –

Spark 2.0.2在rdds /嵌套rdds或數據框或數據集中嵌套K-means

回答

相關問題