getOrCreate部署失敗隨機

Question

當試圖調用H2OContext.getOrCreate具有有效SparkContext，隨機我們不斷看到失敗部署：

17/04/21 17:21:32 ERROR TaskSchedulerImpl: Lost executor 0 on 172.17.0.4: Remote RPC client disassociated. Likely due to containers exceeding thresholds, or network issues. Check driver logs for WARN messages. 
17/04/21 17:21:38 ERROR LiveListenerBus: Listener ExecutorAddNotSupportedListener threw an exception 
java.lang.IllegalArgumentException: Executor without H2O instance discovered, killing the cloud! 
    at org.apache.spark.listeners.ExecutorAddNotSupportedListener.onExecutorAdded(H2OSparkListener.scala:27) 
    at org.apache.spark.scheduler.SparkListenerBus$class.doPostEvent(SparkListenerBus.scala:61) 
    at org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36) 
    at org.apache.spark.scheduler.LiveListenerBus.doPostEvent(LiveListenerBus.scala:36) 
    at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:63) 
    at org.apache.spark.scheduler.LiveListenerBus.postToAll(LiveListenerBus.scala:36) 
    at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(LiveListenerBus.scala:94) 
    at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79) 
    at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(LiveListenerBus.scala:79) 
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:58) 
    at org.apache.spark.scheduler.LiveListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(LiveListenerBus.scala:78) 
    at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1252) 
    at org.apache.spark.scheduler.LiveListenerBus$$anon$1.run(LiveListenerBus.scala:77) 

的H2OContext.getOrCreate導致錯誤：從H2O

Context.spark_session = SparkSession.builder.getOrCreate() 
Context.h2o_context = H2OContext.getOrCreate(Context.spark_session) 

有什麼想法船員？

Answer 1

這是Sparkling Water內部後端在此刻已知的行爲。爲了避免這種情況，可以使用外部Sparkling Water後端。關於這方面的更多信息可以在這裏找到https://github.com/h2oai/sparkling-water/blob/master/doc/backends.md

我目前正在研究這個JIRA，它應該消除上面的行爲。它正在進行中，可以跟蹤JIRA https://0xdata.atlassian.net/browse/SW-369以獲取任務的狀態。