Spark Pi集羣模式下使用紗線的示例：丟失關聯

我有三臺作爲分佈式Spark集羣運行的虛擬機。我正在使用Spark 1.3.0和基礎Hadoop 2.6.0。Spark Pi集羣模式下使用紗線的示例：丟失關聯

如果我運行星火丕例如

/usr/local/spark130/bin/spark-submit 
--class org.apache.spark.examples.SparkPi 
--master yarn-client /usr/local/spark130/examples/target/spark-examples_2.10-1.3.0.jar 10000

我得到這樣的警告/錯誤，並最終建立一個例外：

WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
15/04/08 12:37:06 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://[email protected]:47128] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 
15/04/08 12:37:12 WARN ReliableDeliverySupervisor: Association with remote system [akka.tcp://[email protected]:45975] has failed, address is now gated for [5000] ms. Reason is: [Disassociated]. 
15/04/08 12:37:13 ERROR YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED!

當我檢查容器的日誌，我看到它是SIGTERM -ed

15/04/08 12:37:08 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>) 
15/04/08 12:37:08 INFO yarn.YarnAllocator: Container request (host: Any, capability: <memory:1408, vCores:1>) 
15/04/08 12:37:08 INFO yarn.ApplicationMaster: Started progress reporter thread - sleep time : 5000 
15/04/08 12:37:12 ERROR yarn.ApplicationMaster: RECEIVED SIGNAL 15: SIGTERM 
15/04/08 12:37:12 INFO yarn.ApplicationMaster: Final app status: UNDEFINED, exitCode: 0, (reason: Shutdown hook called before final status was reported.) 
15/04/08 12:37:12 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with UNDEFINED (diag message: Shutdown hook called before final status was reported.)

SOLUTION：我解決了這個問題。我現在使用Java7而不是Java8。這種情況被報告爲錯誤，但它被拒絕，因此https://issues.apache.org/jira/browse/SPARK-6388 然而，更改Java版本確實奏效。

來源

2015-04-08 toobee

我承認。我面臨同樣的問題。將jdk版本更改爲7之後，它可以正常工作。我猜oracle-8-jdk有一些問題！ –

我以前也遇到過類似的問題，直到我發現這個issue

試圖阻止你SparkContext實例明確sc.stop()

來源

2015-04-09 17:10:33 Jonathan

我在Java 8下嘗試了你的想法，問題依然存在。這似乎是Java8的問題，不被開發人員承認（請參閱我上面更新的問題）。 – toobee

該協會可能會由於失去了Java 8過多的內存分配問題：https://issues.apache.org/jira/browse/YARN-4714

您可以強制YARN通過在yarn-site.xml中設置以下屬性來忽略此設置

<property> 
    <name>yarn.nodemanager.pmem-check-enabled</name> 
    <value>false</value> 
</property> 

<property> 
    <name>yarn.nodemanager.vmem-check-enabled</name> 
    <value>false</value> 
</property>

來源

2016-09-12 19:37:55 simpleJack

Spark Pi集羣模式下使用紗線的示例：丟失關聯

回答

相關問題