EMR 5.0 + Spark無盡循環堆棧

我試圖部署Spark 2.0 StreamingAmazon EMR 5.0。似乎應用陷在無限循環INFO客戶端日誌「的無限循環‘：申請報告application_14111979683_1111（狀態：接受）。’ 然後退出EMR 5.0 + Spark無盡循環堆棧

這裏是我如何努力deploy- -

AWS EMR添加步驟--cluster-ID --steps 類型=星火，名稱= 「星火計劃」，ActionOnFailure =繼續，參數數量= [：通過命令行提交mode，cluster， - class ,, s3：//.jar]

有什麼想法？

感謝，葉蘭

16/08/30 15:43:27 INFO SecurityManager: Changing view acls to: hadoop 
16/08/30 15:43:27 INFO SecurityManager: Changing modify acls to: hadoop 
16/08/30 15:43:27 INFO SecurityManager: Changing view acls groups to: 
16/08/30 15:43:27 INFO SecurityManager: Changing modify acls groups to: 
16/08/30 15:43:27 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); groups with view permissions: Set(); users with modify permissions: Set(hadoop); groups with modify permissions: Set() 
16/08/30 15:43:27 INFO Client: Submitting application application_14111979683_1111 to ResourceManager 
16/08/30 15:43:27 INFO YarnClientImpl: Submitted application application_14111979683_1111 
16/08/30 15:43:28 INFO Client: Application report for application_14111979683_1111 (state: ACCEPTED) 
16/08/30 15:43:28 INFO Client: 
    client token: N/A 
    diagnostics: N/A 
    ApplicationMaster host: N/A 
    ApplicationMaster RPC port: -1 
    queue: default 
    start time: 1472571807467 
    final status: UNDEFINED 
    tracking URL: http://xxxxxx:20888/proxy/application_14111979683_1111/ 
    user: hadoop 
16/08/30 15:43:29 INFO Client: Application report for application_14111979683_1111 (state: ACCEPTED)

，這將引發的異常：

16/08/31 08:14:48 INFO Client: 
    client token: N/A 
    diagnostics: Application application_1472630652740_0001 failed 2 times due to AM Container for appattempt_1472630652740_0001_000002 exited with exitCode: 13 
For more detailed output, check application tracking page:http://ip-10-0-0-8.eu-west-1.compute.internal:8088/cluster/app/application_1472630652740_0001Then, click on links to logs of each attempt. 
Diagnostics: Exception from container-launch. 
Container id: container_1472630652740_0001_02_000001 
Exit code: 13 
Stack trace: ExitCodeException exitCode=13: 
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:545) 
    at org.apache.hadoop.util.Shell.run(Shell.java:456) 
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:722) 
    at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:212) 
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302) 
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82) 
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745)

來源

2016-08-30 EranM

是那組參數是否正確？ '[--deploy-mode，cluster， - class ,, s3：//.jar]'s3：//.jar當然不是一個真正的路徑。 – Kristian

不是不完整的路徑。當我寫這個問題時，它似乎被刪除了。 – EranM

這是拋出的真正異常： – EranM

EMR實際上是一個包裝紗。因此，我們需要在部署命令行中添加「--master yarn」作爲參數。示例： aws emr add-steps --cluster-id j-XXXXXXXXX --steps Type = Spark，Name =「Spark Program」，ActionOnFailure = CONTINUE，Args = [ - deploy-mode，cluster， - master， yarn， - class，com.xxx.MyMainClass，s3：//]

需要的另一件事是從spark的初始化中刪除'sparkConf.setMaster（「local [*]」）'，。

來源

2016-09-05 07:50:27 EranM

EMR 5.0 + Spark無盡循環堆棧

回答

相關問題