如何找到我的Spark的驅動程序節點？

Question

我的火花工作是由oozie在色相submmited。火花以紗線簇模式運行。我想監視所謂的駕駛者的4040端口運行的應用程序的狀態，但我找不到4040端口，我檢查過程：

appuser 137872 137870 0 18:55 ? 00:00:00 /bin/bash -c /home/jdk/bin/java -server -Xmx4096m -Djava.io.tmpdir=/data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0547/container_1493800575189_0547_01_000004/tmp '-Dspark.driver.port=36503' '-Dspark.ui.port=0' -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493800575189_0547/container_1493800575189_0547_01_000004 -XX:OnOutOfMemoryError='kill %p' org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.120.117.107:36503 --executor-id 3 --hostname 10.120.117.100 --cores 1 --app-id application_1493800575189_0547 --user-class-path file:/data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0547/container_1493800575189_0547_01_000004/__app__.jar 1> /home/log/hadoop/logs/userlogs/application_1493800575189_0547/container_1493800575189_0547_01_000004/stdout 2> /home/log/hadoop/logs/userlogs/application_1493800575189_0547/container_1493800575189_0547_01_000004/stderr 
appuser 138337 137872 99 18:55 ? 00:05:11 /home/jdk/bin/java -server -Xmx4096m -Djava.io.tmpdir=/data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0547/container_1493800575189_0547_01_000004/tmp -Dspark.driver.port=36503 -Dspark.ui.port=0 -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493800575189_0547/container_1493800575189_0547_01_000004 -XX:OnOutOfMemoryError=kill %p org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://CoarseGrainedScheduler@10.120.117.107:36503 --executor-id 3 --hostname 10.120.117.100 --cores 1 --app-id application_1493800575189_0547 --user-class-path file:/data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0547/container_1493800575189_0547_01_000004/__app__.jar 

我不知道爲什麼spark.ui.port是0而不是4040.當然，我的linux系統不允許使用端口0。所以，我無法從REST api監視應用程序狀態。

有人可以給我一些建議嗎？

---

從馬裏烏什答案十分感謝，是下面的火花ApplicationMaster過程？

[appuser@hz-10-120-117-100 bin]$ ps -ef|grep ApplicationMaster 
appuser 125805 125803 0 May03 ?  00:00:00 /bin/bash -c /home/jdk/bin/java -server -Xmx1024m -Djava.io.tmpdir=/data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0014/container_1493800575189_0014_01_000001/tmp -Dspark.yarn.app.container.log.dir=/home/log/hadoop/logs/userlogs/application_1493800575189_0014/container_1493800575189_0014_01_000001 org.apache.spark.deploy.yarn.ApplicationMaster --class 'com.netease.ecom.data.gjs.statis.online.app.day.AppDayRealtimeStatis' --jar hdfs://datahdfsmaster/user/appuser/bjmazhengbing/jar/spark_streaming/spark-streaming-etl-2.0.jar --arg 'analysis_gjs_online.properties' --arg 'rrr' --properties-file /data6/data/hadoop/tmp/usercache/appuser/appcache/application_1493800575189_0014/container_1493800575189_0014_01_000001/__spark_conf__/__spark_conf__.properties 1> /home/log/hadoop/logs/userlogs/application_1493800575189_0014/container_1493800575189_0014_01_000001/stdout 2> /home/log/hadoop/logs/userlogs/application_1493800575189_0014/container_1493800575189_0014_01_000001/stderr 

據星火的正式文件，驅動程序都應該有用於監視一個4040端口，但是我的驅動程序似乎沒有打開任何端口：

[appuser@hz-10-120-117-100 bin]$ netstat -ntlp|grep 125805 
(Not all processes could be identified, non-owned process info 
will not be shown, you would have to be root to see it all.) 

我最終找出驅動程序端口的目的是監視應用程序狀態。有什麼建議麼？

Answer 1

您列出的過程是執行者，而不是驅動程序。

當您在yarn-cluster模式下運行應用程序時，spark驅動程序和yarn application master在同一個JVM中運行。因此，確定Spark UI地址的最簡單方法是轉至資源管理器的用戶界面，找到您的應用程序並單擊指向Application Master的鏈接。這將是一個代理地址，指向驅動程序的ui端口。