如何在Docker容器中設置執行器IP？

Question

最後3天，我試圖建立一個泊塢窗機3個部分組成： 火花法師，一個星火工人和1名司機（Java）的應用

當從泊塢窗以外啓動驅動程序，一切工作正常。但是開始，所有三個組成部分導致了端口防火牆主機的噩夢

爲了保持它（第一），簡單的我用泊塢窗 - 撰寫 - 這是我的搬運工，compose.yml：

driver: 
    hostname: driver 
    image: driverimage 
    command: -Dexec.args="0 192.168.99.100" -Dspark.driver.port=7001 -Dspark.driver.host=driver -Dspark.executor.port=7006 -Dspark.broadcast.port=15001 -Dspark.fileserver.port=15002 -Dspark.blockManager.port=15003 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory 
    ports: 
    - 10200:10200 # Module REST Port 
    - 4040:4040 # Web UI (Spark) 
    - 7001:7001 # Driver Port (Spark) 
    - 15001:15001 # Broadcast (Spark) 
    - 15002:15002 # File Server (Spark) 
    - 15003:15003 # Blockmanager (Spark) 
    - 7337:7337 # Shuffle? (Spark) 
    extra_hosts: 
    - sparkmaster:192.168.99.100 
    - sparkworker:192.168.99.100 
    environment: 
    SPARK_LOCAL_IP: 192.168.99.100 
    #SPARK_MASTER_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" 
    #SPARK_WORKER_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" 
    SPARK_JAVA_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=15001 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" 

sparkmaster: 
    extra_hosts: 
    - driver:192.168.99.100 
    image: gettyimages/spark 
    command: /usr/spark/bin/spark-class org.apache.spark.deploy.master.Master -h sparkmaster 
    hostname: sparkmaster 
    environment: 
    SPARK_CONF_DIR: /conf 
    MASTER: spark://sparkmaster:7077 
    SPARK_LOCAL_IP: 192.168.99.100 
    SPARK_JAVA_OPTS: "-Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" 
    SPARK_WORKER_OPTS: "-Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" 
    SPARK_MASTER_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" 
    #SPARK_WORKER_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" 
    #SPARK_JAVA_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" 
    expose: 
    - 7001 
    - 7002 
    - 7003 
    - 7004 
    - 7005 
    - 7006 
    - 7077 
    - 6066 
    ports: 
    - 6066:6066 
    - 7077:7077 # Master (Main Port) 
    - 8080:8080 # Web UI 
    #- 7006:7006 # Executor 

sparkworker: 
    extra_hosts: 
    - driver:192.168.99.100 
    image: gettyimages/spark 
    command: /usr/spark/bin/spark-class org.apache.spark.deploy.worker.Worker -h sparkworker spark://sparkmaster:7077 
# volumes: 
# - ./spark/logs:/log/spark 
    hostname: sparkworker 
    environment: 
    SPARK_CONF_DIR: /conf 
    SPARK_WORKER_CORES: 4 
    SPARK_WORKER_MEMORY: 4g 
    SPARK_WORKER_PORT: 8881 
    SPARK_WORKER_WEBUI_PORT: 8081 
    SPARK_LOCAL_IP: 192.168.99.100 
    #SPARK_MASTER_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" 
    SPARK_JAVA_OPTS: "-Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" 
    SPARK_MASTER_OPTS: "-Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" 
    SPARK_WORKER_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=15003 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" 
    #SPARK_JAVA_OPTS: "-Dspark.driver.port=7001 -Dspark.fileserver.port=7002 -Dspark.broadcast.port=7003 -Dspark.replClassServer.port=7004 -Dspark.blockManager.port=7005 -Dspark.executor.port=7006 -Dspark.ui.port=4040 -Dspark.broadcast.factory=org.apache.spark.broadcast.HttpBroadcastFactory" 
    links: 
    - sparkmaster 
    expose: 
    - 7001 
    - 7002 
    - 7003 
    - 7004 
    - 7005 
    - 7006 
    - 7012 
    - 7013 
    - 7014 
    - 7015 
    - 7016 
    - 8881 
    ports: 
    - 8081:8081 # WebUI 
    #- 15003:15003 # Blockmanager+ 
    - 7005:7005 # Executor 
    - 7006:7006 # Executor 
    #- 7006:7006 # Executor 

我甚至不知道哪個端口實際使用等等。我知道的是，我目前的問題是以下。司機可以與師父溝通，師父可以與員工溝通，我認爲司機可以與員工溝通但是！司機不能與執行者通信。我也發現了這個問題。當我打開應用程序UI並打開exectuors選項卡時，它顯示「Executor 0 - Address 172.17.0.1:7005」。

所以問題是，驅動程序使用Docker網關地址執行程序尋址執行程序，該程序不起作用。我嘗試了幾件事情（SPARK_LOCAL_IP，使用顯式主機名等），但驅動程序總是試圖與Docker Gateway進行通信......任何想法如何實現驅動程序可以與執行者/工作者進行通信？

Answer 1

這是由於Spark提供的配置選項不足。 Spark會綁定在SPARK_LOCAL_HOSTNAME上偵聽，並將此確切的主機名傳播到集羣。不幸的是，如果驅動程序在NAT後面，例如Docker容器，此設置不起作用。

您可以解決這跟下面的設置（我用這個成功破解）：

• 前所有必要的端口，1對1（爲你做的）
• 使用自定義的主機名爲司機：設置例如SPARK_LOCAL_HOSTNAME: mydriver
• 對於主節​​點和工作節點，添加192.168.99.100 mydriver到/etc/hosts，以便它們可以到達Spark驅動程序。
• 碼頭集裝箱，地圖mydriver至0.0.0.0。這將使星火驅動結合0.0.0.0，所以這將是可達大師和工人：

做，在泊塢窗，compose.yml，只需添加下面幾行：

extra_hosts: 
    - "mydriver:0.0.0.0"