火花提交「服務‘驅動程序’無法在端口綁定」錯誤

我用下面的命令來運行單詞計數的火花Java示例： -火花提交「服務‘驅動程序’無法在端口綁定」錯誤

time spark-submit --deploy-mode cluster --master spark://192.168.0.7:6066 --class org.apache.spark.examples.JavaWordCount /home/pi/Desktop/example/new/target/javaword.jar /books_50.txt

當運行它，以下是輸出： -

Running Spark using the REST application submission protocol. 
16/07/18 03:55:41 INFO rest.RestSubmissionClient: Submitting a request to launch an application in spark://192.168.0.7:6066. 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: Submission successfully created as driver-20160718035543-0000. Polling submission state... 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: Submitting a request for the status of submission driver-20160718035543-0000 in spark://192.168.0.7:6066. 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: State of driver driver-20160718035543-0000 is now RUNNING. 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: Driver is running on worker worker-20160718041005-192.168.0.12-42405 at 192.168.0.12:42405. 
16/07/18 03:55:44 INFO rest.RestSubmissionClient: Server responded with CreateSubmissionResponse: 
{ 
    "action" : "CreateSubmissionResponse", 
    "message" : "Driver successfully submitted as driver-20160718035543-0000", 
    "serverSparkVersion" : "1.6.2", 
    "submissionId" : "driver-20160718035543-0000", 
    "success" : true 
}

我檢查的具體工作人員（192.168.0.12），用於其日誌和它說： -

Launch Command: "/usr/lib/jvm/jdk-8-oracle-arm32-vfp-hflt/jre/bin/java" "-cp" "/opt/spark/conf/:/opt/spark/lib/spark-assembly-1.6.2-hadoop2.6.0.jar:/opt/spark/lib/datanucleus-api-jdo-3.2.6.jar:/opt/spark/lib/datanucleus-core-3.2.10.jar:/opt/spark/lib/datanucleus-rdbms-3.2.9.jar" "-Xms1024M" "-Xmx1024M" "-Dspark.driver.supervise=false" "-Dspark.app.name=org.apache.spark.examples.JavaWordCount" "-Dspark.submit.deployMode=cluster" "-Dspark.jars=file:/home/pi/Desktop/example/new/target/javaword.jar" "-Dspark.master=spark://192.168.0.7:7077" "-Dspark.executor.memory=10M" "org.apache.spark.deploy.worker.DriverWrapper" "spark://[email protected]:42405" "/opt/spark/work/driver-20160718035543-0000/javaword.jar" "org.apache.spark.examples.JavaWordCount" "/books_50.txt" 
======================================== 

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). 
log4j:WARN Please initialize the log4j system properly. 
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 
16/07/18 04:10:58 INFO SecurityManager: Changing view acls to: pi 
16/07/18 04:10:58 INFO SecurityManager: Changing modify acls to: pi 
16/07/18 04:10:58 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(pi); users with modify permissions: Set(pi) 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
16/07/18 04:11:00 WARN Utils: Service 'Driver' could not bind on port 0. Attempting port 1. 
Exception in thread "main" java.net.BindException: Cannot assign requested address: Service 'Driver' failed after 16 retries! Consider explicitly setting the appropriate port for the service 'Driver' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries. 
    at sun.nio.ch.Net.bind0(Native Method) 
    at sun.nio.ch.Net.bind(Net.java:433) 
    at sun.nio.ch.Net.bind(Net.java:425) 
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223) 
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) 
    at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125) 
    at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485) 
    at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089) 
    at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430) 
    at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415) 
    at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903) 
    at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198) 
    at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348) 
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) 
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) 
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) 
    at java.lang.Thread.run(Thread.java:745)

我spark-env.sh文件（主）包含： -

export SPARK_MASTER_WEBUI_PORT="8080" 
export SPARK_MASTER_IP="192.168.0.7" 
export SPARK_EXECUTOR_MEMORY="10M"

我spark-env.sh文件（工人）包含： -

export SPARK_WORKER_WEBUI_PORT="8080" 
export SPARK_MASTER_IP="192.168.0.7" 
export SPARK_EXECUTOR_MEMORY="10M"

請幫助...！

來源

2016-07-18 itsamineral

，你永遠能解決這個問題？我與spark v2.0.0完全一樣的問題。 – yee379

嗨，我無法找到這個特定問題的任何線索。因此，我開始爲wordcount流python示例。如果您找到解決方法，請讓我知道。 – itsamineral

這是可能的Spark 1.2.1 standalone cluster mode spark-submit is not working

重複我已經試過同樣的步驟，但能夠運行作業。如果可能，請發佈完整的spark-env.sh和spark-defaults。

來源

2016-07-18 07:03:48

我剛更新了這個問題。請檢查。 – itsamineral

您需要在/etc/hosts文件中輸入主機名。喜歡的東西：

127.0.0.1 localhost "hostname"

來源

2016-07-18 07:54:49

感謝您的建議馬納夫。我嘗試在hosts文件中添加/刪除/編輯該行，但不會改變任何內容。 – itsamineral

我很確定它與你的網絡設置有關，而不是你的火花設置。我得到了同樣的錯誤，並能夠通過在hosts文件中添加條目來解決它。可以肯定的是，你需要用你的主機名（在shell中輸入$ hostname cmd）替換最後一個cmd中的「主機名」，也沒有引號 –

這是根本原因。 – okwap

我試圖運行shell時有同樣的問題，並能夠通過設置SPARK_LOCAL_IP環境變量來得到這個工作。你可以在命令行中運行shell時轉讓本：

SPARK_LOCAL_IP=127.0.0.1 ./bin/spark-shell

對於一個更永久的解決方案，建立在你的星火根conf目錄下的一個文件spark-env.sh。添加以下行：

SPARK_LOCAL_IP=127.0.0.1

給執行權限使用chmod +x ./conf/spark-env.sh的腳本，這將設置默認這個環境變量。

來源

2016-12-08 23:16:56

在啓動腳本之前爲'SPARK_LOCAL_IP'做'export'工作 - 快速測試它是否有效。 –

我使用Maven/SBT來管理依賴關係，並且Spark核心包含在一個jar文件中。

您可以通過設置「spark.driver.bindAddress」（在這裏斯卡拉）覆蓋在運行時SPARK_LOCAL_IP：

val config = new SparkConf() 
config.setMaster("local[*]") 
config.setAppName("Test App") 
config.set("spark.driver.bindAddress", "127.0.0.1") 
val sc = new SparkContext(config)

來源

2017-04-05 15:33:29

火花提交「服務‘驅動程序’無法在端口綁定」錯誤

回答

相關問題