如何激發提交到ZooKeeper管理的Mesos集羣（給出java.net.UnknownHostException：zk for mesos：// zk：// master URL）？

Question

我正在運行Spark 2.0.2和Mesos 0.28.2。

我試圖提出申請，星火，使用的ZooKeeper管理Mesos集羣作爲主：

$SPARK_HOME/bin/spark-submit --verbose \ 
--conf spark.mesos.executor.docker.image=$DOCKER_IMAGE \ 
--conf spark.mesos.executor.home=$SPARK_HOME \ 
--conf spark.executorEnv.MESOS_NATIVE_JAVA_LIBRARY=/usr/lib/libmesos.so \ 
--deploy-mode cluster \ 
--master mesos://zk://<ip 1>:2181,<ip 2>:2181,<ip 3>:2181/mesos \ 
--class $APP_MAIN_CLASS \ 
file://$APP_JAR_PATH 

（<ip 1>，<ip 2>，並且<ip 3>都在10.0.0.0/8 IPv4地址塊）

按照documentation，我似乎對主正確的格式：

大師網址˚F或Mesos的格式爲mesos：// host：單主Master Mesos集羣的主機：5050或多主Mesos集羣的Mesos：// zk：// host1：2181，host2：2181，host3：2181/mesos使用ZooKeeper。

然而，似乎火花讀mesos://zk://...字符串，然後嘗試連接到zk：

17/04/07 20:10:06 INFO RestSubmissionClient: Submitting a request to launch an application in mesos://zk://<ip 1>:2181,<ip 2>:2181,<ip 3>:2181/mesos. 
Exception in thread "main" java.net.UnknownHostException: zk 
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184) 
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) 
    at java.net.Socket.connect(Socket.java:589) 
    at java.net.Socket.connect(Socket.java:538) 
    at sun.net.NetworkClient.doConnect(NetworkClient.java:180) 
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:432) 
    at sun.net.www.http.HttpClient.openServer(HttpClient.java:527) 
    at sun.net.www.http.HttpClient.<init>(HttpClient.java:211) 
    at sun.net.www.http.HttpClient.New(HttpClient.java:308) 
    at sun.net.www.http.HttpClient.New(HttpClient.java:326) 
    at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1202) 
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1138) 
    at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1032) 
    at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:966) 
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316) 
    at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291) 
    at org.apache.spark.deploy.rest.RestSubmissionClient.org$apache$spark$deploy$rest$RestSubmissionClient$$postJson(RestSubmissionClient.scala:214) 
    at org.apache.spark.deploy.rest.RestSubmissionClient$$anonfun$createSubmission$3.apply(RestSubmissionClient.scala:89) 
    at org.apache.spark.deploy.rest.RestSubmissionClient$$anonfun$createSubmission$3.apply(RestSubmissionClient.scala:85) 
    at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733) 
    at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33) 
    at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) 
    at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732) 
    at org.apache.spark.deploy.rest.RestSubmissionClient.createSubmission(RestSubmissionClient.scala:85) 
    at org.apache.spark.deploy.rest.RestSubmissionClient$.run(RestSubmissionClient.scala:417) 
    at org.apache.spark.deploy.rest.RestSubmissionClient$.main(RestSubmissionClient.scala:430) 
    at org.apache.spark.deploy.rest.RestSubmissionClient.main(RestSubmissionClient.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:736) 
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:185) 
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:210) 
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:124) 
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 

如何獲得星火認識到，應該使用三個ZooKeeper的節點，而不是試圖連接到不存在的zk主機？

Answer 1

TL;博士它不會工作，除非你要麼改變--deploy-mode到client或使用主URL與單個Mesos主機，例如mesos://host:port。

以下行提供了提示在何處查找相關代碼。

17/04/07 20:10:06 INFO RestSubmissionClient：在mesos：// zk：//：2181，：2181，：2181/mesos中提交啓動應用程序的請求。

看起來這條消息只是用Spark Standalone和Apache Mesos打印出來的--deploy-mode cluster。將其更改爲默認client，並且部署路徑將更改並希望接受主URL。

查看自己負責羣集部署的代碼 - RestSubmissionClient。

HereRestSubmissionClient說：

private val supportedMasterPrefixes = Seq("spark://", "mesos://") 

這證明mesos://網址覆蓋，但here你看到下列情況：

private val masters: Array[String] = if (master.startsWith("spark://")) { 
    Utils.parseStandaloneMasterUrls(master) 
} else { 
    Array(master) 
} 

即打印出來here與上述INFO消息，顯示URL只能是單個Mesos主人。