java.lang.IllegalStateException：找不到任何編譯目錄

Question

我想在Intellij中運行Spark主和Worker。我已經成功啓動了火花大師和工作人員。工人也沒有任何問題連接到主人。我可以通過查看日誌和觸發Web UI來確認這一點。但是當我嘗試在這個設置上運行任何應用程序時，問題就開始了。 當前，我試圖在此設置上運行spark-shell，但是當它試圖掌握時，我在主服務器上收到了一些錯誤消息，並在該工作服上發生了IlligalStateException。

主日誌（啓動成功）

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 
16/09/21 16:42:33 INFO Master: Started daemon with process name: 22237@rana-pc 
16/09/21 16:42:33 INFO SignalUtils: Registered signal handler for TERM 
16/09/21 16:42:33 INFO SignalUtils: Registered signal handler for HUP 
16/09/21 16:42:33 INFO SignalUtils: Registered signal handler for INT 
16/09/21 16:42:34 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
16/09/21 16:42:34 INFO SecurityManager: Changing view acls to: rana 
16/09/21 16:42:34 INFO SecurityManager: Changing modify acls to: rana 
16/09/21 16:42:34 INFO SecurityManager: Changing view acls groups to: 
16/09/21 16:42:34 INFO SecurityManager: Changing modify acls groups to: 
16/09/21 16:42:34 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rana); groups with view permissions: Set(); users with modify permissions: Set(rana); groups with modify permissions: Set() 
16/09/21 16:42:35 INFO Utils: Successfully started service 'sparkMaster' on port 7077. 
16/09/21 16:42:35 INFO Master: Starting Spark master at spark://MASTER_IP:7077 
16/09/21 16:42:35 INFO Master: Running Spark version 2.0.0 
16/09/21 16:42:35 INFO log: Logging initialized @3645ms 
16/09/21 16:42:36 INFO Server: jetty-9.2.16.v20160414 
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@12fb5c3b{/app,null,AVAILABLE} 
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@48b5405c{/app/json,null,AVAILABLE} 
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@6bf42129{/,null,AVAILABLE} 
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@17c4d9fc{/json,null,AVAILABLE} 
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@5f1655bb{/static,null,AVAILABLE} 
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@3dc349a5{/app/kill,null,AVAILABLE} 
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@7774da5d{/driver/kill,null,AVAILABLE} 
16/09/21 16:42:36 INFO ServerConnector: Started ServerConnector@678fff3{HTTP/1.1}{0.0.0.0:8080} 
16/09/21 16:42:36 INFO Server: Started @3999ms 
16/09/21 16:42:36 INFO Utils: Successfully started service 'MasterUI' on port 8080. 
16/09/21 16:42:36 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://MASTER_IP:8080 
16/09/21 16:42:36 INFO Server: jetty-9.2.16.v20160414 
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@49c1de92{/,null,AVAILABLE} 
16/09/21 16:42:36 INFO ServerConnector: Started ServerConnector@3e8a8bfb{HTTP/1.1}{MASTER_IP:6066} 
16/09/21 16:42:36 INFO Server: Started @4050ms 
16/09/21 16:42:36 INFO Utils: Successfully started service on port 6066. 
16/09/21 16:42:36 INFO StandaloneRestServer: Started REST server for submitting applications on port 6066 
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@185eae98{/metrics/master/json,null,AVAILABLE} 
16/09/21 16:42:36 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@6f634fcf{/metrics/applications/json,null,AVAILABLE} 
16/09/21 16:42:36 INFO Master: I have been elected leader! New state: ALIVE 
16/09/21 16:42:46 INFO Master: Registering worker MASTER_IP:42839 with 6 cores, 6.7 GB RAM 

工人日誌（已成功啓動）

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 
16/09/21 17:24:08 INFO Worker: Started daemon with process name: 24007@rana-pc 
16/09/21 17:24:08 INFO SignalUtils: Registered signal handler for TERM 
16/09/21 17:24:08 INFO SignalUtils: Registered signal handler for HUP 
16/09/21 17:24:08 INFO SignalUtils: Registered signal handler for INT 
16/09/21 17:24:08 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
16/09/21 17:24:08 INFO SecurityManager: Changing view acls to: rana 
16/09/21 17:24:08 INFO SecurityManager: Changing modify acls to: rana 
16/09/21 17:24:08 INFO SecurityManager: Changing view acls groups to: 
16/09/21 17:24:08 INFO SecurityManager: Changing modify acls groups to: 
16/09/21 17:24:08 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rana); groups with view permissions: Set(); users with modify permissions: Set(rana); groups with modify permissions: Set() 
16/09/21 17:24:09 INFO Utils: Successfully started service 'sparkWorker' on port 50059. 
16/09/21 17:24:09 INFO Worker: Starting Spark worker MASTER_IP:50059 with 6 cores, 6.7 GB RAM 
16/09/21 17:24:09 INFO Worker: Running Spark version 2.0.0 
16/09/21 17:24:09 INFO Worker: Spark home: /usr/local/spark 
16/09/21 17:24:09 INFO log: Logging initialized @2328ms 
16/09/21 17:24:09 INFO Server: jetty-9.2.16.v20160414 
16/09/21 17:24:09 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@251c286b{/logPage,null,AVAILABLE} 
16/09/21 17:24:09 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@c5c7b5f{/logPage/json,null,AVAILABLE} 
16/09/21 17:24:09 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@4aa264e6{/,null,AVAILABLE} 
16/09/21 17:24:09 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@1f240248{/json,null,AVAILABLE} 
16/09/21 17:24:09 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@5b3f3eb7{/static,null,AVAILABLE} 
16/09/21 17:24:09 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@34d6200c{/log,null,AVAILABLE} 
16/09/21 17:24:09 INFO ServerConnector: Started ServerConnector@4eaf8f4d{HTTP/1.1}{0.0.0.0:8081} 
16/09/21 17:24:09 INFO Server: Started @2440ms 
16/09/21 17:24:09 INFO Utils: Successfully started service 'WorkerUI' on port 8081. 
16/09/21 17:24:09 INFO WorkerWebUI: Bound WorkerWebUI to 0.0.0.0, and started at http://MASTER_IP:8081 
16/09/21 17:24:09 INFO Worker: Connecting to master MASTER_IP:7077... 
16/09/21 17:24:09 INFO ContextHandler: Started o.e.j.s.ServletContextHandler@270d60ee{/metrics/json,null,AVAILABLE} 
16/09/21 17:24:09 INFO TransportClientFactory: Successfully created connection to /MASTER_IP:7077 after 35 ms (0 ms spent in bootstraps) 
16/09/21 17:24:10 INFO Worker: Successfully registered with master spark://MASTER_IP:7077 

主日誌（當嘗試運行火花殼）

16/09/21 17:26:08 INFO Master: Registering app Spark shell 
16/09/21 17:26:08 INFO Master: Registered app Spark shell with ID app-20160921172608-0000 
16/09/21 17:26:08 INFO Master: Launching executor app-20160921172608-0000/0 on worker worker-20160921172409-MASTER_IP-50059 
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/0 because it is FAILED 
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/1 on worker worker-20160921172409-MASTER_IP-50059 
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/1 because it is FAILED 
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/2 on worker worker-20160921172409-MASTER_IP-50059 
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/2 because it is FAILED 
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/3 on worker worker-20160921172409-MASTER_IP-50059 
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/3 because it is FAILED 
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/4 on worker worker-20160921172409-MASTER_IP-50059 
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/4 because it is FAILED 
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/5 on worker worker-20160921172409-MASTER_IP-50059 
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/5 because it is FAILED 
16/09/21 17:26:09 INFO Master: Launching executor app-20160921172608-0000/6 on worker worker-20160921172409-MASTER_IP-50059 
16/09/21 17:26:09 INFO Master: Removing executor app-20160921172608-0000/6 because it is FAILED 

個

工人日誌（當嘗試運行火花殼）

16/09/21 17:26:09 INFO Worker: Asked to launch executor app-20160921172608-0000/0 for Spark shell 
16/09/21 17:26:09 INFO SecurityManager: Changing view acls to: rana 
16/09/21 17:26:09 INFO SecurityManager: Changing modify acls to: rana 
16/09/21 17:26:09 INFO SecurityManager: Changing view acls groups to: 
16/09/21 17:26:09 INFO SecurityManager: Changing modify acls groups to: 
16/09/21 17:26:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rana); groups with view permissions: Set(); users with modify permissions: Set(rana); groups with modify permissions: Set() 
16/09/21 17:26:09 ERROR ExecutorRunner: Error running executor 
java.lang.IllegalStateException: Cannot find any build directories. 
    at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248) 
    at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:241) 
    at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:195) 
    at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:118) 
    at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39) 
    at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:47) 
    at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63) 
    at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51) 
    at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145) 
    at org.apache.spark.deploy.worker.ExecutorRunner$$anon$1.run(ExecutorRunner.scala:73) 
16/09/21 17:26:09 INFO Worker: Executor app-20160921172608-0000/0 finished with state FAILED message java.lang.IllegalStateException: Cannot find any build directories. 
16/09/21 17:26:09 INFO Worker: Asked to launch executor app-20160921172608-0000/1 for Spark shell 
16/09/21 17:26:09 INFO SecurityManager: Changing view acls to: rana 
16/09/21 17:26:09 INFO SecurityManager: Changing modify acls to: rana 
16/09/21 17:26:09 INFO SecurityManager: Changing view acls groups to: 
16/09/21 17:26:09 INFO SecurityManager: Changing modify acls groups to: 
16/09/21 17:26:09 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(rana); groups with view permissions: Set(); users with modify permissions: Set(rana); groups with modify permissions: Set() 
16/09/21 17:26:09 ERROR ExecutorRunner: Error running executor 
java.lang.IllegalStateException: Cannot find any build directories. 
    at org.apache.spark.launcher.CommandBuilderUtils.checkState(CommandBuilderUtils.java:248) 
    at org.apache.spark.launcher.AbstractCommandBuilder.getScalaVersion(AbstractCommandBuilder.java:241) 
    at org.apache.spark.launcher.AbstractCommandBuilder.buildClassPath(AbstractCommandBuilder.java:195) 
    at org.apache.spark.launcher.AbstractCommandBuilder.buildJavaCommand(AbstractCommandBuilder.java:118) 
    at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:39) 
    at org.apache.spark.launcher.WorkerCommandBuilder.buildCommand(WorkerCommandBuilder.scala:47) 
    at org.apache.spark.deploy.worker.CommandUtils$.buildCommandSeq(CommandUtils.scala:63) 
    at org.apache.spark.deploy.worker.CommandUtils$.buildProcessBuilder(CommandUtils.scala:51) 
    at org.apache.spark.deploy.worker.ExecutorRunner.org$apache$spark$deploy$worker$ExecutorRunner$$fetchAndRunExecutor(ExecutorRunner.scala:145) 

有人可以幫我解決這個問題？

Answer 1

我已經找到了問題。其實，我得到這個錯誤是因爲主人無法找到應用程序的啓動器（spark-shell）。 主人正在尋找目錄$ SPARK_HOME/launcher/target/scala-2.11，但沒有找到它。當我在IntelliJ中運行spark時，所以我的$ SPARK_HOME應該是啓動器文件夾所在的項目目錄。所以我改變了這一點，它開始工作完美。現在我可以在此設置上運行spark-shell應用程序。

如果您不想更改現有設置，那麼也可以通過對文件本身的AbstractCommandBuilder.java進行硬編碼來完成。