1. 首頁
  2. 問答
  3. 火花獨立簇輪胎訪問本地python.exe

火花獨立簇輪胎訪問本地python.exe

2017-04-07 45 views
0

當火花集羣上執行的應用程序的python我碰上以下異常：火花獨立簇輪胎訪問本地python.exe

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 
Setting default log level to "WARN". 
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel). 
17/04/07 10:57:01 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
[Stage 0:>               (0 + 2)/2]17/04/07 10:57:07 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, 192.168.2.113, executor 0): java.io.IOException: Cannot run program "C:\Users\<local-user>\Anaconda2\python.exe": CreateProcess error=2, The system cannot find the file specified 
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) 
    at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:120) 
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:67) 
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:116) 
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:128) 
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63) 
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) 
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) 
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) 
    at org.apache.spark.scheduler.Task.run(Task.scala:99) 
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified 
    at java.lang.ProcessImpl.create(Native Method) 
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:386) 
    at java.lang.ProcessImpl.start(ProcessImpl.java:137) 
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) 
    ... 13 more 

17/04/07 10:57:07 ERROR TaskSetManager: Task 0 in stage 0.0 failed 4 times; aborting job 
Traceback (most recent call last): 
    File "<redacted-absolute-path>.py", line 49, in <module> 
    eco_rdd 
    File "C:\spark\python\pyspark\rdd.py", line 2139, in zipWithIndex 
    nums = self.mapPartitions(lambda it: [sum(1 for i in it)]).collect() 
    File "C:\spark\python\pyspark\rdd.py", line 809, in collect 
    port = self.ctx._jvm.PythonRDD.collectAndServe(self._jrdd.rdd()) 
    File "C:\spark\python\lib\py4j-0.10.4-src.zip\py4j\java_gateway.py", line 1133, in __call__ 
    File "C:\spark\python\pyspark\sql\utils.py", line 63, in deco 
    return f(*a, **kw) 
    File "C:\spark\python\lib\py4j-0.10.4-src.zip\py4j\protocol.py", line 319, in get_return_value 
py4j.protocol.Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe. 
: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 6, 192.168.2.113, executor 0): java.io.IOException: Cannot run program "C:\Users\<local-user>\Anaconda2\python.exe": CreateProcess error=2, The system cannot find the file specified 
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) 
    at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:120) 
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:67) 
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:116) 
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:128) 
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63) 
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) 
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) 
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) 
    at org.apache.spark.scheduler.Task.run(Task.scala:99) 
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    at java.lang.Thread.run(Thread.java:745) 
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified 
    at java.lang.ProcessImpl.create(Native Method) 
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:386) 
    at java.lang.ProcessImpl.start(ProcessImpl.java:137) 
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) 
    ... 13 more 

Driver stacktrace: 
    at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435) 
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423) 
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422) 
    at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) 
    at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48) 
    at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422) 
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802) 
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802) 
    at scala.Option.foreach(Option.scala:257) 
    at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802) 
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650) 
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605) 
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594) 
    at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48) 
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628) 
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1918) 
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1931) 
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1944) 
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1958) 
    at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:935) 
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) 
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) 
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:362) 
    at org.apache.spark.rdd.RDD.collect(RDD.scala:934) 
    at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:453) 
    at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) 
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) 
    at py4j.Gateway.invoke(Gateway.java:280) 
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) 
    at py4j.commands.CallCommand.execute(CallCommand.java:79) 
    at py4j.GatewayConnection.run(GatewayConnection.java:214) 
    at java.lang.Thread.run(Thread.java:745) 
Caused by: java.io.IOException: Cannot run program "C:\Users\<local-user>\Anaconda2\python.exe": CreateProcess error=2, The system cannot find the file specified 
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) 
    at org.apache.spark.api.python.PythonWorkerFactory.createSimpleWorker(PythonWorkerFactory.scala:120) 
    at org.apache.spark.api.python.PythonWorkerFactory.create(PythonWorkerFactory.scala:67) 
    at org.apache.spark.SparkEnv.createPythonWorker(SparkEnv.scala:116) 
    at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:128) 
    at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63) 
    at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323) 
    at org.apache.spark.rdd.RDD.iterator(RDD.scala:287) 
    at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87) 
    at org.apache.spark.scheduler.Task.run(Task.scala:99) 
    at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
    ... 1 more 
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified 
    at java.lang.ProcessImpl.create(Native Method) 
    at java.lang.ProcessImpl.<init>(ProcessImpl.java:386) 
    at java.lang.ProcessImpl.start(ProcessImpl.java:137) 
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029) 
    ... 13 more

不知何故（在同一網絡上的遠程PC）的簇嘗試訪問本地的Python（即安裝在我的本地工作站執行該司機）：

Caused by: java.io.IOException: Cannot run program "C:\Users\<local-user>\Anaconda2\python.exe": CreateProcess error=2, The system cannot find the file specified
  • 星火2.1.0
  • 火花獨立集羣在Windows 10
  • 工作站在Windows 7
  • 運行連接到集羣，並與spark-shell（交互）執行任務運行的工作沒有問題
  • 連接到集羣，並與pyspark（互動）工程執行任務沒有問題
  • 從pycharm運行直接導致上述
  • 異常使用​​要執行造成了類似的問題（=>試圖訪問我的本地蟒蛇）
  • 使用findspark
    • 個
  • 的PYSPARK_PYTHON和PYSPARK_DRIVER_PYTHON環境變量設置
  • Python的集羣和工作站：Python的2.7.13 ::蟒蛇定製（64位）

預先感謝您的幫助

來源

2017-04-07 TheJP

+0

只是一個瘋狂的想法 - 你是否試圖強制執行器conf中的'PYSPARK_PYTHON'，即在'spark-defaults.conf'文件中添加'spark.executorEnv.PYSPARK_PYTHON C：\ some \ path \ python.exe'。 （或通過'--conf'在命令行中）？ –

+0

我試圖以各種不同的方式設置配置，但得到了相同的錯誤。在某些情況下，它試圖用遠程python實例（=另一個異常）執行本地驅動程序。 你必須在本地工作站上設置這些配置嗎？我試圖設置它（本地和遠程），只是爲了確保。 – TheJP

+0

兩個env變量都由'spark-submit'工具使用;理論上，執行者應該使用'PYSPARK_PYTHON'的值，它應該指向執行者機器上的Python安裝;和「PYSPARK_DRIVER_PYTHON」的值（如果存在，否則默認爲PYSPARK_PYTHON值）應該被驅動程序使用，並且它應該指向驅動程序機器上的Python安裝*（它可能與'spark-submit'機器不同 - 例如''yarn-cluster''）* –

回答

0

所以我現在爲我的問題找到了一個解決方案。對於我的設置問題是findspark庫。刪除允許我運行羣集上的所有任務。

如果您有類似的問題，但它不是與findspark相關的，我會向您指出上面的Samson Scharfrichter的意見。

來源

2017-04-10 08:22:25 TheJP

相關問題

 相關問題