1. 首頁
  2. 問答
  3. Spark應用程序可以在獨立模式下運行，但不能在紗線羣中運行

Spark應用程序可以在獨立模式下運行，但不能在紗線羣中運行

2016-10-28 57 views
2

恩，大家好，任務困擾我很久。我可以通過這個命令
spark-submit --master spark://fuxiuyin-virtual-machine:7077 test_app.py
以單機模式運行我的火花應用程序，但這一程序未能通過這個命令
spark-submit --master yarn test_app.pySpark應用程序可以在獨立模式下運行，但不能在紗線羣中運行

我覺得我的紗線集羣是健康的紗線集羣運行。
的jps輸出是

$ jps 
8289 Worker 
14882 NameNode 
15475 ResourceManager 
8134 Master 
15751 NodeManager 
15063 DataNode 
17212 Jps 
15295 SecondaryNameNode

和「羣集的節點頁面是
here

的​​輸出是

$ /opt/spark/bin/spark-submit --master yarn test_app.py 
16/10/28 16:54:39 INFO spark.SparkContext: Running Spark version 2.0.1 
16/10/28 16:54:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
16/10/28 16:54:39 INFO spark.SecurityManager: Changing view acls to: fuxiuyin 
16/10/28 16:54:39 INFO spark.SecurityManager: Changing modify acls to: fuxiuyin 
16/10/28 16:54:39 INFO spark.SecurityManager: Changing view acls groups to: 
16/10/28 16:54:39 INFO spark.SecurityManager: Changing modify acls groups to: 
16/10/28 16:54:39 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(fuxiuyin); groups with view permissions: Set(); users with modify permissions: Set(fuxiuyin); groups with modify permissions: Set() 
16/10/28 16:54:39 INFO util.Utils: Successfully started service 'sparkDriver' on port 42519. 
16/10/28 16:54:39 INFO spark.SparkEnv: Registering MapOutputTracker 
16/10/28 16:54:39 INFO spark.SparkEnv: Registering BlockManagerMaster 
16/10/28 16:54:39 INFO storage.DiskBlockManager: Created local directory at /opt/spark/blockmgr-1dcd1d1a-4cf4-4778-9b71-53e238a62c97 
16/10/28 16:54:39 INFO memory.MemoryStore: MemoryStore started with capacity 366.3 MB 
16/10/28 16:54:40 INFO spark.SparkEnv: Registering OutputCommitCoordinator 
16/10/28 16:54:40 INFO util.log: Logging initialized @1843ms 
16/10/28 16:54:40 INFO server.Server: jetty-9.2.z-SNAPSHOT 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/jobs,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/jobs/json,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/jobs/job,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/jobs/job/json,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/stages,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/stages/json,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/stages/stage,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/stages/stage/json,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/stages/pool,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/stages/pool/json,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/storage,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/storage/json,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/storage/rdd,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/storage/rdd/json,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/environment,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/environment/json,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/executors,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/executors/json,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/executors/threadDump,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/executors/threadDump/json,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/static,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/api,null,AVAILABLE} 
16/10/28 16:54:40 INFO handler.ContextHandler: Started [email protected]{/stages/stage/kill,null,AVAILABLE} 
16/10/28 16:54:40 INFO server.ServerConnector: Started [email protected]{HTTP/1.1}{fuxiuyin-virtual-machine:4040} 
16/10/28 16:54:40 INFO server.Server: Started @1962ms 
16/10/28 16:54:40 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 
16/10/28 16:54:40 INFO ui.SparkUI: Bound SparkUI to fuxiuyin-virtual-machine, and started at http://192.168.102.133:4040 
16/10/28 16:54:40 INFO client.RMProxy: Connecting to ResourceManager at fuxiuyin-virtual-machine/192.168.102.133:8032 
16/10/28 16:54:41 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers 
16/10/28 16:54:41 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (8192 MB per container) 
16/10/28 16:54:41 INFO yarn.Client: Will allocate AM container, with 896 MB memory including 384 MB overhead 
16/10/28 16:54:41 INFO yarn.Client: Setting up container launch context for our AM 
16/10/28 16:54:41 INFO yarn.Client: Setting up the launch environment for our AM container 
16/10/28 16:54:41 INFO yarn.Client: Preparing resources for our AM container 
16/10/28 16:54:41 WARN yarn.Client: Neither spark.yarn.jars nor spark.yarn.archive is set, falling back to uploading libraries under SPARK_HOME. 
16/10/28 16:54:42 INFO yarn.Client: Uploading resource file:/opt/spark/spark-97ecc15d-7f26-4b73-a67e-953fdc127898/__spark_libs__697818607740390689.zip -> hdfs://fuxiuyin-virtual-machine:9000/user/fuxiuyin/.sparkStaging/application_1477644823180_0001/__spark_libs__697818607740390689.zip 
16/10/28 16:54:45 INFO yarn.Client: Uploading resource file:/opt/spark/python/lib/pyspark.zip -> hdfs://fuxiuyin-virtual-machine:9000/user/fuxiuyin/.sparkStaging/application_1477644823180_0001/pyspark.zip 
16/10/28 16:54:45 INFO yarn.Client: Uploading resource file:/opt/spark/python/lib/py4j-0.10.3-src.zip -> hdfs://fuxiuyin-virtual-machine:9000/user/fuxiuyin/.sparkStaging/application_1477644823180_0001/py4j-0.10.3-src.zip 
16/10/28 16:54:45 INFO yarn.Client: Uploading resource file:/opt/spark/spark-97ecc15d-7f26-4b73-a67e-953fdc127898/__spark_conf__7760765070208746118.zip -> hdfs://fuxiuyin-virtual-machine:9000/user/fuxiuyin/.sparkStaging/application_1477644823180_0001/__spark_conf__.zip 
16/10/28 16:54:45 INFO spark.SecurityManager: Changing view acls to: fuxiuyin 
16/10/28 16:54:45 INFO spark.SecurityManager: Changing modify acls to: fuxiuyin 
16/10/28 16:54:45 INFO spark.SecurityManager: Changing view acls groups to: 
16/10/28 16:54:45 INFO spark.SecurityManager: Changing modify acls groups to: 
16/10/28 16:54:45 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(fuxiuyin); groups with view permissions: Set(); users with modify permissions: Set(fuxiuyin); groups with modify permissions: Set() 
16/10/28 16:54:45 INFO yarn.Client: Submitting application application_1477644823180_0001 to ResourceManager 
16/10/28 16:54:45 INFO impl.YarnClientImpl: Submitted application application_1477644823180_0001 
16/10/28 16:54:45 INFO cluster.SchedulerExtensionServices: Starting Yarn extension services with app application_1477644823180_0001 and attemptId None 
16/10/28 16:54:46 INFO yarn.Client: Application report for application_1477644823180_0001 (state: ACCEPTED) 
16/10/28 16:54:46 INFO yarn.Client: 
    client token: N/A 
    diagnostics: N/A 
    ApplicationMaster host: N/A 
    ApplicationMaster RPC port: -1 
    queue: default 
    start time: 1477644885891 
    final status: UNDEFINED 
    tracking URL: http://fuxiuyin-virtual-machine:8088/proxy/application_1477644823180_0001/ 
    user: fuxiuyin 
16/10/28 16:54:47 INFO yarn.Client: Application report for application_1477644823180_0001 (state: ACCEPTED) 
16/10/28 16:54:48 INFO yarn.Client: Application report for application_1477644823180_0001 (state: ACCEPTED) 
16/10/28 16:54:49 INFO yarn.Client: Application report for application_1477644823180_0001 (state: ACCEPTED) 
16/10/28 16:54:50 INFO yarn.Client: Application report for application_1477644823180_0001 (state: ACCEPTED) 
16/10/28 16:54:51 INFO yarn.Client: Application report for application_1477644823180_0001 (state: ACCEPTED) 
16/10/28 16:54:52 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null) 
16/10/28 16:54:52 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> fuxiuyin-virtual-machine, PROXY_URI_BASES -> http://fuxiuyin-virtual-machine:8088/proxy/application_1477644823180_0001), /proxy/application_1477644823180_0001 
16/10/28 16:54:52 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 
16/10/28 16:54:52 INFO yarn.Client: Application report for application_1477644823180_0001 (state: RUNNING) 
16/10/28 16:54:52 INFO yarn.Client: 
    client token: N/A 
    diagnostics: N/A 
    ApplicationMaster host: 192.168.102.133 
    ApplicationMaster RPC port: 0 
    queue: default 
    start time: 1477644885891 
    final status: UNDEFINED 
    tracking URL: http://fuxiuyin-virtual-machine:8088/proxy/application_1477644823180_0001/ 
    user: fuxiuyin 
16/10/28 16:54:52 INFO cluster.YarnClientSchedulerBackend: Application application_1477644823180_0001 has started running. 
16/10/28 16:54:52 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 39951. 
16/10/28 16:54:52 INFO netty.NettyBlockTransferService: Server created on 192.168.102.133:39951 
16/10/28 16:54:53 INFO storage.BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.102.133, 39951) 
16/10/28 16:54:53 INFO storage.BlockManagerMasterEndpoint: Registering block manager 192.168.102.133:39951 with 366.3 MB RAM, BlockManagerId(driver, 192.168.102.133, 39951) 
16/10/28 16:54:53 INFO storage.BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.102.133, 39951) 
16/10/28 16:54:53 INFO handler.ContextHandler: Started [email protected]{/metrics/json,null,AVAILABLE} 
16/10/28 16:54:57 INFO cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(null) 
16/10/28 16:54:57 INFO cluster.YarnClientSchedulerBackend: Add WebUI Filter. org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter, Map(PROXY_HOSTS -> fuxiuyin-virtual-machine, PROXY_URI_BASES -> http://fuxiuyin-virtual-machine:8088/proxy/application_1477644823180_0001), /proxy/application_1477644823180_0001 
16/10/28 16:54:57 INFO ui.JettyUtils: Adding filter: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter 
16/10/28 16:54:59 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.102.133:45708) with ID 1 
16/10/28 16:54:59 INFO storage.BlockManagerMasterEndpoint: Registering block manager fuxiuyin-virtual-machine:33074 with 366.3 MB RAM, BlockManagerId(1, fuxiuyin-virtual-machine, 33074) 
16/10/28 16:55:00 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Registered executor NettyRpcEndpointRef(null) (192.168.102.133:45712) with ID 2 
16/10/28 16:55:00 INFO cluster.YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8 
16/10/28 16:55:00 INFO storage.BlockManagerMasterEndpoint: Registering block manager fuxiuyin-virtual-machine:43740 with 366.3 MB RAM, BlockManagerId(2, fuxiuyin-virtual-machine, 43740) 
16/10/28 16:55:00 INFO spark.SparkContext: Starting job: collect at /home/fuxiuyin/test_app.py:8 
16/10/28 16:55:00 INFO scheduler.DAGScheduler: Got job 0 (collect at /home/fuxiuyin/test_app.py:8) with 2 output partitions 
16/10/28 16:55:00 INFO scheduler.DAGScheduler: Final stage: ResultStage 0 (collect at /home/fuxiuyin/test_app.py:8) 
16/10/28 16:55:00 INFO scheduler.DAGScheduler: Parents of final stage: List() 
16/10/28 16:55:00 INFO scheduler.DAGScheduler: Missing parents: List() 
16/10/28 16:55:00 INFO scheduler.DAGScheduler: Submitting ResultStage 0 (PythonRDD[1] at collect at /home/fuxiuyin/test_app.py:8), which has no missing parents 
16/10/28 16:55:00 INFO memory.MemoryStore: Block broadcast_0 stored as values in memory (estimated size 3.8 KB, free 366.3 MB) 
16/10/28 16:55:00 INFO memory.MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.5 KB, free 366.3 MB) 
16/10/28 16:55:00 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.102.133:39951 (size: 2.5 KB, free: 366.3 MB) 
16/10/28 16:55:00 INFO spark.SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1012 
16/10/28 16:55:00 INFO scheduler.DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (PythonRDD[1] at collect at /home/fuxiuyin/test_app.py:8) 
16/10/28 16:55:00 INFO cluster.YarnScheduler: Adding task set 0.0 with 2 tasks 
16/10/28 16:55:00 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, fuxiuyin-virtual-machine, partition 0, PROCESS_LOCAL, 5450 bytes) 
16/10/28 16:55:00 INFO scheduler.TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, fuxiuyin-virtual-machine, partition 1, PROCESS_LOCAL, 5469 bytes) 
16/10/28 16:55:00 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 0 on executor id: 2 hostname: fuxiuyin-virtual-machine. 
16/10/28 16:55:00 INFO cluster.YarnSchedulerBackend$YarnDriverEndpoint: Launching task 1 on executor id: 1 hostname: fuxiuyin-virtual-machine. 
16/10/28 16:55:01 ERROR cluster.YarnClientSchedulerBackend: Yarn application has already exited with state FINISHED! 
16/10/28 16:55:01 INFO server.ServerConnector: Stopped [email protected]{HTTP/1.1}{fuxiuyin-virtual-machine:4040} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/stages/stage/kill,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/api,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/static,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/executors/threadDump/json,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/executors/threadDump,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/executors/json,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/executors,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/environment/json,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/environment,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/storage/rdd/json,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/storage/rdd,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/storage/json,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/storage,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/stages/pool/json,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/stages/pool,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/stages/stage/json,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/stages/stage,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/stages/json,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/stages,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/jobs/job/json,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/jobs/job,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/jobs/json,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO handler.ContextHandler: Stopped [email protected]{/jobs,null,UNAVAILABLE} 
16/10/28 16:55:01 INFO ui.SparkUI: Stopped Spark web UI at http://192.168.102.133:4040 
16/10/28 16:55:01 INFO scheduler.DAGScheduler: Job 0 failed: collect at /home/fuxiuyin/test_app.py:8, took 0.383872 s 
16/10/28 16:55:01 INFO scheduler.DAGScheduler: ResultStage 0 (collect at /home/fuxiuyin/test_app.py:8) failed in 0.233 s 
16/10/28 16:55:01 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerStageCompleted([email protected]) 
Traceback (most recent call last): 
    File "/home/fuxiuyin/test_app.py", line 8, in <module> 
    print(data.collect()) 
    File "/opt/spark/python/lib/pyspark.zip/pyspark/rdd.py", line 776, in collect 
    File "/opt/spark/python/lib/py4j-0.10.3-src.zip/py4j/java_gateway.py", line 1133, in __call__ 
    File "/opt/spark/python/lib/py4j-0.10.3-src.zip/py4j/protocol.py", line 319, in get_return_value 
py4j.protocol.Py4JJavaError16/10/28 16:55:01 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerJobEnd(0,1477644901073,JobFailed(org.apache.spark.SparkException: Job 0 cancelled because SparkContext was shut down)) 
: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.collectAndServe. 
: org.apache.spark.SparkException: Job 0 cancelled because SparkContext was shut down 
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:818) 
    at org.apache.spark.scheduler.DAGScheduler$$anonfun$cleanUpAfterSchedulerStop$1.apply(DAGScheduler.scala:816) 
    at scala.collection.mutable.HashSet.foreach(HashSet.scala:78) 
    at org.apache.spark.scheduler.DAGScheduler.cleanUpAfterSchedulerStop(DAGScheduler.scala:816) 
    at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onStop(DAGScheduler.scala:1685) 
    at org.apache.spark.util.EventLoop.stop(EventLoop.scala:83) 
    at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1604) 
    at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1798) 
    at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1287) 
    at org.apache.spark.SparkContext.stop(SparkContext.scala:1797) 
    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:108) 
    at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:632) 
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1890) 
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1903) 
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1916) 
    at org.apache.spark.SparkContext.runJob(SparkContext.scala:1930) 
    at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:912) 
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151) 
    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112) 
    at org.apache.spark.rdd.RDD.withScope(RDD.scala:358) 
    at org.apache.spark.rdd.RDD.collect(RDD.scala:911) 
    at org.apache.spark.api.python.PythonRDD$.collectAndServe(PythonRDD.scala:453) 
    at org.apache.spark.api.python.PythonRDD.collectAndServe(PythonRDD.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:498) 
    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:237) 
    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) 
    at py4j.Gateway.invoke(Gateway.java:280) 
    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) 
    at py4j.commands.CallCommand.execute(CallCommand.java:79) 
    at py4j.GatewayConnection.run(GatewayConnection.java:214) 
    at java.lang.Thread.run(Thread.java:745) 

16/10/28 16:55:01 ERROR client.TransportClient: Failed to send RPC 9187551343857476032 to /192.168.102.133:45698: java.nio.channels.ClosedChannelException 
java.nio.channels.ClosedChannelException 
16/10/28 16:55:01 ERROR cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Sending RequestExecutors(0,0,Map()) to AM was unsuccessful 
java.io.IOException: Failed to send RPC 9187551343857476032 to /192.168.102.133:45698: java.nio.channels.ClosedChannelException 
    at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:249) 
    at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:233) 
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) 
    at io.netty.util.concurrent.DefaultPromise$LateListeners.run(DefaultPromise.java:845) 
    at io.netty.util.concurrent.DefaultPromise$LateListenerNotifier.run(DefaultPromise.java:873) 
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) 
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) 
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) 
    at java.lang.Thread.run(Thread.java:745) 
Caused by: java.nio.channels.ClosedChannelException 
16/10/28 16:55:01 INFO cluster.SchedulerExtensionServices: Stopping SchedulerExtensionServices 
(serviceOption=None, 
services=List(), 
started=false) 
16/10/28 16:55:01 ERROR util.Utils: Uncaught exception in thread Yarn application state monitor 
org.apache.spark.SparkException: Exception thrown in awaitResult 
    at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:77) 
    at org.apache.spark.rpc.RpcTimeout$$anonfun$1.applyOrElse(RpcTimeout.scala:75) 
    at scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:36) 
    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) 
    at org.apache.spark.rpc.RpcTimeout$$anonfun$addMessageIfTimeout$1.applyOrElse(RpcTimeout.scala:59) 
    at scala.PartialFunction$OrElse.apply(PartialFunction.scala:167) 
    at org.apache.spark.rpc.RpcTimeout.awaitResult(RpcTimeout.scala:83) 
    at org.apache.spark.scheduler.cluster.CoarseGrainedSchedulerBackend.requestTotalExecutors(CoarseGrainedSchedulerBackend.scala:508) 
    at org.apache.spark.scheduler.cluster.YarnSchedulerBackend.stop(YarnSchedulerBackend.scala:93) 
    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.stop(YarnClientSchedulerBackend.scala:151) 
    at org.apache.spark.scheduler.TaskSchedulerImpl.stop(TaskSchedulerImpl.scala:455) 
    at org.apache.spark.scheduler.DAGScheduler.stop(DAGScheduler.scala:1605) 
    at org.apache.spark.SparkContext$$anonfun$stop$8.apply$mcV$sp(SparkContext.scala:1798) 
    at org.apache.spark.util.Utils$.tryLogNonFatalError(Utils.scala:1287) 
    at org.apache.spark.SparkContext.stop(SparkContext.scala:1797) 
    at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:108) 
Caused by: java.io.IOException: Failed to send RPC 9187551343857476032 to /192.168.102.133:45698: java.nio.channels.ClosedChannelException 
    at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:249) 
    at org.apache.spark.network.client.TransportClient$3.operationComplete(TransportClient.java:233) 
    at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680) 
    at io.netty.util.concurrent.DefaultPromise$LateListeners.run(DefaultPromise.java:845) 
    at io.netty.util.concurrent.DefaultPromise$LateListenerNotifier.run(DefaultPromise.java:873) 
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) 
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) 
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) 
    at java.lang.Thread.run(Thread.java:745) 
Caused by: java.nio.channels.ClosedChannelException 
16/10/28 16:55:01 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 
16/10/28 16:55:01 INFO storage.DiskBlockManager: Shutdown hook called 
16/10/28 16:55:01 INFO util.ShutdownHookManager: Shutdown hook called 
16/10/28 16:55:01 INFO util.ShutdownHookManager: Deleting directory /opt/spark/spark-97ecc15d-7f26-4b73-a67e-953fdc127898/userFiles-f51df2cd-8ec0-4caa-862f-77db0cc72505 
16/10/28 16:55:01 INFO util.ShutdownHookManager: Deleting directory /opt/spark/spark-97ecc15d-7f26-4b73-a67e-953fdc127898/pyspark-5216f977-d3c3-495f-b91a-88fa2218696d 
16/10/28 16:55:01 INFO util.ShutdownHookManager: Deleting directory /opt/spark/spark-97ecc15d-7f26-4b73-a67e-953fdc127898 
16/10/28 16:55:01 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on fuxiuyin-virtual-machine:43740 (size: 2.5 KB, free: 366.3 MB) 
16/10/28 16:55:01 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(2, fuxiuyin-virtual-machine, 43740),broadcast_0_piece0,StorageLevel(memory, 1 replicas),2517,0)) 
16/10/28 16:55:01 INFO storage.BlockManagerInfo: Added broadcast_0_piece0 in memory on fuxiuyin-virtual-machine:33074 (size: 2.5 KB, free: 366.3 MB) 
16/10/28 16:55:01 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerBlockUpdated(BlockUpdatedInfo(BlockManagerId(1, fuxiuyin-virtual-machine, 33074),broadcast_0_piece0,StorageLevel(memory, 1 replicas),2517,0)) 
16/10/28 16:55:01 INFO memory.MemoryStore: MemoryStore cleared 
16/10/28 16:55:01 INFO storage.BlockManager: BlockManager stopped

和紗線的ResourceManager的日誌在

yarn-fuxiuyin-resourcemanager-fuxiuyin-virtual-machine.log

我這個用戶提交應用程序：

uid=1000(fuxiuyin) gid=1000(fuxiuyin) 組=1000(fuxiuyin),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),108(lpadmin),124(sambashare)

我test_app是

from pyspark import SparkContext, SparkConf 

conf = SparkConf().setAppName("test_app") 

sc = SparkContext(conf=conf) 
data = sc.parallelize([1, 2, 3]) 
data = data.map(lambda x: x + 1) 
print(data.collect())

我不如何​​解決它。

想想。

來源

2016-10-28 fuxiuyin

+0

您好，您使用來啓動這個應用程序是什麼用戶，什麼是它的ID？ （在終端中輸入id）。 – kulssaka

+0

請上傳您初始化上下文的代碼段。 –

+0

@KamalPradhan好的，我添加了我的測試應用程序的代碼。 – fuxiuyin

回答

0

肇事司機已打印因此使用下面的代碼之前收集來自工作節點的所有數據..

我認爲錯誤是由於

print(data.collect())

使用

for x in data.collect(): 
     print x

並使用spark提交爲：

spark-submit --master yarn deploy-mode cluster test_app.py

來源

2016-10-28 12:03:25

+0

我改變了我的代碼，但它也無法運行。紗線中一定有問題，因爲我的test_app可以在spark獨立模式下運行。 – fuxiuyin

+0

您可以將部署模式羣集添加到spark提交命令中嗎？ –

+0

是由ambari管理的集羣，或者是手動管理的集羣。 –

0

代替spark-submit --master yarn test_app.py 試試這個命令spark-submit --master yarn-client test_app.py

來源

2016-12-25 06:02:03 edge

相關問題

 相關問題