Spark worker拋出ERROR SendingConnection：異常時讀取連接ConnectionManagerId

我想執行一個簡單的應用程序示例代碼與火花。使用火花提交執行作業。火花提交--class 「SimpleJob」 --master火花：//：7077目標/斯卡拉2.10 /簡單project_2.10-1.0.jarSpark worker拋出ERROR SendingConnection：異常時讀取連接ConnectionManagerId

15/03/08 23:21:53 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
15/03/08 23:21:53 WARN LoadSnappy: Snappy native library not loaded 
15/03/08 23:22:09 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory 
Lines with a: 21, Lines with b: 21

的工作給出了正確的結果，但給出了以下錯誤它下面：

15/03/08 23:22:28 ERROR SendingConnection: Exception while reading SendingConnection to ConnectionManagerId(<worker-host.domain.com>,53628) 
java.nio.channels.ClosedChannelException 
    at sun.nio.ch.SocketChannelImpl.ensureReadOpen(SocketChannelImpl.java:252) 
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:295) 
    at org.apache.spark.network.SendingConnection.read(Connection.scala:390) 
    at org.apache.spark.network.ConnectionManager$$anon$6.run(ConnectionManager.scala:205) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
    at java.lang.Thread.run(Thread.java:745) 
15/03/08 23:22:28 ERROR ConnectionManager: Corresponding SendingConnection to ConnectionManagerId(<worker-host.domain.com>,53628) not found 
15/03/08 23:22:28 WARN ConnectionManager: All connections not cleaned up

以下是火花defaults.conf

spark.serializer     org.apache.spark.serializer.KryoSerializer 
spark.driver.memory    5g 
spark.master      spark://<master-ip>:7077 
spark.eventLog.enabled   true 
spark.executor.extraClassPath $SPARK-HOME/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar 
spark.cassandra.connection.conf.factory com.datastax.spark.connector.cql.DefaultConnectionFactory 
spark.cassandra.auth.conf.factory  com.datastax.spark.connector.cql.DefaultAuthConfFactory 
spark.cassandra.query.retry.count  10

以下是spark-env.sh

SPARK_LOCAL_IP=<master-ip in master worker-ip in workers> 
SPARK_MASTER_HOST='<master-hostname>' 
SPARK_MASTER_IP=<master-ip> 
SPARK_MASTER_PORT=7077 
SPARK_WORKER_CORES=2 
SPARK_WORKER_MEMORY=2g 
SPARK_WORKER_INSTANCES=4

來源

2015-03-08 tarun tiwari

得到了一個答案，

即使我被命令添加卡桑德拉連接到類路徑中，我不是發送相同的路徑羣集的所有節點。

我現在用下面的命令序列做正確

spark-shell --driver-class-path ~/Installers/spark-cassandra-connector-1.1.1/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.1.1.jar 
import org.apache.spark.SparkConf 
import org.apache.spark.SparkContext 
import com.datastax.spark.connector._ 
sc.addJar("~/Installers/spark-cassandra-connector-1.1.1/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.1.1.jar")

這些命令我能夠運行所有的讀&寫了我的卡桑德拉集羣中使用的火花RDDS正常後。

來源

2015-03-27 06:44:08

Spark worker拋出ERROR SendingConnection：異常時讀取連接ConnectionManagerId

回答

相關問題