線程「main」中的異常java.lang.NoClassDefFoundError：org/apache/hadoop/fs/FSDataInputStream

我正在向Hadoop集羣提交遠程Spark任務。但獲取下面的錯誤消息幫助我解決這個問題。線程「main」中的異常java.lang.NoClassDefFoundError：org/apache/hadoop/fs/FSDataInputStream

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream 
    at org.apache.spark.Logging$.<init>(Logging.scala:162) 
    at org.apache.spark.Logging$.<clinit>(Logging.scala) 
    at org.apache.spark.Logging$class.initializeIfNecessary(Logging.scala:103) 
    at org.apache.spark.Logging$class.log(Logging.scala:50) 
    at org.apache.spark.deploy.yarn.ApplicationMaster$.log(ApplicationMaster.scala:635) 
    at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:649) 
    at org.apache.spark.deploy.yarn.ExecutorLauncher$.main(ApplicationMaster.scala:674) 
    at org.apache.spark.deploy.yarn.ExecutorLauncher.main(ApplicationMaster.scala) 
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.fs.FSDataInputStream 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424) 
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357) 
    ... 8 more

來源

2017-03-27 shiva k

問題出在classpath上。

我已經添加下面的代碼的應用程序，它解決了這個問題：

.set("spark.hadoop.yarn.application.classpath", 
            "/etc/hadoop/conf," 
              + "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/*," 
              + "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop/lib/*," 
              + "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop-hdfs/*," 
              + "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop-hdfs/lib/*," 
              + "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop-mapreduce/*," 
              + "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop-mapreduce/lib/*," 
              + "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop-yarn/*," 
              + "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/hadoop-yarn/lib/*," 
              + "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/spark/*," 
              + "/opt/cloudera/parcels/CDH-5.7.1-1.cdh5.7.1.p0.11/lib/spark/lib/*")

來源

2017-04-17 08:36:51

看起來您正在使用爲用戶提供的Hadoop預先構建的Spark分配。嘗試下載一個說明「爲Hadoop 2.x或更高版本預先構建」的分發版，其中x將是您現有的Hadoop集羣版本，並查看它是否適合您。

來源

2017-04-07 23:40:24 Roman

我使用CDH集羣。我已經使用包裹安裝了火花。 –

對不起，沒有使用CDH的經驗，但我在我的獨立Spark羣集上發生了同樣的錯誤，該羣集已按上述方式成功解析。 – Roman

要解決此問題，運行火花用下面的命令：

/usr/local/spark/bin/spark-shell

來源

2017-06-07 12:38:40 Naresh

線程「main」中的異常java.lang.NoClassDefFoundError：org/apache/hadoop/fs/FSDataInputStream

回答

相關問題