NullPointerException當我嘗試查找文件到HDFS中時，火花

我正在集羣方案中工作。代碼工作得很好，如果我認爲一個文件到HDFS，但是當我嘗試將所有文件列出到一個目錄，以處理它們每次一個文件，如下NullPointerException當我嘗試查找文件到HDFS中時，火花

val sc = new SparkContext(conf) 
val sqlContext = new org.apache.spark.sql.SQLContext(sc) 
import sqlContext.implicits._ 
val files = new File("hdfs://nh-me:9000/user/user1/TotalFIle/").listFiles.map(_.getName).toList

我碰到下面的錯誤，請注意，第53行中出現錯誤是第四線之上的一段代碼，我貼：

17/02/14 23:57:19 ERROR yarn.ApplicationMaster: User class threw exception: java.lang.NullPointerException 
java.lang.NullPointerException 
at  scala.collection.mutable.ArrayOps$ofRef$.length$extension(ArrayOps.scala:192) 
at scala.collection.mutable.ArrayOps$ofRef.length(ArrayOps.scala:192) 
at scala.collection.SeqLike$class.size(SeqLike.scala:106) 
at scala.collection.mutable.ArrayOps$ofRef.size(ArrayOps.scala:186) 
at scala.collection.mutable.Builder$class.sizeHint(Builder.scala:69) 
at scala.collection.mutable.ArrayBuilder.sizeHint(ArrayBuilder.scala:22) 
at scala.collection.TraversableLike$class.builder$1(TraversableLike.scala:230) 
at scala.collection.TraversableLike$class.map(TraversableLike.scala:233) 
at scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) 
at SparkGraphGen$.main(CorrDistGraph.scala:53) 
at SparkGraphGen.main(CorrDistGraph.scala) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
at java.lang.reflect.Method.invoke(Method.java:497) 
at org.apache.spark.deploy.yarn.ApplicationMaster$$anon$2.run(ApplicationMaster.scala:627) 
17/02/14 23:57:19 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 15, (reason: User class threw exception: java.lang.NullPointerException)

來源

2017-02-14 user7331766

不能使用Java文件API來訪問HDFS文件。以下是使用HDFS API訪問文件的示例：

import org.apache.hadoop.fs._ 
import org.apache.hadoop.conf.Configuration 

val path = new Path("hdfs://nh-me:9000/user/user1/TotalFIle/") 
val fs = path.getFileSystem(new Configuration()) 
val files = fs.listStatus(path).map(_.getPath.getName).toList

來源

2017-02-14 23:27:05 zsxwing

NullPointerException當我嘗試查找文件到HDFS中時，火花

回答

相關問題