齊柏林星火解釋（sc.textFile）拋出的NoSuchMethodError

Vesions齊柏林星火解釋（sc.textFile）拋出的NoSuchMethodError

飛艇版本：0.7快照版本。

星火1.6

CDH 5.7.1

斯卡拉2.10

sc.textFile導致

java.lang.NoSuchMethodError: org.apache.hadoop.fs.BlockLocation.<init>([Ljava/lang/String;[Ljava/lang/String;[Ljava/lang/String;[Ljava/lang/String;[Ljava/lang/String;JJZ)V

val dataset=sc.textFile("/tmp/expenses.csv") 
dataset.count() 
dataset.first()

全STRACK跟蹤

dataset: org.apache.spark.rdd.RDD[String] = /tmp/expenses.csv MapPartitionsRDD[1] at textFile at <console>:29 
java.lang.NoSuchMethodError: org.apache.hadoop.fs.BlockLocation.<init>([Ljava/lang/String;[Ljava/lang/String;[Ljava/lang/String;[Ljava/lang/String;[Ljava/lang/String;JJZ)V 
at org.apache.hadoop.hdfs.DFSUtil.locatedBlocks2Locations(DFSUtil.java:522) 
at org.apache.hadoop.hdfs.DFSUtil.locatedBlocks2Locations(DFSUtil.java:486) 
at org.apache.hadoop.hdfs.DFSClient.getBlockLocations(DFSClient.java:1305) 
at org.apache.hadoop.hdfs.DistributedFileSystem$1.doCall(DistributedFileSystem.java:221) 
at org.apache.hadoop.hdfs.DistributedFileSystem$1.doCall(DistributedFileSystem.java:217) 
at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81) 
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileBlockLocations(DistributedFileSystem.java:217) 
at org.apache.hadoop.hdfs.DistributedFileSystem.getFileBlockLocations(DistributedFileSystem.java:209) 
at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:343) 
at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199) 
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:239) 
at org.apache.spark.rdd.RDD$$anonfun$partitions$2.apply(RDD.scala:237) 
at scala.Option.getOrElse(Option.scala:120

來源

2016-11-15 slash dev slash null