使用sc.addFile（）的火花管道

Question

在使用sc.addFile（rscript）添加R腳本時，遇到以下錯誤。

java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries. 
at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:278) 
at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:300) 
at org.apache.hadoop.util.Shell.<clinit>(Shell.java:293) 
at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:867) 
at org.apache.hadoop.fs.FileUtil.chmod(FileUtil.java:853) 
at org.apache.spark.util.Utils$.fetchFile(Utils.scala:407) 
at org.apache.spark.SparkContext.addFile(SparkContext.scala:1386) 
at org.apache.spark.SparkContext.addFile(SparkContext.scala:1340) 
at kafkaStreaming.Stream$.main(Stream.scala:22) 
at kafkaStreaming.Stream.main(Stream.scala) 

Answer 1

下載該文件從下面的鏈接：

https://github.com/srccodes/hadoop-common-2.2.0-bin

之前初始化sparkContext做：

System.setProperty("hadoop.home.dir", """/path/to/hadoop-common-2.2.0-bin-master""")