閱讀CSV的Spark問題

Question

我經歷了火花外殼的奇怪行爲。如果工作目錄是/我無法讀取CSV，但是如果它是任何其他目錄，這將工作。 ：

trehiou@cds-stage-ms4 ~> docker run -it --rm -v /data/spark- 

test:/mnt/data localhost:5000/spark-master:2.0.1 sh 
/# /app/spark-2.0.1/bin/spark-shell 
Setting default log level to "WARN". 
To adjust logging level use sc.setLogLevel(newLevel). 
16/11/02 10:01:59 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
16/11/02 10:01:59 WARN spark.SparkContext: Use an existing SparkContext, some configuration may not take effect. 
Spark context Web UI available at http://172.17.0.1:4040 
Spark context available as 'sc' (master = local[*], app id = local-1478080919699). 
Spark session available as 'spark'. 
Welcome to 
     ____    __ 
    /__/__ ___ _____/ /__ 
    _\ \/ _ \/ _ `/ __/ '_/ 
    /___/ .__/\_,_/_/ /_/\_\ version 2.0.1 
     /_/ 

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_102) 
Type in expressions to have them evaluated. 
Type :help for more information. 

scala> spark.read.option("header", "true").option("inferSchema", "true").csv("file:///mnt/data/test.csv").printSchema() 
java.io.IOException: No FileSystem for scheme: null 
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660) 
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667) 
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) 
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) 
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) 
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) 
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) 
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.makeQualifiedPath(SessionCatalog.scala:115) 
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.createDatabase(SessionCatalog.scala:145) 
    at org.apache.spark.sql.catalyst.catalog.SessionCatalog.<init>(SessionCatalog.scala:89) 
    at org.apache.spark.sql.internal.SessionState.catalog$lzycompute(SessionState.scala:95) 
    at org.apache.spark.sql.internal.SessionState.catalog(SessionState.scala:95) 
    at org.apache.spark.sql.internal.SessionState$$anon$1.<init>(SessionState.scala:112) 
    at org.apache.spark.sql.internal.SessionState.analyzer$lzycompute(SessionState.scala:112) 
    at org.apache.spark.sql.internal.SessionState.analyzer(SessionState.scala:111) 
    at org.apache.spark.sql.execution.QueryExecution.assertAnalyzed(QueryExecution.scala:49) 
    at org.apache.spark.sql.Dataset$.ofRows(Dataset.scala:64) 
    at org.apache.spark.sql.SparkSession.baseRelationToDataFrame(SparkSession.scala:382) 
    at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:143) 
    at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:413) 
    at org.apache.spark.sql.DataFrameReader.csv(DataFrameReader.scala:349) 
    ... 48 elided 

/# cd /mnt/data/ 
/mnt/data # /app/spark-2.0.1/bin/spark-shell 
Setting default log level to "WARN". 
To adjust logging level use sc.setLogLevel(newLevel). 
16/11/02 10:02:26 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
16/11/02 10:02:26 WARN spark.SparkContext: Use an existing SparkContext, some configuration may not take effect. 
Spark context Web UI available at http://172.17.0.1:4040 
Spark context available as 'sc' (master = local[*], app id = local-1478080946728). 
Spark session available as 'spark'. 
Welcome to 
     ____    __ 
    /__/__ ___ _____/ /__ 
    _\ \/ _ \/ _ `/ __/ '_/ 
    /___/ .__/\_,_/_/ /_/\_\ version 2.0.1 
     /_/ 

Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_102) 
Type in expressions to have them evaluated. 
Type :help for more information. 

scala> spark.read.option("header", "true").option("inferSchema", "true").csv("file:///mnt/data/test.csv").printSchema() 
root 
|-- 2MASS: string (nullable = true) 
|-- RAJ2000: double (nullable = true) 
|-- DEJ2000: double (nullable = true) 
|-- errHalfMaj: double (nullable = true) 
|-- errHalfMin: double (nullable = true) 
|-- errPosAng: integer (nullable = true) 
|-- Jmag: double (nullable = true) 
|-- Hmag: double (nullable = true) 
|-- Kmag: double (nullable = true) 
|-- e_Jmag: double (nullable = true) 
|-- e_Hmag: double (nullable = true) 
|-- e_Kmag: double (nullable = true) 
|-- Qfl: string (nullable = true) 
|-- Rfl: integer (nullable = true) 
|-- X: integer (nullable = true) 
|-- MeasureJD: double (nullable = true) 

我想了解爲什麼會發生這種情況，因爲這沒有任何意義。在這個例子中的PWD是我讀的文件存儲，但我測試了爲什麼其他隨機路徑，它也工作。也許這是一個錯誤，我應該提交一個錯誤報告？

編輯：在從HDFS

EDIT 2閱讀行爲是完全一樣的：我得到完全相同的行爲，通過開展工作時火花提交

Answer 1

爲spark.sql.warehouse.dir的默認值是{working-dir}/spark-warehouse。當從根目錄運行時，這將變爲//spark-warehouse，並且在此類路徑上調用hadoopPath.getFileSystem(hadoopConf)（在SessionCatalog.makeQualifiedPath中），Hadoop無法識別該方案。

解決方法很簡單 - 啓動火花外殼，同時覆蓋此參數與一些理智的價值開始file:/，如：

/app/spark-2.0.1/bin/spark-shell --conf spark.sql.warehouse.dir=file:/tmp/spark-warehouse 

這種不愉快的行爲可能與this open issue，但我不知道 - 可能是一個單獨的問題，因爲它適用於非根值。