PrintWriter的是隻考慮一個斜槓，而不是在HDFS路徑雙斜槓

Question

我的代碼是：

val df = sqlContext.read 
     .format("com.databricks.spark.xml") 
     .option("rowTag", header) 
     .load("/input/du3_init.dat") 
val dfCI2 = df.select("CI2") 
dfCI2.printSchema() 
val path="hdfs://nameservice/user/CI2_Schema" 
new PrintWriter(path) { write(dfCI2.schema.treeString);close} 

當我在火花執行，我得到

Exception in thread "main" java.io.FileNotFoundException: hdfs:/nameservice/user/CI2_Schema (No such file or directory) 
     at java.io.FileOutputStream.open(Native Method) 
     at java.io.FileOutputStream.<init>(FileOutputStream.java:221) 
     at java.io.FileOutputStream.<init>(FileOutputStream.java:110) 

只有一個斜線的存在異常中顯示的hdfs路徑。如何解決這個問題？在此先感謝

Answer 1

如果要寫入hdfs，則不能使用PrintWriter。 PrintWriter不應該理解網絡路徑，例如hdfs://或ftp://。它適用於本地文件系統。

您可以通過獲取hdfs配置形式的spark上下文來致信hdfs。

import org.apache.hadoop.fs.FileSystem 
import java.io.BufferedOutputStream 


val hdfsConf = sparkContext.hadoopConfiguration 

val fileSystem: FileSystem = FileSystem.get(hdfsConf) 

val filePath = "hdfs://nameservice1/user/dhdpbankcrtbtch/CIW2_Schema" 

val hdfsFileOS: FSDataOutputStream = fileSystem.create(new Path(filePath)); 

// create a buffered output stream using the FSDataOutputStream 
val bos = new BufferedOutputStream(hdfsFileOS) 

bos.write(dfCIW2.schema.treeString.toBytes("utf-8")) 

bos.close()