我可以使用下面的命令如何使用SparkR在apache spark中編寫csv文件？
sc = sparkR.init(master = 'local', sparkPackages = 'com.databricks:spark-csv_2.11:1.4.0')
sqlContext <- sparkRSQL.init(sc)
ss <- read.df(sqlContext, '/home/anmol/Downloads/Rgraphics/dataSets/states.csv', source = "com.databricks.spark.csv", inferSchema = "true")
head(ss)
我曾嘗試下面的命令來成功加載數據
write.df(df, '/home/anmol/faithfull.csv', source = 'com.databricks.spark.csv', 'overwrite')
，但它提供了以下錯誤
16/06/10 18:28:26 ERROR RBackendHandler: save on 261 failed Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) : java.lang.NoClassDefFoundError: Could not initialize class com.databricks.spark.csv.util.CompressionCodecs$ at com.databricks.spark.csv.DefaultSource.createRelation(DefaultSource.scala:189) at org.apache.spark.sql.execution.datasources.ResolvedDataSource$.apply(ResolvedDataSource.scala:222) at org.apache.spark.sql.DataFrameWriter.save(DataFrameWriter.scala:148) at org.apache.spark.sql.DataFrame.save(DataFrame.scala:2027) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.api.r.RBackendHandler.handleMethodCall(RBackendHandler.scala:141) at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:86) at org.apache.spark.api.r.RBackendHandler.channelRead0(RBackendHandler.scala:38) at io.netty.channel.
什麼讓你覺得你使用Scala的2.11？ – zero323
絕對是一個類似zero323的版本錯誤指出的。使用spark-csv_2.10而不是 – eliasah
我沒有使用Scala 2.11 zero323 –