SparkR將數據幀寫入csv和實木複合地時發生錯誤

將火花數據幀寫入csv和實木複合地板時發生錯誤。我已經嘗試安裝winutil，但仍未解決錯誤。SparkR將數據幀寫入csv和實木複合地時發生錯誤

我的代碼

INVALID_IMEI <- c("","000000000000000") 
    setwd("D:/Revas/Jatim Old") 
    fileList <- list.files() 
    cdrSchema <- structType(structField("date","string"), 
         structField("time","string"), 
         structField("a_number","string"), 
         structField("b_number", "string"), 
         structField("duration","integer"), 
         structField("lac_cid","string"), 
         structField("imei","string")) 
    file <- fileList[1] 
    filePath <- paste0("D:/Revas/Jatim Old/",file) 
    dataset <- read.df(filePath, header="false",source="csv",delimiter="|",schema=cdrSchema) 
    dataset <- filter(dataset, ifelse(dataset$imei %in% INVALID_IMEI,FALSE,TRUE)) 
    dataset <- filter(dataset, ifelse(isnan(dataset$imei),FALSE,TRUE)) 
    dataset <- filter(dataset, ifelse(isNull(dataset$imei),FALSE,TRUE))

要導出數據框，我嘗試下面的代碼

write.df(dataset, "D:/spark/dataset",mode="overwrite") 
    write.parquet(dataset, "D:/spark/dataset",mode="overwrite")

，我得到以下錯誤

Error: Error in save : org.apache.spark.SparkException: Job aborted. 
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply$mcV$sp(FileFormatWriter.scala:215) 
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:173) 
at org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1.apply(FileFormatWriter.scala:173) 
at org.apache.spark.sql.execution.SQLExecution$.withNewExecutionId(SQLExecution.scala:65) 
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.write(FileFormatWriter.scala:173) 
at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand.run(InsertIntoHadoopFsRelationCommand.scala:145) 
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult$lzycompute(commands.scala:58) 
at org.apache.spark.sql.execution.command.ExecutedCommandExec.sideEffectResult(commands.scala:56) 
at org.apache.spark.sql.execution.comma

來源

2017-09-22 Wahyu Sejati Roso

我已經找到了可能的原因。這個問題似乎在於winutil版本，以前使用2.6。將其改爲2.8可解決問題

來源

2017-09-23 03:07:43

SparkR將數據幀寫入csv和實木複合地時發生錯誤

回答

相關問題