與sparklyr一起使用left_joint時刪除了表格

Question

我正在處理一些我想加入的表格，因爲我使用sparklyr（由於表格大小）和dplyr的left_joint。 這裏是代碼示例：

query.1 <- left_join(pa11, pa12, by = c("CODIGO_HAB_D","ID_EST","ID_ME","ID_PARTE_D","ID_PAR", "ID_REP")) %>% left_join(., pa13, by = c("ID_SINI" = "ID_SINI")) 

query.1 <- left_join(query.1, a14, by = "ID_REP") 
query.1 <-left_join(query.1, a16, by = c("ID_MEJ" = "ID_ME")) 
query.1 <-left_join(query.1, a17, by = c("ID_EST" = "ID_ESTE")) 
query.1 <-left_join(query.1, a18, by = "ID_PARTE_D") 
query.1 <-left_join(query.1, a19, by = "CODI") 
query.1 <-left_join(query.1, a110, by = c("ID_PROF.x" = "ID_PROF")) 
query.1 <-left_join(query.1, a111, by = c("ID_COM.x" = "ID_COM")) 
query.1 <-left_join(query.1, a113, by = c("ID_GRANDES.x" = "ID_GRANDES")) 

當我left_joint 5個第一表，一切都如預期。當我重複這與多個表我得到這個錯誤

Error in as.vector(x, "character") : 
cannot coerce type 'environment' to vector of type 'character' 

然後，當我嘗試看看星火表我在Rstudio得到一個錯誤。

Answer 1

由於某些其他原因，我不時收到這些錯誤。

從我的經驗，增加了Sparklyr內存和執行overheadmemory幫助

config <- spark_config() 
    config$`sparklyr.shell.driver-memory` <- "8G" 
    config$`sparklyr.shell.executor-memory` <- "8G" 
    config$spark.yarn.executor.memoryOverhead <- "2g"