在Hive中加載SparkR數據幀

我需要加載SparkR中創建的DataFrame才能加載到Hive中。在Hive中加載SparkR數據幀

#created a dataframe df_test 
df_test <- createDataFrame(sqlContext, data.frame(mon = c(1,2,3,4,5), year = c(2011,2012,2013,2014,2015))) 

#initialized the Hive context 
>sc <- sparkR.init() 
>hiveContext <- sparkRHive.init(sc) 

#used the saveAsTable fn to save dataframe "df_test" in hive table named "table_hive" 
>saveAsTable(df_test, "table_hive")

16/08/24 23時08分36秒ERROR RBackendHandler：saveAsTable在13 invokeJava（isStatic = FALSE，OBJID $ ID，方法名，...）失敗錯誤：的java.lang .RuntimeException：使用SQLContext創建的表必須是TEMPORARY。改爲使用HiveContext。 at scala.sys.package $ .error（package.scala：27） at org.apache.spark.sql.execution.SparkStrategies $ DDLStrategy $ .apply（SparkStrategies.scala：392） at org.apache.spark。 sql.catalyst.planning.QueryPlanner $$ anonfun $ 1.apply（QueryPlanner.scala：58） at org.apache.spark.sql.catalyst.planning.QueryPlanner $$ anonfun $ 1.apply（QueryPlanner.scala：58） at scala.collection.Iterator $$ anon $ 13.hasNext（Iterator.scala：371） at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan（QueryPlanner.scala：59） at org.apache.spark。 sql.execution.QueryExecution.sparkPlan $ lzycompute（QueryExecution.scala：47） at org.apache.spark.sql.execution.QueryExecution.sparkPlan（QueryExecution.scala：45） at org.apache.spark.sql.e xecution.QueryExecution.executedPlan $ lzycompute（QueryExecution.scala：52）在org.apache.spark.sql.execution.QueryExecution.executedPlan（QueryExecution.scala：52）在org.apache.spark.sql.execution

引發上述錯誤。請幫助。

來源

2016-08-24 Arun Gunalan

在範圍內有HiveContext是不夠的。每個數據幀被綁定到特定SQLContext/SparkSession實例和df_test顯然與不同的上下文創建比hiveContext

讓我們示出了一個例子：

Welcome to 
    ____    __ 
/__/__ ___ _____/ /__ 
    _\ \/ _ \/ _ `/ __/ '_/ 
/___/ .__/\_,_/_/ /_/\_\ version 1.6.1 
    /_/ 


Spark context is available as sc, SQL context is available as sqlContext 
> library(magrittr) 
> createDataFrame(sqlContext, mtcars) %>% saveAsTable("foo") 
16/08/24 20:22:13 ERROR RBackendHandler: saveAsTable on 22 failed 
Error in invokeJava(isStatic = FALSE, objId$id, methodName, ...) : 
    java.lang.RuntimeException: Tables created with SQLContext must be TEMPORARY. Use a HiveContext instead. 
    at scala.sys.package$.error(package.scala:27) 
    at org.apache.spark.sql.execution.SparkStrategies$DDLStrategy$.apply(SparkStrategies.scala:392) 
    at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) 
    at org.apache.spark.sql.catalyst.planning.QueryPlanner$$anonfun$1.apply(QueryPlanner.scala:58) 
    at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:396) 
    at org.apache.spark.sql.catalyst.planning.QueryPlanner.plan(QueryPlanner.scala:59) 
    at org.apache.spark.sql.execution.QueryExecution.sparkPlan$lzycompute(QueryExecution.scala:47) 
    at org.apache.spark.sql.execution.QueryExecution.sparkPlan(QueryExecution.scala:45) 
    at org.apache.spark.sql.execution.QueryExecution.executedPlan$lzycompute(QueryExecution.scala:52) 
    at org.apache.spark.sql.execution.QueryExecution.executedPlan(QueryExecution.scala:52) 
    at org.apache.spark.sql.execu 
> 
> hiveContext <- sparkRHive.init(sc) 
> createDataFrame(hiveContext, mtcars) %>% saveAsTable("foo") 
NULL

來源

2016-08-24 18:24:36 zero323

在Hive中加載SparkR數據幀

回答

相關問題