負載的RDD成蜂巢

Question

我想一個RDD（K = TABLE_NAME，V =內容）pyspark火花1.6.x版

整體而裝入分區蜂巢表（年，月，日）試圖使用這個SQL查詢的邏輯：

ALTER TABLE db_schema.%FILENAME_WITHOUT_EXTENSION% DROP IF EXISTS PARTITION (year=%YEAR%, month=%MONTH%, day=%DAY%);LOAD DATA INTO TABLE db_schema.%FILENAME_WITHOUT_EXTENSION% PARTITION (year=%YEAR%, month=%MONTH%, day=%DAY%); 

有人請給出一些建議嗎？

Answer 1

spark = SparkSession.builder.enableHiveSupport().getOrCreate() 
df = spark.sparkContext.parallelize([(1, 'cat', '2016-12-20'), (2, 'dog', '2016-12-21')]) 
df = spark.createDataFrame(df, schema=['id', 'val', 'dt']) 
df.write.saveAsTable(name='default.test', format='orc', mode='overwrite', partitionBy='dt') 

使用enableHiveSupport（）和df.write.saveAsTable（）