如何spark-submit由IDE編寫的hiveContext？

我想部署我的代碼，其中包含hiveContext在Spark集羣上。如何spark-submit由IDE編寫的hiveContext？

./spark-submit --class com.dt.sparkSQL.DataFrameToHive --master spark://SparkMaster:7077 /root/Documents/DataFrameToHive.jar 但現在的問題是

17/08/13 10:29:46 INFO hive.metastore: Trying to connect to metastore with URI thrift://SparkMaster:9083 
17/08/13 10:29:46 WARN hive.metastore: Failed to connect to the MetaStore Server... 
17/08/13 10:29:46 INFO hive.metastore: Waiting 1 seconds before next connection attempt. 
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient

當我做了火花外殼

./spark-shell --master spark://SparkMaster:7077

我可以SparkMaster連接：9083 successfully.Here是我的火花/ conf目錄/ hive- site.xml

<configuration> 
<property> 
        <name>hive.metastore.uris</name> 
        <value>thrift://SparkMaster:9083</value> 
        <description>thrift URI for the remote metastore.Used by metastore client to connect to remote metastore. </description> 
</property> 
</configuration>

我的問題是爲什麼它會連接w ith SparkMaster：9083當我進行spark-submit時，SparkMaster：9083有什麼問題？這裏是IDE上的代碼

package com.dt.sparkSQL 

import org.apache.spark.{SparkConf, SparkContext} 
import org.apache.spark.sql.SQLContext 
import org.apache.spark.sql.hive.HiveContext 
object DataFrameToHive { 
    def main(args: Array[String]): Unit = { 
    val conf = new SparkConf() 
    conf.setAppName("DataFrameToHive").setMaster("spark://SparkMaster:7077") 
    val sc = new SparkContext(conf) 
    val hiveContext = new HiveContext(sc) 
    hiveContext.sql("use userdb") 
    hiveContext.sql("DROP TABLE IF EXISTS people") 
    hiveContext.sql("CREATE TABLE IF NOT EXISTS people(name STRING, age INT)ROW FORMAT DELIMITED FIELDS TERMINATED BY '\\t' LINES TERMINATED BY '\\n'") 
    hiveContext.sql("LOAD DATA LOCAL INPATH '/root/Documents/people.txt' INTO TABLE people") 
    hiveContext.sql("use userdb") 
    hiveContext.sql("DROP TABLE IF EXISTS peopleScores") 
    hiveContext.sql("CREATE TABLE IF NOT EXISTS peopleScores(name STRING, score INT) ROW FORMAT DELIMITED FIELDS TERMINATED BY '\\t' LINES TERMINATED BY '\\n'") 
    hiveContext.sql("LOAD DATA LOCAL INPATH '/root/Documents/peopleScore.txt' INTO TABLE peopleScores") 
    val resultDF = hiveContext.sql("select pi.name,pi.age,ps.score " 
     +" from people pi join peopleScores ps on pi.name=ps.name" 
     +" where ps.score>90") 
    hiveContext.sql("drop table if exists peopleResult") 
    resultDF.saveAsTable("peopleResult") 
    val dataframeHive = hiveContext.table("peopleResult") 
    dataframeHive.show() 
    } 
} 
`

來源

2017-08-13 Jason Shu

我已經成功地解決了這個問題。部署hiveContext與普通的jar有所不同。

./spark-submit --class com.dt.sparkSQL.DataFrameToHive --files /usr/local/hive/apache-hive-1.2.1-bin/conf/hive-site.xml --master spark://SparkMaster:7077 /root/Documents/DataFrameToHive.jar

來源

2017-08-14 01:57:41

如何spark-submit由IDE編寫的hiveContext？

回答

相關問題