彈簧數據配置單元與配置單元模板集成

Question

我正在嘗試使用彈簧數據hadoop將配置單元整合到我的應用程序中，並遇到一些問題。我不確定的第一件事是<hdp:hive-server host="some-other-host" port="10001" />這是爲了連接到現有的蜂巢服務器或類似創建一個新的蜂巢服務器，然後能夠連接到它。其次我的配置不會引發任何錯誤，所以它似乎確定，甚至hiveTemplate自動裝配也可以正常工作，但是當我執行查詢時，我似乎沒有得到任何迴應。應用程序在這一點上陷入困​​境。

這裏是配置

<hive-client-factory host="${hive-${env}.server}" port="${hive-${env}.port}" /> 

<hive-template /> 

，這裏是用它

log.debug("before hive query"); 

for(String result : hiveTemplate.query("show tables;")){ 
    log.debug("=> " + result); 
} 

log.debug("after hive query"); 

所有我在日誌輸出看到的是before hive query ..以後不會發生什麼意外怎麼IM。我將不勝感激任何幫助。任何想法，我可能做錯了。

Answer 1

1. 嘗試10000作爲端口號。
2. 通常Thrift服務器部署端口號爲10000.檢查您的安裝是否使用HiveServer2或HiveServer。我能夠使Spring Batch工作流程與HiveServer一起工作，但我還沒有成功使用HiveServer2。
3. 確保您有HiveServer/HiveServer啓動並運行OU運行您的程序

Answer 2

如果我們使用彈簧，請確保您所有的依賴關係是，讓讀寫春季version.After兼容之前使用以下命令的權限。

hadoop fs -mkdir /tmp 
hadoop fs -chmod a+w /tmp 
hadoop fs -mkdir -p /user/hive/warehouse 
hadoop fs -chmod a+w /user/hive/warehouse 

我用以下依賴的pom.xml的版本

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> 
    <modelVersion>4.0.0</modelVersion> 

    <artifactId>spring-hadoop-samples-hive</artifactId> 

    <name>Spring Hadoop Samples - Hive</name> 

    <parent> 
     <groupId>org.springframework.samples</groupId> 
     <artifactId>spring-hadoop-samples</artifactId> 
     <version>1.0.0.BUILD-SNAPSHOT</version> 
     <relativePath>../parent/pom.xml</relativePath> 
    </parent> 

    <properties> 
     <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> 
     <spring.hadoop.version>2.3.0.M1</spring.hadoop.version> 
     <hadoop.version>2.7.1</hadoop.version> 
     <hive.version>1.2.1</hive.version> 
     <!-- <hive.version>2.1.1</hive.version> --> 
    </properties> 

    <dependencies> 

     <dependency> 
      <groupId>org.springframework.data</groupId> 
      <artifactId>spring-data-hadoop</artifactId> 
      <version>${spring.hadoop.version}</version> 
      <exclusions> 
       <exclusion> 
        <groupId>org.springframework</groupId> 
        <artifactId>spring-context-support</artifactId> 
       </exclusion> 
      </exclusions> 
     </dependency> 

     <dependency> 
      <groupId>org.springframework</groupId> 
      <artifactId>spring-jdbc</artifactId> 
      <version>${spring.version}</version> 
     </dependency> 

     <dependency> 
      <groupId>org.springframework</groupId> 
      <artifactId>spring-test</artifactId> 
      <version>${spring.version}</version> 
     </dependency> 

     <dependency> 
      <groupId>org.springframework</groupId> 
      <artifactId>spring-tx</artifactId> 
      <version>${spring.version}</version> 
     </dependency> 

     <dependency> 
      <groupId>org.apache.hadoop</groupId> 
      <artifactId>hadoop-common</artifactId> 
      <version>${hadoop.version}</version> 
      <scope>compile</scope> 
     </dependency> 

     <dependency> 
      <groupId>org.apache.hive</groupId> 
      <artifactId>hive-metastore</artifactId> 
      <version>${hive.version}</version> 
     </dependency> 

     <dependency> 
      <groupId>org.apache.hive</groupId> 
      <artifactId>hive-service</artifactId> 
      <version>${hive.version}</version> 
     </dependency> 

     <dependency> 
      <groupId>org.apache.thrift</groupId> 
      <artifactId>libfb303</artifactId> 
      <version>0.9.1</version> 
     </dependency> 

     <!-- runtime Hive deps start --> 
     <dependency> 
      <groupId>org.apache.hive</groupId> 
      <artifactId>hive-common</artifactId> 
      <version>${hive.version}</version> 
      <scope>runtime</scope> 
     </dependency> 


     <dependency> 
      <groupId>org.apache.hive</groupId> 
      <artifactId>hive-jdbc</artifactId> 
      <version>${hive.version}</version> 
      <scope>runtime</scope> 
     </dependency> 

     <dependency> 
      <groupId>org.apache.hive</groupId> 
      <artifactId>hive-shims</artifactId> 
      <version>${hive.version}</version> 
      <scope>runtime</scope> 
     </dependency> 

     <dependency> 
      <groupId>org.apache.hive</groupId> 
      <artifactId>hive-serde</artifactId> 
      <version>${hive.version}</version> 
      <scope>runtime</scope> 
     </dependency> 

     <dependency> 
      <groupId>org.apache.hive</groupId> 
      <artifactId>hive-contrib</artifactId> 
      <version>${hive.version}</version> 
      <scope>runtime</scope> 
     </dependency> 

     <!-- runtime Hive deps end --> 

     <dependency> 
      <groupId>org.codehaus.groovy</groupId> 
      <artifactId>groovy</artifactId> 
      <version>1.8.5</version> 
      <scope>runtime</scope> 
     </dependency> 

    </dependencies> 

    <repositories> 
     <repository> 
      <id>spring-milestone</id> 
      <url>http://repo.spring.io/libs-milestone</url> 
     </repository> 
    </repositories> 

    <build> 
     <plugins> 
      <plugin> 
       <groupId>org.codehaus.mojo</groupId> 
       <artifactId>appassembler-maven-plugin</artifactId> 
       <version>1.2.2</version> 
       <configuration> 
        <repositoryLayout>flat</repositoryLayout> 
        <configurationSourceDirectory>src/main/config</configurationSourceDirectory> 
        <copyConfigurationDirectory>true</copyConfigurationDirectory> 
        <!-- Extra JVM arguments that will be included in the bin scripts --> 
        <extraJvmArguments>-Xms512m -Xmx1024m -Dhive.version=${hive.version}</extraJvmArguments> 
        <programs> 
         <program> 
          <mainClass>org.springframework.samples.hadoop.hive.HiveApp</mainClass> 
          <name>hiveApp</name> 
         </program> 
         <program> 
          <mainClass>org.springframework.samples.hadoop.hive.HiveClientApp</mainClass> 
          <name>hiveClientApp</name> 
         </program> 
         <program> 
          <mainClass>org.springframework.samples.hadoop.hive.HiveAppWithApacheLogs</mainClass> 
          <name>hiveAppWithApacheLogs</name> 
         </program> 
        </programs> 
       </configuration> 
       <executions> 
        <execution> 
         <id>package</id> 
         <goals> 
          <goal>assemble</goal> 
         </goals> 
        </execution> 
       </executions> 
      </plugin> 
      <plugin> 
       <groupId>org.apache.maven.plugins</groupId> 
       <artifactId>maven-antrun-plugin</artifactId> 
       <executions> 
        <execution> 
         <id>config</id> 
         <phase>package</phase> 
         <configuration> 
          <tasks> 
           <copy todir="target/appassembler/data"> 
            <fileset dir="data"/> 
           </copy> 
          </tasks> 
         </configuration> 
         <goals> 
          <goal>run</goal> 
         </goals> 
        </execution> 
       </executions> 
      </plugin> 
     </plugins> 
    </build> 

</project>