2013-01-02 83 views
1

執行MapReduce程序時出現以下錯誤。 我已將所有jar放在hadoop/lib目錄中,並且還提到了-libjars中的jar。hcatalog with mapreduce

這是我執行CMD:

$HADOOP_HOME/bin/hadoop --config $HADOOP_HOME/conf jar /home/shash/distinct.jar HwordCount -libjars $LIB_JARS WordCount HWordCount2 

java.lang.RuntimeException: java.lang.ClassNotFoundException: 
org.apache.hcatalog.mapreduce.HCatOutputFormat at 
org.apache.hadoop.conf.Configuration.getClass(Configuration.java:996) at 
org.apache.hadoop.mapreduce.JobContext.getOutputFormatClass(JobContext.java:248) at org.apache.hadoop.mapred.Task.initialize(Task.java:501) at 
org.apache.hadoop.mapred.MapTask.run(MapTask.java:306) at org.apache.hadoop.mapred.Child$4.run(Child.java:270) at 
java.security.AccessController.doPrivileged(Native Method) at 
javax.security.auth.Subject.doAs(Subject.java:415) at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1127) at 
org.apache.hadoop.mapred.Child.main(Child.java:264) Caused by: java.lang.ClassNotFoundException: org.apache.hcatalog.mapreduce.HCatOutputFormat 
at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at 
java.net.URLClassLoader$1.run(URLClassLoader.java:355) at 
java.security.AccessController.doPrivileged(Native Method) at 
java.net.URLClassLoader.findClass(URLClassLoader.java:354) at 
java.lang.ClassLoader.loadClass(ClassLoader.java:423) at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at 
java.lang.ClassLoader.loadClass(ClassLoader.java:356) at 
java.lang.Class.forName0(Native Method) at 
java.lang.Class.forName(Class.java:264) at 
org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:943) 
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:994) ... 
8 more 

回答

2

確保LIB_JARS是一個逗號分隔的列表(而不是冒號分隔狀CLASSPATH)

0

應用到CDH 5.0.x版CDH 5.1.x CDH 5.2.x CDH 5.3.x Sqoop

原因Sqoop無法提取HCatalog庫,因爲Cloudera Manager未設置HIVE_HOME環境。需要手動設置 。

這個問題跟蹤下面JIRA: https://issues.apache.org/jira/browse/SQOOP-2145

這個JIRA的修復,因爲5.4.0版本已被列入CDH。

解決方法:適用於低於5.4.0的CDH版本。

執行下面在外殼命令調用Sqoop命令或將其添加到之前/etc/sqoop/conf/sqoop-env.sh(創建一個,如果沒有已經存在的話):

export HIVE_HOME=/opt/cloudera/parcels/CDH/lib/hive (for parcel installation) 
export HIVE_HOME=/usr/lib/hive (for package installation)