hadoop 2.2，在Windows 7上失敗的Word Count示例

Question

執行hadoop jar命令運行默認字數統計程序後，我收到了跟蹤stderr。

java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService and Could not find the 
main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster 

我相信我沒有明確設置Hadoop的類路徑。 正如我設定的環境。變量如下： 或我在這裏錯過了什麼？

%HADOOP_HOME% =c:\hadoop 
HADOOP_COMMON_HOME =%HADOOP_HOME% 
HADOOP_CONF_DIR=%HADOOP_HOME%\etc\hadoop 
HADOOP_HDFS_HOME =%HADOOP_HOME% 
HADOOP_MAPRED_HOME =%HADOOP_HOME% 

我執行的命令是

hadoop jar c:/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar wordcount /input /output 

The output @ sdk command prompt is 

13/12/20 16:26:00 INFO mapreduce.Job: Job job_1387536911324_0001 failed with sta 
te FAILED due to: Application application_1387536911324_0001 failed 2 times due 
to AM Container for appattempt_1387536911324_0001_000002 exited with exitCode: 
1 due to: Exception from container-launch: 
org.apache.hadoop.util.Shell$ExitCodeException: 
     at org.apache.hadoop.util.Shell.runCommand(Shell.java:464) 
     at org.apache.hadoop.util.Shell.run(Shell.java:379) 
     at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java: 
589) 
     at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.la 
unchContainer(DefaultContainerExecutor.java:195) 
     at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.C 
ontainerLaunch.call(ContainerLaunch.java:283) 
     at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.C 
ontainerLaunch.call(ContainerLaunch.java:79) 
     at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) 
     at java.util.concurrent.FutureTask.run(FutureTask.java:138) 
     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExec 
utor.java:895) 
     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor 
.java:918) 
     at java.lang.Thread.run(Thread.java:662) 

     1 file(s) moved. 

ANd the stderr file is 

java.lang.NoClassDefFoundError: org/apache/hadoop/service/CompositeService 
    at java.lang.ClassLoader.defineClass1(Native Method) 
    at java.lang.ClassLoader.defineClassCond(ClassLoader.java:631) 
    at java.lang.ClassLoader.defineClass(ClassLoader.java:615) 
    at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:141) 
    at java.net.URLClassLoader.defineClass(URLClassLoader.java:283) 
    at java.net.URLClassLoader.access$000(URLClassLoader.java:58) 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:197) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306) 
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247) 
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.service.CompositeService 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:202) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:190) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:306) 
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:247) 
    ... 12 more 
Could not find the main class: org.apache.hadoop.mapreduce.v2.app.MRAppMaster. Program will exit. 
Exception in thread "main" 

Answer 1

麻煩的是與yarn.application.classpath財產。默認值使用Linux風格的環境變量引用，如$ HADOOP_HOME而不是像％HADOOP_HOME％這樣的Windows風格引用，因此您必須通過添加屬性來覆蓋％HADOOP_HOME％\ etc \ hadoop \ yarn-site.xml中的默認值這個：

 
<property> 
    <description>CLASSPATH for YARN applications. A comma-separated list of CLASSPATH entries</description> 
    <name>yarn.application.classpath</name> 
    <value> 
     %HADOOP_HOME%\etc\hadoop, 
     %HADOOP_HOME%\share\hadoop\common\*, 
     %HADOOP_HOME%\share\hadoop\common\lib\*, 
     %HADOOP_HOME%\share\hadoop\hdfs\*, 
     %HADOOP_HOME%\share\hadoop\hdfs\lib\*, 
     %HADOOP_HOME%\share\hadoop\mapreduce\*, 
     %HADOOP_HOME%\share\hadoop\mapreduce\lib\*, 
     %HADOOP_HOME%\share\hadoop\yarn\*, 
     %HADOOP_HOME%\share\hadoop\yarn\lib\* 
    </value> 
</property> 

Answer 2

我正在運行Linux和我有完全相同的問題。它是由添加該紗線-site.xml中解決：

 <property> 
       <description>Classpath for typical applications.</description> 
       <name>yarn.application.classpath</name> 
       <value> 
         $HADOOP_CONF_DIR, 
         $HADOOP_COMMON_HOME/share/hadoop/common/*, 
         $HADOOP_COMMON_HOME/share/hadoop/common/lib/*, 
         $HADOOP_HDFS_HOME/share/hadoop/hdfs/*, 
         $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*, 
         $HADOOP_YARN_HOME/share/hadoop/yarn/*, 
         $HADOOP_YARN_HOME/share/hadoop/yarn/lib/* 
       </value> 
     </property> 

Answer 3

我發現的Hadoop不能解決$ HADOOP_HOME和$ YARN_HOME環境變量，同時遍歷YarnConfiguration屬性。運行在您的紗線客戶端以下將打印未解決的配置一樣，

$ HADOOP_HOME/，$ HADOOP_HOME/lib目錄/

YarnConfiguration conf = new YarnConfiguration() 
    for (String c : conf.getStrings(
       YarnConfiguration.YARN_APPLICATION_CLASSPATH, 
       YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH)) { 
     System.out.println(c); 
    } 

所以，如果你提供的完整路徑yarn.application.classpath屬性，NoClassDefFoundError問題得到解決。

<property> 
    <description>CLASSPATH for YARN applications. A comma-separated list of CLASSPATH entries</description> 
    <name>yarn.application.classpath</name> 
    <value> 
     /etc/hadoop/conf, 
     /usr/lib/hadoop/*, 
     /usr/lib/hadoop/lib/*, 
     /usr/lib/hadoop-hdfs/*, 
     /usr/lib/hadoop-hdfs/lib/*, 
     /usr/lib/hadoop-mapreduce/*, 
     /usr/lib/hadoop-mapreduce/lib/*, 
     /usr/lib/hadoop-yarn/*, 
     /usr/lib/hadoop-yarn/lib/* 
    </value> 
    </property>