2011-03-15 63 views
5

我正在嘗試運行Ruby Hadoop流式處理程序,該程序在「Ruby的權威指南」中給出。爲什麼我的流式傳輸命令在MapReduce基礎程序中失敗?

這是我所使用的命令:

​​

的文件路徑是正確的。當我運行命令,我得到這樣的錯誤:

packageJobJar: [/var/lib/hadoop-0.20/cache/cloudera/hadoop-unjar7505856816337735719/] [] /tmp/streamjob5440613434295562761.jar tmpDir=null 
11/03/14 22:58:56 INFO mapred.FileInputFormat: Total input paths to process : 1 
11/03/14 22:58:56 INFO streaming.StreamJob: getLocalDirs(): [/var/lib/hadoop-0.20/cache/cloudera/mapred/local] 
11/03/14 22:58:56 INFO streaming.StreamJob: Running job: job_201103140235_0015 
11/03/14 22:58:56 INFO streaming.StreamJob: To kill this job, run: 
11/03/14 22:58:56 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=localhost:8021 -kill job_201103140235_0015 
11/03/14 22:58:56 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp... 
11/03/14 22:58:57 INFO streaming.StreamJob: map 0% reduce 0% 
11/03/14 22:59:54 INFO streaming.StreamJob: map 100% reduce 100% 
11/03/14 22:59:54 INFO streaming.StreamJob: To kill this job, run: 
11/03/14 22:59:54 INFO streaming.StreamJob: /usr/lib/hadoop-0.20/bin/hadoop job -Dmapred.job.tracker=localhost:8021 -kill job_201103140235_0015 
11/03/14 22:59:54 INFO streaming.StreamJob: Tracking URL: http://localhost:50030/jobdetails.jsp... 
11/03/14 22:59:54 ERROR streaming.StreamJob: Job not Successful! 
11/03/14 22:59:54 INFO streaming.StreamJob: killJob... 
Streaming Command Failed! 

我看着地圖上的日誌文件,這就是錯誤:

java.lang.RuntimeException: Error in configuring object 
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) 
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) 
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) 
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:379) 
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:317) 
at org.apache.hadoop.mapred.Child$4.run(Child.java:217) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:396) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1063) 
at org.apache.hadoop.mapred.Child.main(Child.java:211) 
Caused by: java.lang.reflect.InvocationTargetException 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
at java.lang.reflect.Method.invoke(Method.java:597) 
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) 
... 9 more 
Caused by: java.lang.RuntimeException: Error in configuring object 
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) 
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) 
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) 
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) 
... 14 more 
Caused by: java.lang.reflect.InvocationTargetException 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
at java.lang.reflect.Method.invoke(Method.java:597) 
at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) 
... 17 more 
Caused by: java.lang.RuntimeException: configuration exception 
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:230) 
at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) 
... 22 more 
Caused by: java.io.IOException: Cannot run program "/home/cloudera/projects/max_temp/map.rb": java.io.IOException: error=13, Permission denied 
at java.lang.ProcessBuilder.start(ProcessBuilder.java:460) 
at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214) 
... 23 more 
Caused by: java.io.IOException: java.io.IOException: error=13, Permission denied 
at java.lang.UNIXProcess.(UNIXProcess.java:148) 
at java.lang.ProcessImpl.start(ProcessImpl.java:65) 
at java.lang.ProcessBuilder.start(ProcessBuilder.java:453) 
... 24 more 

回答

4

感謝列弗Khomich您的回覆。我已經通過在命令提檔參數修復了這個問題:

hadoop jar /usr/lib/hadoop-0.20/contrib/streaming/hadoop-streaming-0.20.2+737.jar \ 
-input input/temperature -output output -mapper /home/cloudera/projects/max_temp/map.rb \ 
-reducer /home/cloudera/projects/max_temp/reduce.rb \ 
-file /home/cloudera/projects/words/map.rb \ 
-file /home/cloudera/projects/words/reduce.rb 
0

我有同樣的錯誤你,並通過複製「map.rb」和「reduce.rb」文件到/ tmp /固定的問題,然後用parms運行它作爲​​

相關問題