2013-03-12 70 views
5

我試圖按照此頁面上的說明: http://www.michael-noll.com/tutorials/writing-an-hadoop-mapreduce-program-in-python/運行Python代碼Hadoop的失敗

$bin/hadoop jar contrib/streaming/hadoop-streaming-1.0.4.jar -input /user/root/wordcountpythontxt -output /user/root/wordcountpythontxt-output -mapper /user/root/wordcountpython/mapper.py -reducer /user/root/wordcountpython/reducer.py -file /user/root/mapper.py -file /user/root/reducer.py 

它說

File: /user/root/mapper.py does not exist, or is not readable. 
Streaming Command Fail 

當我通過URL瀏覽:jobdetails.jsp/

我發現很多異常

java.lang.RuntimeException: Error in configuring object 
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) 
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) 
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) 
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432) 
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) 
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at javax.security.auth.Subject.doAs(Subject.java:396) 
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) 
    at org.apache.hadoop.mapred.Child.main(Child.java:249) 
Caused by: java.lang.reflect.InvocationTargetException 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:597) 
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) 
    ... 9 more 
Caused by: java.lang.RuntimeException: Error in configuring object 
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) 
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) 
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) 
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) 
    ... 14 more 
Caused by: java.lang.reflect.InvocationTargetException 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:597) 
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) 
    ... 17 more 
Caused by: java.lang.RuntimeException: configuration exception 
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:230) 
    at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) 
    ... 22 more 
Caused by: java.io.IOException: Cannot run program "/user/root/wordcountpython/mapper.py": error=2, No such file or directory 
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:460) 
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214) 
    ... 23 more 
Caused by: java.io.IOException: error=2, No such file or directory 
    at java.lang.UNIXProcess.forkAndExec(Native Method) 
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:53) 
    at java.lang.ProcessImpl.start(ProcessImpl.java:91) 
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453) 
    ... 24 more 

我不能修復它請幫我運行python pgm。

回答

3

如果你仔細檢查了link的指示,

[email protected]:/usr/local/hadoop$ bin/hadoop jar contrib/streaming/hadoop-*streaming*.jar -file /home/hduser/mapper.py -mapper /home/hduser/mapper.py -file /home/hduser/reducer.py -reducer /home/hduser/reducer.py -input /user/hduser/gutenberg/* -output /user/hduser/gutenberg-output 

有它清楚地表明沒有必要複製mapper.py和reducer.py到HDFS,你可以將文件從兩個鏈接本地文件系統:as/path/to/mapper。我相信你可以避免上述錯誤。

+1

是的,我連接然後從我的本地fs – 2013-03-21 03:52:34

+0

不被接受?有沒有理由恢復? – 2013-04-04 06:12:33

+0

@studhadoop如果解決方案解決了您的問題,您可以接受答案。謝謝 – 2013-09-08 13:44:50

1

看來問題是符合的。

Caused by: java.io.IOException: Cannot run program "/user/root/wordcountpython/mapper.py": error=2, No such file or directory 

請您檢查文件/user/root/wordcountpython/mapper.py是否存在。如果它存在,那麼該文件的權限是什麼。

您正在運行的用戶hadoop有權限執行和讀取此文件?

+0

是我從我的本地文件sys.but鏈接它再次無法正常工作。 – 2013-04-04 04:09:42

3

您可能想要檢查您的#後沒有dos樣式的新行!在mapper.py中行。如果你這樣做,hadoop可能無法找到你的python解釋器,因爲它會看到一個額外的CR。例如。/usr/local/bin/python^M而不是/ usr/local/bin/python其中^ M是CR。在你的mapper和reducer上嘗試dos2unix命令。

+0

我該怎麼做mapper和reducer – 2013-03-21 04:00:11

+0

dos2unix mapper.py – 2013-03-22 11:31:07

+0

......對不起......只要運行'dos2unix mapper.py',假設你已經安裝了dos2unix。然後對於reducer.py也是如此。你可以grep的DOS新行確認它在那裏之前,並在之後:[http://stackoverflow.com/questions/73833/how-do-you-search-for-files-containing-dos-line-endings- CRLF與 - grep的-下-1]。另外,如果這是您的問題,那麼您可以考慮更改編輯器保存文件的方式。例如。如果您使用Eclipse,則默認值爲dos樣式的新行 – 2013-03-22 11:38:46