在hadoop中找不到shell腳本

我是hadoop和hadoop流媒體的新手，所以這個錯誤可能很明顯，我錯過了。在hadoop中找不到shell腳本

我運行內聯awk映射器命令，它工作正常。

hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-1.0.3.jar -input input -output output -mapper "/usr/bin/awk -F'\t' '\$1==\"and\"'" -reducer NONE

但是，當我把awk命令放到一個文件並運行它時。我在集羣中的所有機器上都收到了Java IOException。

hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-1.0.3.jar -input one_to_one/part* -output 4 -mapper test.sh -reducer NONE -file test.sh

test.sh

/usr/bin/awk -F '\t' '(NR == 1) {print NR,OFS,$0;}'

例外：

java.lang.RuntimeException: Error in configuring object 
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) 
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) 
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) 
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:432) 
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:372) 
    at org.apache.hadoop.mapred.Child$4.run(Child.java:255) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at javax.security.auth.Subject.doAs(Subject.java:396) 
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1121) 
    at org.apache.hadoop.mapred.Child.main(Child.java:249) 
Caused by: java.lang.reflect.InvocationTargetException 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:597) 
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) 
    ... 9 more 
Caused by: java.lang.RuntimeException: Error in configuring object 
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) 
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) 
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) 
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34) 
    ... 14 more 
Caused by: java.lang.reflect.InvocationTargetException 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:597) 
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) 
    ... 17 more 
Caused by: java.lang.RuntimeException: configuration exception 
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:230) 
    at org.apache.hadoop.streaming.PipeMapper.configure(PipeMapper.java:66) 
    ... 22 more 
Caused by: java.io.IOException: Cannot run program "/Users/hadoop/tmp/taskTracker/hadoop/jobcache/job_201207051227_0012/attempt_201207051227_0012_m_000000_0/work/./test.sh": error=2, No such file or directory 
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:460) 
    at org.apache.hadoop.streaming.PipeMapRed.configure(PipeMapRed.java:214) 
    ... 23 more 
Caused by: java.io.IOException: error=2, No such file or directory 
    at java.lang.UNIXProcess.forkAndExec(Native Method) 
    at java.lang.UNIXProcess.<init>(UNIXProcess.java:53) 
    at java.lang.ProcessImpl.start(ProcessImpl.java:91) 
    at java.lang.ProcessBuilder.start(ProcessBuilder.java:453) 
    ... 24 more

來源

2012-07-06 PokerIncome.com

嘗試：

添加散列/她爆炸到你test.sh：

#!/bin/bash
修改-file到-files

hadoop jar $HADOOP_HOME/contrib/streaming/hadoop-streaming-1.0.3.jar -input one_to_one/part* -output 4 -mapper test.sh -reducer NONE -files test.sh

來源

2012-07-06 00:59:27

在hadoop中找不到shell腳本

回答

相關問題