1. 首頁
  2. 問答
  3. Hadoop流式傳輸權限問題

Hadoop流式傳輸權限問題

2013-12-12 85 views
0

在hadoop流式傳輸過程中需要有關調試權限問題的幫助。我嘗試啓動AWK流：Hadoop流式傳輸權限問題

// MKDIR到所有節點

[[email protected] ~]$ for i in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 ; do ssh -f oscbda$i mkdir -p /home/pocal/KS/comverse/awk/; done;

//複製流文件到所有節點

[[email protected] ~]$ for i in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 ; do scp * oscbda$i:/home/pocal/KS/comverse/awk/; done;

//給予777權限的所有文件

[[email protected] ~]$ for i in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 ; do ssh -f oscbda$i chmod 777 /home/pocal/KS/comverse/awk/*; done;

//開始流式傳輸

[[email protected] ~]$ hadoop fs -rm -r /user/pocal/ks/comverse/one/out;\ 
hadoop jar /usr/lib/hadoop-mapreduce/hadoop-streaming-2.0.0-cdh4.3.0.jar \ 
-Dmapreduce.job.reduces=0 \ 
-Dmapred.reduce.tasks=0 \ 
-mapper "awk -f /home/pocal/KS/comverse/awk/data_change.awk -f /home/pocal/KS/comverse/awk/selfcare.awk -f /home/pocal/KS/comverse/awk/selfcare_secondary_mapping.awk -f /home/pocal/KS/comverse/awk/out_sort.awk" \ 
-input "/user/pocal/ks/comverse/one/" \ 
-output "/user/pocal/ks/comverse/one/out"

而得到錯誤...

……….. 
attempt_201311041208_1379_m_000010_2: awk: fatal: can't open source file `/home/pocal/KS/comverse/awk/data_change.awk' for reading (Permission denied) 
13/12/12 09:01:32 INFO mapred.JobClient: Task Id : attempt_201311041208_1379_m_000004_2, Status : FAILED 
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2 
     at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320) 
     at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533) 
     at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) 
     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) 
     at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) 
     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) 
     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) 
     at org.apache.hadoop.mapred.Child$4.run(Child.java:268) 
     at java.security.AccessController.doPrivileged(Native Method) 
     at javax.security.auth.Subject.doAs(Subject.java:396) 
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) 
     at org.apache.hadoop.mapred.Child.main(Child.java:262) 

attempt_201311041208_1379_m_000004_2: awk: fatal: can't open source file `/home/pocal/KS/comverse/awk/data_change.awk' for reading (Permission denied) 
13/12/12 09:01:33 INFO mapred.JobClient: Task Id : attempt_201311041208_1379_m_000003_2, Status : FAILED 
java.lang.RuntimeException: PipeMapRed.waitOutputThreads(): subprocess failed with code 2 
     at org.apache.hadoop.streaming.PipeMapRed.waitOutputThreads(PipeMapRed.java:320) 
     at org.apache.hadoop.streaming.PipeMapRed.mapRedFinished(PipeMapRed.java:533) 
     at org.apache.hadoop.streaming.PipeMapper.close(PipeMapper.java:130) 
     at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57) 
     at org.apache.hadoop.streaming.PipeMapRunner.run(PipeMapRunner.java:34) 
     at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) 
     at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) 
     at org.apache.hadoop.mapred.Child$4.run(Child.java:268) 
     at java.security.AccessController.doPrivileged(Native Method) 
     at javax.security.auth.Subject.doAs(Subject.java:396) 
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) 
     at org.apache.hadoop.mapred.Child.main(Child.java:262) 

attempt_201311041208_1379_m_000003_2: awk: fatal: can't open source file `/home/pocal/KS/comverse/awk/data_change.awk' for reading (Permission denied) 
13/12/12 09:01:37 INFO mapred.JobClient: Job complete: job_201311041208_1379 
13/12/12 09:01:37 INFO mapred.JobClient: Counters: 8 
13/12/12 09:01:37 INFO mapred.JobClient: Job Counters 
13/12/12 09:01:37 INFO mapred.JobClient:  Failed map tasks=1 
13/12/12 09:01:37 INFO mapred.JobClient:  Launched map tasks=52 
13/12/12 09:01:37 INFO mapred.JobClient:  Data-local map tasks=12 
13/12/12 09:01:37 INFO mapred.JobClient:  Rack-local map tasks=40 
13/12/12 09:01:37 INFO mapred.JobClient:  Total time spent by all maps in occupied slots (ms)=348738 
13/12/12 09:01:37 INFO mapred.JobClient:  Total time spent by all reduces in occupied slots (ms)=2952 
13/12/12 09:01:37 INFO mapred.JobClient:  Total time spent by all maps waiting after reserving slots (ms)=0 
13/12/12 09:01:37 INFO mapred.JobClient:  Total time spent by all reduces waiting after reserving slots (ms)=0 
13/12/12 09:01:37 ERROR streaming.StreamJob: Job not Successful! 
Streaming Command Failed!

檢查一臺機器上：

[[email protected] ~]$ ssh oscbda10 ls -l /home/pocal/KS/comverse/awk/data_change.awk 
-rwxrwxrwx 1 pocal pocal 1548 Dec 10 10:05 /home/pocal/KS/comverse/awk/data_change.awk

權限好吧......

是否有人有任何想法？

來源

2013-12-12 Kukuruka

回答

0

是問題父目錄：

[[email protected] .ssh]$ ssh oscbda05 ls -l /home/|grep pocal 
total 24 
drwx------ 5 pocal pocal 4096 Dec 10 09:52 pocal 
[[email protected] .ssh]$ ssh oscbda05 ls -l /home/pocal 
total 4 
drwxrwxrwx 3 pocal pocal 4096 Dec 10 09:52 KS 
[[email protected] .ssh]$ ssh oscbda05 ls -l /home/pocal/KS 
total 4 
drwxrwxrwx 3 pocal pocal 4096 Dec 10 09:52 comverse 
[[email protected] .ssh]$ ssh oscbda05 ls -l /home/pocal/KS/comverse/ 
total 4 
drwxrwxrwx 2 pocal pocal 4096 Dec 10 10:05 awk 
[[email protected] .ssh]$ ssh oscbda05 ls -l /home/pocal/KS/comverse/awk/ 
total 216 
-rwxrwxrwx 1 pocal pocal 4398 Dec 10 10:05 calltype_checker.awk 
-rwxrwxrwx 1 pocal pocal 16173 Dec 10 10:05 ch_rebuild.c 
-rwxrwxrwx 1 pocal pocal 14643 Dec 10 10:05 ch_rebuild.dat 
-rwxrwxrwx 1 pocal pocal 1548 Dec 10 10:05 data_change.awk 
-rwxrwxrwx 1 pocal pocal 4080 Dec 10 10:05 decompress_incomming_data.sh 
-rwxrwxrwx 1 pocal pocal 720 Dec 10 10:05 fms.awk 
-rwxrwxrwx 1 pocal pocal 2502 Dec 10 10:05 load_func 
-rwxrwxrwx 1 pocal pocal 1308 Dec 10 10:05 load_vars 
-rwxrwxrwx 1 pocal pocal 199 Dec 10 10:05 load_vars_dynamic 
-rwxrwxrwx 1 pocal pocal 1358 Dec 10 10:05 out.awk 
-rwxrwxrwx 1 pocal pocal 1296 Dec 10 10:05 out_nosort.awk 
-rwxrwxrwx 1 pocal pocal 1358 Dec 10 10:05 out_sort.awk 
-rwxrwxrwx 1 pocal pocal 70041 Dec 10 10:05 selfcare.awk 
-rwxrwxrwx 1 pocal pocal 54204 Dec 10 10:05 selfcare_secondary_mapping.awk 
-rwxrwxrwx 1 pocal pocal 1847 Dec 10 10:05 stat.awk 

Add permission: 
[[email protected] .ssh]$ for i in 01 02 03 04 05 06 07 08 09 10 11 12 13 14 15 16 17 18 ; do ssh -f oscbda$i chmod +x /home/pocal; done; 
Check: 
[[email protected] .ssh]$ ssh oscbda05 ls -l /home/|grep poc 
drwx--x--x 5 pocal pocal 4096 Dec 10 09:52 pocal

和它的作品！

來源

2013-12-12 09:48:05 Kukuruka

相關問題

 相關問題