1. 首頁
  2. 問答
  3. 問題在地圖減少程序

問題在地圖減少程序

2015-06-20 40 views
1

我正在執行一個hadoop Map-Reduce作業，使用Putty進行簡單的字數統計。 我在虛擬機上配置了Hadoop，並驗證了Hadoop的所有組件都使用jps運行。問題在地圖減少程序

當我使用命令 Hadoop的罐子Untitled.jar

我收到錯誤

15/06/20 19:36:48 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 
15/06/20 19:37:09 INFO util.NativeCodeLoader: Loaded the native-hadoop library 
15/06/20 19:37:09 WARN snappy.LoadSnappy: Snappy native library not loaded 
15/06/20 19:37:09 INFO mapred.FileInputFormat: Total input paths to process : 0 
15/06/20 19:37:10 INFO mapred.JobClient: Running job: job_201506201820_0004 
15/06/20 19:37:11 INFO mapred.JobClient: map 0% reduce 0% 
15/06/20 19:37:12 INFO mapred.JobClient: Task Id : attempt_201506201820_0004_m_000001_0, Status : FAILED 
Error initializing attempt_201506201820_0004_m_000001_0: 
ENOENT: No such file or directory 
     at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) 
     at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:701) 
     at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:656) 
     at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514) 
     at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349) 
     at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240) 
     at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205) 
     at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1336) 
     at java.security.AccessController.doPrivileged(Native Method) 
     at javax.security.auth.Subject.doAs(Subject.java:422) 
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) 
     at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1311) 
     at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226) 
     at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603) 
     at java.lang.Thread.run(Thread.java:745) 

15/06/20 19:37:13 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000001_0&filter=stdout 
15/06/20 19:37:13 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000001_0&filter=stderr 
15/06/20 19:37:14 INFO mapred.JobClient: Task Id : attempt_201506201820_0004_m_000001_1, Status : FAILED 
Error initializing attempt_201506201820_0004_m_000001_1: 
ENOENT: No such file or directory 
     at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) 
     at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:701) 
     at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:656) 
     at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514) 
     at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349) 
     at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240) 
     at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205) 
     at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1336) 
     at java.security.AccessController.doPrivileged(Native Method) 
     at javax.security.auth.Subject.doAs(Subject.java:422) 
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) 
     at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1311) 
     at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226) 
     at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603) 
     at java.lang.Thread.run(Thread.java:745) 

15/06/20 19:37:15 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000001_1&filter=stdout 
15/06/20 19:37:15 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000001_1&filter=stderr 
15/06/20 19:37:16 INFO mapred.JobClient: Task Id : attempt_201506201820_0004_m_000001_2, Status : FAILED 
15/06/20 19:37:16 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000001_2&filter=stdout 
15/06/20 19:37:16 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000001_2&filter=stderr 
15/06/20 19:37:17 INFO mapred.JobClient: Task Id : attempt_201506201820_0004_m_000000_0, Status : FAILED 
Error initializing attempt_201506201820_0004_m_000000_0: 
ENOENT: No such file or directory 
     at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) 
     at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:701) 
     at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:656) 
     at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514) 
     at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349) 
     at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240) 
     at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205) 
     at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1336) 
     at java.security.AccessController.doPrivileged(Native Method) 
     at javax.security.auth.Subject.doAs(Subject.java:422) 
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) 
     at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1311) 
     at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226) 
     at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603) 
     at java.lang.Thread.run(Thread.java:745) 

15/06/20 19:37:17 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000000_0&filter=stdout 
15/06/20 19:37:17 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000000_0&filter=stderr 
15/06/20 19:37:18 INFO mapred.JobClient: Task Id : attempt_201506201820_0004_m_000000_1, Status : FAILED 
Error initializing attempt_201506201820_0004_m_000000_1: 
ENOENT: No such file or directory 
     at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) 
     at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:701) 
     at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:656) 
     at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514) 
     at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349) 
     at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240) 
     at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205) 
     at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1336) 
     at java.security.AccessController.doPrivileged(Native Method) 
     at javax.security.auth.Subject.doAs(Subject.java:422) 
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) 
     at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1311) 
     at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226) 
     at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603) 
     at java.lang.Thread.run(Thread.java:745) 

15/06/20 19:37:19 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000000_1&filter=stdout 
15/06/20 19:37:19 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000000_1&filter=stderr 
15/06/20 19:37:20 INFO mapred.JobClient: Task Id : attempt_201506201820_0004_m_000000_2, Status : FAILED 
Error initializing attempt_201506201820_0004_m_000000_2: 
ENOENT: No such file or directory 
     at org.apache.hadoop.io.nativeio.NativeIO.chmod(Native Method) 
     at org.apache.hadoop.fs.FileUtil.execSetPermission(FileUtil.java:701) 
     at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:656) 
     at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:514) 
     at org.apache.hadoop.fs.RawLocalFileSystem.mkdirs(RawLocalFileSystem.java:349) 
     at org.apache.hadoop.mapred.JobLocalizer.initializeJobLogDir(JobLocalizer.java:240) 
     at org.apache.hadoop.mapred.DefaultTaskController.initializeJob(DefaultTaskController.java:205) 
     at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java:1336) 
     at java.security.AccessController.doPrivileged(Native Method) 
     at javax.security.auth.Subject.doAs(Subject.java:422) 
     at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190) 
     at org.apache.hadoop.mapred.TaskTracker.initializeJob(TaskTracker.java:1311) 
     at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:1226) 
     at org.apache.hadoop.mapred.TaskTracker$5.run(TaskTracker.java:2603) 
     at java.lang.Thread.run(Thread.java:745) 

15/06/20 19:37:20 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000000_2&filter=stdout 
15/06/20 19:37:20 WARN mapred.JobClient: Error reading task outputhttp://ankit-VirtualBox:50060/tasklog?plaintext=true&attemptid=attempt_201506201820_0004_m_000000_2&filter=stderr 
15/06/20 19:37:21 INFO mapred.JobClient: Job complete: job_201506201820_0004 
15/06/20 19:37:21 INFO mapred.JobClient: Counters: 4 
15/06/20 19:37:21 INFO mapred.JobClient: Job Counters 
15/06/20 19:37:21 INFO mapred.JobClient:  SLOTS_MILLIS_REDUCES=0 
15/06/20 19:37:21 INFO mapred.JobClient:  Total time spent by all reduces waiting after reserving slots (ms)=0 
15/06/20 19:37:21 INFO mapred.JobClient:  SLOTS_MILLIS_MAPS=0 
15/06/20 19:37:21 INFO mapred.JobClient:  Total time spent by all maps waiting after reserving slots (ms)=0 
15/06/20 19:37:21 INFO mapred.JobClient: Job Failed: JobCleanup Task Failure, Task: task_201506201820_0004_m_000000 
java.io.IOException: Job failed! 
     at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357) 
     at WordCount.main(WordCount.java:32) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:483) 
     at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

我缺少的是執行代碼。

來源

2015-06-20 Ankit Bansal

+0

你可以添加'jps'結果，數據節點和NameNode的日誌嗎？ – chomp

回答

1

你似乎沒有提供任何輸入路徑。看看錯誤：

15/06/20 19時37分09秒INFO mapred.FileInputFormat：總輸入路徑的過程：0

來源

2015-06-21 03:49:01 K246

0

從日誌中，您可以看到問題出現在第一次執行映射時自身，對於映射和減少進程爲0％。接下來是實際的錯誤。 「沒有這樣的文件」。我們處理地圖縮小和文件的唯一時間是輸入和輸出。鑑於我們正處於地圖處理的初始階段，我認爲問題可能在於輸入路徑及其權限。請看看它們。在運行作業之前，輸出目錄也不應該存在。它創造我想。快樂編碼

來源

2015-06-20 15:46:48 Ramzy

相關問題

 相關問題