爲什麼MapReduce進度報告不單調增加？

我將MapReduce作業提交給Hadoop並在屏幕上觀看進度報告。對於地圖任務和減少任務，進度報告應單調增加（例如0％，10％，25％，60％，78％，95％和100％）。但作爲事實上報告的進展並不單調遞增：爲什麼MapReduce進度報告不單調增加？

14/01/21 11:05:37 INFO mapred.JobClient: Running job: job_201401201555_0036 
14/01/21 11:05:38 INFO mapred.JobClient: map 0% reduce 0% 
14/01/21 11:06:07 INFO mapred.JobClient: map 11% reduce 0% 
14/01/21 11:06:10 INFO mapred.JobClient: map 0% reduce 0% 
14/01/21 11:06:19 INFO mapred.JobClient: map 9% reduce 0% 
14/01/21 11:06:22 INFO mapred.JobClient: map 22% reduce 0% 
14/01/21 11:06:25 INFO mapred.JobClient: map 31% reduce 0% 
14/01/21 11:06:28 INFO mapred.JobClient: map 39% reduce 0% 
14/01/21 11:06:29 INFO mapred.JobClient: map 53% reduce 0% 
14/01/21 11:06:30 INFO mapred.JobClient: map 57% reduce 0% 
14/01/21 11:06:32 INFO mapred.JobClient: map 50% reduce 0% 
14/01/21 11:06:33 INFO mapred.JobClient: map 55% reduce 0% 
14/01/21 11:06:34 INFO mapred.JobClient: map 43% reduce 0% 
14/01/21 11:06:35 INFO mapred.JobClient: map 48% reduce 0% 
14/01/21 11:06:36 INFO mapred.JobClient: map 40% reduce 0% 
14/01/21 11:06:38 INFO mapred.JobClient: map 30% reduce 0% 
14/01/21 11:06:40 INFO mapred.JobClient: map 40% reduce 0% 
14/01/21 11:06:41 INFO mapred.JobClient: map 49% reduce 0% 
14/01/21 11:06:43 INFO mapred.JobClient: map 57% reduce 0% 
14/01/21 11:06:44 INFO mapred.JobClient: map 70% reduce 0% 
14/01/21 11:06:46 INFO mapred.JobClient: map 73% reduce 0% 
14/01/21 11:06:47 INFO mapred.JobClient: map 82% reduce 0% 
14/01/21 11:06:48 INFO mapred.JobClient: map 93% reduce 0% 
14/01/21 11:06:50 INFO mapred.JobClient: map 94% reduce 0% 
14/01/21 11:06:52 INFO mapred.JobClient: map 95% reduce 0% 
14/01/21 11:06:53 INFO mapred.JobClient: map 96% reduce 0% 
14/01/21 11:06:56 INFO mapred.JobClient: map 98% reduce 0% 
14/01/21 11:06:59 INFO mapred.JobClient: map 99% reduce 0% 
14/01/21 11:07:00 INFO mapred.JobClient: map 100% reduce 0% 
14/01/21 11:07:19 INFO mapred.JobClient: map 100% reduce 4% 
14/01/21 11:07:22 INFO mapred.JobClient: map 100% reduce 8% 
14/01/21 11:07:25 INFO mapred.JobClient: map 100% reduce 66% 
14/01/21 11:07:29 INFO mapred.JobClient: map 100% reduce 67% 
14/01/21 11:07:32 INFO mapred.JobClient: map 100% reduce 68% 
14/01/21 11:07:35 INFO mapred.JobClient: map 100% reduce 69% 
14/01/21 11:07:41 INFO mapred.JobClient: map 100% reduce 70% 
14/01/21 11:07:47 INFO mapred.JobClient: map 100% reduce 71% 
14/01/21 11:07:53 INFO mapred.JobClient: map 100% reduce 72% 
14/01/21 11:07:59 INFO mapred.JobClient: map 100% reduce 73% 
14/01/21 11:08:02 INFO mapred.JobClient: map 100% reduce 100% 
14/01/21 11:08:03 INFO mapred.JobClient: Job complete: job_201401201555_0036

進度已經通過所有的輸入分裂中被處理的比例拆分指示。但是爲什麼進度報告沒有單調增加？

來源

2014-01-21 Tao Xiao

檢查任務跟蹤器和jobtracker的日誌。地圖階段有沒有失敗？如果一臺機器無法執行任務，或者主機無法再到達該任務，則該任務將從另一臺機器重新開始，重新開始執行;所以進展並不單調增加。

來源

2014-01-21 15:13:21 vefthym

我檢查了UI（http：// jobtracker：50030），發現沒有失敗/殺死的任務嘗試，我認爲這意味着在地圖階段沒有失敗。但這項工作的進展並未單調增加。 –

你是否在任何地方使用'conf.setProgress（float progress）'（我不這麼認爲）？如果是的話，那麼也許你把它設置在一個較低的值。你使用多少個節點？一個或多個？是否有任何異常被捕獲（檢查每個tasktracker的日誌）。你檢查了jobtracker的* log *嗎？ – vefthym

我已經看過很多次了。請檢查日誌：應該可能發生錯誤。 – javadba

爲什麼MapReduce進度報告不單調增加？

回答

相關問題