掃描hbase時，地圖任務卡住

我正在運行Map reduce，它會掃描hbase並收集所需的數據。 hadoop有7TB的數據，它還有10TB的空閒空間。 HBase擁有約2億條記錄。在完成99.44％掃描HBase後，5個map任務保持運行狀態，完成率爲0.0％。這些過程根本沒有嘗試過。它永遠呆在那個狀態。因此，我的MR根本沒有完成。當從我的本地機器運行這個MR指向一個小桌子時，它運行良好。任何想法PLZ爲什麼會卡住，如何解決它？在此先感謝掃描hbase時，地圖任務卡住

例外，我在工作日誌跟蹤發現

2012-03-27 13:27:56,117 INFO org.apache.hadoop.mapred.JobInProgress: Choosing rack-local task task_201203271236_0001_m_000955 
2012-03-27 13:27:56,699 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201203271236_0001_m_000865_0' has completed task_201203271236_0001_m_000865 successfully. 
2012-03-27 13:27:56,699 INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:910 completedMapsInputSize:910 completedMapsOutputSize:60738755 
2012-03-27 13:27:56,700 INFO org.apache.hadoop.mapred.JobInProgress: Choosing a non-local task task_201203271236_0001_m_000707 
2012-03-27 13:27:56,700 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201203271236_0001_m_000707_0' to tip task_201203271236_0001_m_000707, for tracker 'tracker_nafhdi12809mwh.io.askjeeves.info:localhost.localdomain/127.0.0.1:55120' 
2012-03-27 13:27:56,704 INFO org.apache.hadoop.ipc.Server: IPC Server handler 4 on 9001, call heartbeat([email protected], false, false, true, 1022) from 10.76.99.112:49072: error: java.io.IOException: java.lang.NullPointerException 
java.io.IOException: java.lang.NullPointerException 
    at org.apache.hadoop.mapred.JobInProgress.getMatchingLevelForNodes(JobInProgress.java:1235) 
    at org.apache.hadoop.mapred.JobInProgress.addRunningTaskToTIP(JobInProgress.java:1317) 
    at org.apache.hadoop.mapred.JobInProgress.obtainNewMapTask(JobInProgress.java:957) 
    at org.apache.hadoop.mapred.DefaultTaskSelector.obtainNewMapTask(DefaultTaskSelector.java:63) 
    at org.apache.hadoop.mapred.FairScheduler.assignTasks(FairScheduler.java:265) 
    at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:2586) 
    at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:597) 
    at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:508) 
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:961) 
    at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:957) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at javax.security.auth.Subject.doAs(Subject.java:396) 
    at org.apache.hadoop.ipc.Server$Handler.run(Server.java:955) 
2012-03-27 13:27:56,713 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201203271236_0001_m_000865_1' to tip task_201203271236_0001_m_000865, for tracker 'tracker_nafhdi12809mwh.io.askjeeves.info:localhost.localdomain/127.0.0.1:55120' 
2012-03-27 13:27:56,714 INFO org.apache.hadoop.mapred.JobInProgress: Choosing data-local task task_201203271236_0001_m_000865 
2012-03-27 13:27:56,714 INFO org.apache.hadoop.mapred.JobTracker: Removed completed task 'attempt_201203271236_0001_m_000865_0' from 'tracker_nafhdi12809mwh.io.askjeeves.info:localhost.localdomain/127.0.0.1:55120' 
2012-03-27 13:27:56,841 INFO org.apache.hadoop.mapred.JobInProgress: Task 'attempt_201203271236_0001_m_000877_0' has completed task_201203271236_0001_m_000877 successfully. 
2012-03-27 13:27:56,841 INFO org.apache.hadoop.mapred.ResourceEstimator: completedMapsUpdates:911 completedMapsInputSize:911 completedMapsOutputSize:60800404 
2012-03-27 13:27:56,842 INFO org.apache.hadoop.mapred.JobInProgress: Choosing a non-local task task_201203271236_0001_m_000753 
2012-03-27 13:27:56,842 INFO org.apache.hadoop.mapred.JobTracker: Adding task 'attempt_201203271236_0001_m_000753_0' to tip task_201203271236_0001_m_000753, for tracker 'tracker_nafhdi12711mwh.io.askjeeves.info:localhost.localdomain/127.0.0.1:46050' 
2012-03-27 13:27:56,842 INFO org.apache.hadoop.ipc.Server: IPC Server handler 3 on 9001, call heartbeat([email protected], false, false, true, 1023) from 10.76.99.79:37228: error: java.io.IOException: java.lang.NullPointerException 
java.io.IOException: java.lang.NullPointerException

來源

2012-03-27 sriram

終於想通了什麼問題。問題是在topology.data文件中，當從windows機器移動文件時，windows字符被附加到行尾。正因爲如此，hadoop代碼的getMatchingLevelForNodes（）方法中節點變爲空，因此發生空指針異常。謝謝你們的支持:)

來源

2012-03-27 23:39:00 sriram

，你可以嘗試運行hbase hbck，看看是否有你的表

來源

2012-03-27 05:41:20

@Aron：Ran那。我得到狀態爲「OK」和「檢測到0個不一致」。 – sriram 2012-03-27 18:19:41

然後很可能它與您正在運行的實際代碼有關，而不是數據的問題 – 2012-03-27 18:40:05

代碼在0-1,1-2，...... E-F的行範圍內掃描HBase。我得到1-2的輸出。 2-3的問題開始。因此，即使它沒有產生1-2的輸出，我也可以確認代碼是錯誤的... – sriram 2012-03-27 19:22:59

掃描hbase時，地圖任務卡住

回答

相關問題