做使用couchbase插件1.1 Sqoop進口,進口發出警告Sqoop進口與失敗:最大塊的位置超出了分裂
警告:
「最大塊位置拆分超標」。
錯誤:
錯誤:com.cloudera.sqoop.lib.LargeObjectLoader(Lorg /阿帕奇/ hadoop的/ CONF /配置; Lorg /阿帕奇/ hadoop的/ FS /路徑;)V 14/11/03 2點38分21秒INFO mapreduce.Job:地圖100%減少0%
Sqoop 1.4.5-cdh5.2.0 的Hadoop 2.3.0-cdh5.1.0 Couchbase hadoop的插件1.1
的警告和錯誤似乎是相關的。因此,我進行了如下修改:
我試圖從命令行改變最大分割大小的值: * -D mapred.max.split.size = 1048576
-D mapreduce.job.max.split .locations = 100 *
日誌:
[根@ cinthrweb02冰雹]#sqoop進口-D mapreduce.input.fileinputformat.split.maxsize = 1048576 -D mapreduce.job.max.split.locations = 256 - 連接url -P - 表DUMP
警告:/usr/lib/sqoop/../accumulo不存在! Accumulo進口將失敗。 請將$ ACCUMULO_HOME設置爲您的Accumulo安裝的根目錄。 14/11/03 02:57:16信息sqoop.Sqoop:運行Sqoop版本:1.4.5-cdh5.2.0 輸入密碼: 14/11/03 02:57:20 INFO tool.CodeGenTool:開始代碼生成 14/11/03 02:57:20信息orm.CompilationManager:HADOOP_MAPRED_HOME是/ usr/lib/hadoop-mapreduce 注意:/tmp/sqoop-root/compile/4f331caf65f98ef369053846ba619981/DUMP.java使用或覆蓋棄用的API。 注意:使用-Xlint:deprecation重新編譯以獲取詳細信息。 14/11/03 02:57:21信息orm.CompilationManager:編寫jar文件:/tmp/sqoop-root/compile/4f331caf65f98ef369053846ba619981/DUMP.jar 14/11/03 02:57:21信息mapreduce.ImportJobBase:開始導入DUMP 14/11/03 02:57:21 INFO Configuration.deprecation:mapred.job.tracker已棄用。相反,使用mapreduce.jobtracker.address 14/11/03 02:57:22 INFO Configuration.deprecation:mapred.jar已棄用。相反,使用mapreduce.job.jar 14/11/03 02:57:22信息Configuration.deprecation:mapred.map.tasks已棄用。相反,使用mapreduce.job.maps 14/11/03 02:57:22 WARN util.Jars:沒有這樣的類couchbase不使用可用的jdbc驅動程序。 14/11/03 02:57:22信息client.RMProxy:連接到ResourceManager在cinthrweb01.sling.com/10.84.80.157:8032 14/11/03 02:57:24信息Configuration.deprecation:mapreduce.map .class已棄用。相反,使用mapreduce.job.map.class 14/11/03 02:57:24 INFO client.CouchbaseConnection:添加了{QA sa =/10.84.80.140:11210,#Rops = 0,#Wops = 0,#iq = 0,topRop = null,topWop = null,toWrite = 0,interested = 0}連接隊列 14/11/03 02:57:24 INFO client.CouchbaseConnection:Added {QA sa =/10.84.80.141:11210, #Rops = 0,#Wops = 0,#iq = 0,topRop = null,topWop = null,toWrite = 0,interested = 0}連接隊列 14/11/03 02:57:24 INFO client。CouchbaseConnection:添加{QA sa =/10.84.80.142:11210,#Rops = 0,#Wops = 0,#iq = 0,topRop = null,topWop = null,toWrite = 0,interested = 0}連接隊列 14/11/03 02:57:24 INFO client.CouchbaseConnection:連接狀態已更改爲[email protected] 14/11/03 02:57:24 INFO client.CouchbaseConnection:連接狀態已更改爲sun.nio .ch.SelectionKeyImpl @ 6d1576d7 14/11/03 02:57:24 INFO client.CouchbaseConnection:連接狀態已更改爲[email protected] 14/11/03 02:57:24 INFO client.CouchbaseConnection :關閉Couchbase客戶端 14/11/03 02:57:24信息mapreduce.JobSubmitter:分割數量:4 14/11/03 02:57:24信息mapreduce.JobSubmitter:提交作業的標記:job_1409339839617_0107 14/11/03 02:57:24 INFO impl.YarnClie ntImpl:已提交應用程序application_1409339839617_0107 14/11/03 02:57:24信息mapreduce.Job:跟蹤作業的網址:http://cinthrweb01.sling.com:8088/proxy/application_1409339839617_0107/ 14/11/03 02:57:24信息mapreduce.Job:正在運行的作業:job_1409339839617_0107 14/11/03 02:57:31信息mapreduce.Job:Job job_1409339839617_0107以超級模式運行:false 14/11/03 02:57:31信息mapreduce.Job:map 0%reduce 0% 14/11/03 02:57:36信息mapreduce.Job:Task Id:attempt_1409339839617_0107_m_000000_0,Status:FAILED 錯誤:com.cloudera.sqoop.lib.LargeObjectLoader。(Lorg/apache/hadoop/conf/Configuration; Lorg/apache/hadoop/fs/Path;)V 14/11/03 02:57:37信息mapreduce.Job:任務ID:attempt_1409339839617_0107_m_000002_0,狀態:失敗 錯誤:com.cloudera.sqoop.lib.LargeObjectLoader。 (Lorg/apache/hadoop/conf/Configuration; Lorg/apache/hadoop/fs/Path;)V 14/11/03 02:57:37信息mapreduce.Job:任務ID:attempt_1409339839617_0107_m_000003_0,狀態:失敗 錯誤: com.cloudera.sqoop.lib.LargeObjectLoader。(Lorg/apache/hadoop/conf/Configuration; Lorg/apache/hadoop/fs/Path;)V 由ApplicationMaster終止的容器。 容器在請求時死亡。退出代碼是143 集裝箱退出,非零退出代碼143
14/11/03 2時57分37秒INFO mapreduce.Job:任務標識:attempt_1409339839617_0107_m_000001_0,狀態:失敗 錯誤:com.cloudera.sqoop .lib.LargeObjectLoader。(Lorg/apache/hadoop/conf/Configuration; Lorg/apache/hadoop/fs/Path;)V 14/11/03 02:57:40信息mapreduce.Job:Task Id:attempt_1409339839617_0107_m_000000_1,Status :失敗 錯誤:com.cloudera.sqoop.lib.LargeObjectLoader。(Lorg/apache/hadoop/conf/Configuration; Lorg/apache/hadoop/fs/Path;)V 14/11/03 02:57:42 INFO mapreduce.Job:任務ID:attempt_1409339839617_0107_m_000003_1,狀態:失敗 錯誤:com.cloudera.sqoop.lib.LargeObjectLoader。(Lorg/apache/hadoop/conf/Config uration; Lorg /阿帕奇/ hadoop的/ FS /路徑;)V 14/11/03 2點57分42秒INFO mapreduce.Job:任務標識:attempt_1409339839617_0107_m_000002_1,狀態:FAILED 錯誤:com.cloudera.sqoop.lib.LargeObjectLoader 。(Lorg/apache/hadoop/conf/Configuration; Lorg/apache/hadoop/fs/Path;)V 14/11/03 02:57:42信息mapreduce.Job:Task Id:attempt_1409339839617_0107_m_000001_1,Status:FAILED Error :com.cloudera.sqoop.lib.LargeObjectLoader。(Lorg/apache/hadoop/conf/Configuration; Lorg/apache/hadoop/fs/Path;)V 14/11/03 02:57:46信息mapreduce.Job: Task Id:attempt_1409339839617_0107_m_000000_2,Status:FAILED 錯誤:com.cloudera.sqoop.lib.LargeObjectLoader。(Lorg/apache/hadoop/conf/Configuration; Lorg/apache/hadoop/fs/Path;)V 14/11/03 02:57:46信息mapreduce.Job:任務ID:attempt_1409339839617_0107_m_0 00002_2,狀態:失敗 錯誤:com.cloudera.sqoop.lib.LargeObjectLoader。(Lorg/apache/hadoop/conf/Configuration; Lorg/apache/hadoop/fs/Path;)V 14/11/03 02:57 :47信息mapreduce.Job:任務ID:attempt_1409339839617_0107_m_000003_2,狀態:失敗 錯誤:com.cloudera.sqoop.lib.LargeObjectLoader。(Lorg/apache/hadoop/conf/Configuration; Lorg/apache/hadoop/fs/Path;)V 14/11/03 02:57:48信息mapreduce.Job:Task Id:attempt_1409339839617_0107_m_000001_2,Status:FAILED 錯誤: com.cloudera.sqoop.lib.LargeObjectLoader。(Lorg/apache/hadoop/conf/Configuration; Lorg/apache/hadoop/fs/Path;)V 14/11/03 02:57:52信息mapreduce.Job:map 100%降至0% 14/11/03 2點57分52秒INFO mapreduce.Job:工作job_1409339839617_0107失敗狀態失敗的原因:任務失敗,task_1409339839617_0107_m_000000 作業失敗的任務失敗。失敗地圖:1失敗減少:0
沒有運氣呢。
請幫忙。
我也試圖改變Hadoop的塊大小。 – minion 2014-11-03 09:41:53