使用Sqoop執行Oozie工作流時出錯

我已經編寫了一個Sqoop導入腳本來將數據從teradata導入配置單元。它的正常工作，當我從命令行中運行，但是當我把它放在shell腳本，並試圖通過Oozie的工作流程來執行，我得到下面的錯誤使用Sqoop執行Oozie工作流時出錯

`[[email protected]] **Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.ShellMain],** exit code [1] 
2015-04-02 08:50:55,440 INFO ActionEndXCommand:539 - USER[qjdht93] GROUP[-TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[[email protected]] end executor for wf action 0000069-150114201015959-oozie-oozi-W with wf job 0000069-150114201015959-oozie-oozi-W 
2015-04-02 08:50:55,459 INFO ActionEndXCommand:539 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[[email protected]] ERROR is considered as FAILED for SLA 
2015-04-02 08:50:55,505 INFO ActionStartXCommand:539 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[[email protected]] Start action [[email protected]] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10] 
2015-04-02 08:50:55,505 WARN ActionStartXCommand:542 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[[email protected]] [***[email protected]***]Action status=DONE 
2015-04-02 08:50:55,505 WARN ActionStartXCommand:542 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[[email protected]] [***[email protected]***]Action updated in DB! 
2015-04-02 08:50:55,522 INFO ActionEndXCommand:539 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[[email protected]] end executor for wf action 0000069-150114201015959-oozie-oozi-W with wf job 0000069-150114201015959-oozie-oozi-W 
2015-04-02 08:50:55,556 WARN CoordActionUpdateXCommand:542 - USER[qjdht93] GROUP[-] TOKEN[] APP[sqoop-shell-wf] JOB[0000069-150114201015959-oozie-oozi-W] ACTION[-] **E1100: Command precondition does not hold before execution, [, coord action is null], Error Code: E1100**`

下面是我workflow.xml

`<workflow-app xmlns='uri:oozie:workflow:0.3' name='sqoop-shell-wf'> 
<start to='sqoop-shell' /> 
<action name='sqoop-shell'> 
    <shell xmlns="uri:oozie:shell-action:0.1"> 
     <job-tracker>${resourceManager}</job-tracker> 
    <name-node>${nameNode}</name-node> 
    <configuration> 
     <property> 
      <name>mapred.job.queue.name</name> 
      <value>${queueName}</value> 
     </property> 
    </configuration> 
    <exec>Sqoopscript.sh</exec> 
    <file>Sqoopscript.sh#script.sh</file> 
</shell> 
<ok to="end" /> 
<error to="fail" /> 
</action> 
<kill name="fail"> 
<message>Script failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> 
</kill>

地圖減少日誌

error: error reading /usr/lib/hadoop/lib/smore.jar; /usr/lib/hadoop/lib/smore.jar (Permission denied) 
error: error reading /usr/lib/hadoop/lib/janusclient.jar; /usr/lib/hadoop/lib/janusclient  .jar (Permission denied) 
error: error reading /usr/lib/hadoop/lib/aster- networking.jar; /usr/lib/hadoop/lib/aster-networking.jar (Permission denied) 
error: error reading /usr/lib/hadoop/lib/adfs-api-loader.jar; /usr/lib/hadoop/lib/adfs-api-loader.jar (Permission denied) 
error: error reading /usr/lib/hadoop/lib/aftp.jar; /usr/lib/hadoop/lib/aftp.jar (Permission denied) 
error: error reading /usr/lib/hadoop/lib/adfs.jar; /usr/lib/hadoop/lib/adfs.jar (Permission denied) 
Note: /tmp/sqoop yarn/compile/f3aca99f37fa19e505b47bab1499bf5d/EXT_EVNT_ERR_ARC.java uses or overrides a deprecated API. 
Note: Recompile with -Xlint:deprecation for details. 
Intercepting System.exit(1) 
Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1] 
che.sqoop.hive.HiveImport - **Caused by: java.io.FileNotFoundException: File does not exist: hdfs:/user/yarn**`

來源

2015-04-02 chhaya vishwakarma

當你可以/應該使用Sqoop動作時，它看起來像你正在使用shell動作。運行Sqoop作業時需要包含幾個罐子。他們中的大多數都包含在oozie sharelib中。這裏有一個例子工作流程：

<workflow-app name="sqoop-to-hive" xmlns="uri:oozie:workflow:0.4"> 
    <start to="sqoop2hive"/> 
    <action name="sqoop2hive"> 
     <sqoop xmlns="uri:oozie:sqoop-action:0.2"> 
      <job-tracker>${jobTracker}</job-tracker> 
      <name-node>${nameNode}</name-node> 
      <command>import --connect jdbc:mysql://mysql.example.com/sqoop --username sqoop --password sqoop --table test --hive-import --hive-table test</command> 
      <archive>/tmp/mysql-connector-java-5.1.31-bin.jar#mysql-connector-java-5.1.31-bin.jar</archive> 
      <file>/tmp/hive-site.xml#hive-site.xml</file> 
     </sqoop> 
     <ok to="end"/> 
     <error to="kill"/> 
    </action> 
    <kill name="kill"> 
     <message>Action failed</message> 
    </kill> 
    <end name="end"/> 
</workflow-app>

我建議以下關於如何使用Oozie的和sqoop一個guide。

來源

2015-04-03 18:33:46 abeaamase

嘿，謝謝!! ..實際上我需要在shell裏寫sqoop命令，因爲'將會傳遞參數文件到shell，它將包含像sqoop所需的信息（IP，tablenamee，DBname）等等，然後清理hdfs目錄並做錯誤處理 – 2015-04-06 07:35:27

嗨更改我的工作流.xml由你建議但得到相同的錯誤「啓動器錯誤，原因：主類[org.apache.oozie.action.hadoop.SqoopMain]，主要（）拋出異常」 – 2015-04-06 10:25:21

你可能需要在job.properties中指定oozie.use.system.libpath = true。 – abeaamase 2015-04-06 10:28:50

使用Sqoop執行Oozie工作流時出錯

回答

相關問題