2013-06-03 133 views
2

我試圖運行sqoop作爲oozie行動。在jobtracker節點上我有MySQL作爲Metastore。我在某處讀到oozie sqoop無法從導入創建配置單元表。所以我試圖將數據轉儲到hdfs。這仍然是真的嗎?Oozie Sqoop工作

我檢查了共享庫。

我正試圖從MS SQL數據庫的Sqoop。

當我運行sqoop命令,它不需要從shell的元存儲它的作品。

sqoop import --connect 'jdbc:sqlserver://host;username=sqoopimport;password=password;database=db1' --table t1--target-dir /user/root/sqoop-import/tmp/t1 

當我嘗試運行此作爲sqoop動作我得到這個錯誤

>>> Invoking Sqoop command line now >>> 

2151 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration. 
2259 [main] WARN org.apache.sqoop.ConnFactory - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration. 
2285 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Got error creating database manager: java.io.IOException: No manager for connect string: 'jdbc:sqlserver://host;username=sqoopimport;password=password;database=db1' 
at org.apache.sqoop.ConnFactory.getManager(ConnFactory.java:185) 
at org.apache.sqoop.tool.BaseSqoopTool.init(BaseSqoopTool.java:217) 
at org.apache.sqoop.tool.ImportTool.init(ImportTool.java:83) 
at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:464) 
at org.apache.sqoop.Sqoop.run(Sqoop.java:145) 
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) 
at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:181) 
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:220) 
at org.apache.sqoop.Sqoop.runTool(Sqoop.java:229) 
at org.apache.sqoop.Sqoop.main(Sqoop.java:238) 
at org.apache.oozie.action.hadoop.SqoopMain.runSqoopJob(SqoopMain.java:203) 
at org.apache.oozie.action.hadoop.SqoopMain.run(SqoopMain.java:172) 
at org.apache.oozie.action.hadoop.LauncherMain.run(LauncherMain.java:37) 
at org.apache.oozie.action.hadoop.SqoopMain.main(SqoopMain.java:45) 
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
at java.lang.reflect.Method.invoke(Method.java:597) 
at org.apache.oozie.action.hadoop.LauncherMapper.map(LauncherMapper.java:495) 
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) 
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:417) 
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332) 
at org.apache.hadoop.mapred.Child$4.run(Child.java:268) 
at java.security.AccessController.doPrivileged(Native Method) 
at javax.security.auth.Subject.doAs(Subject.java:396) 
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) 
at org.apache.hadoop.mapred.Child.main(Child.java:262) 

workflow.xml

<?xml version="1.0" encoding="UTF-8"?> 
<workflow-app xmlns="uri:oozie:workflow:0.2" name="sqoop-wf"> 
    <start to="sqoop-node"/> 

    <action name="sqoop-node"> 
     <sqoop xmlns="uri:oozie:sqoop-action:0.2"> 
      <job-tracker>${jobTracker}</job-tracker> 
      <name-node>${nameNode}</name-node> 
      <configuration> 
       <property> 
        <name>mapred.job.queue.name</name> 
        <value>${queueName}</value> 
       </property> 
       <property> 
      <name>oozie.use.system.libpath</name> 
      <value>true</value> 
      </property> 
      <property> 
      <name>oozie.libpath</name> 
      <value>/user/oozie/share/lib/sqoop</value> 
      </property> 
      </configuration> 
      <command>import --connect 'jdbc:sqlserver://host;username=sqoopimport;password=password;database=db1' --table t1--target-dir /user/root/sqoop-import/tmp/t1</command> 
     </sqoop> 
     <ok to="end"/> 
     <error to="fail"/> 
    </action> 

    <kill name="fail"> 
     <message>Sqoop failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message> 
    </kill> 
    <end name="end"/> 
</workflow-app> 

我覺得這是最主要的原因。我有股lib目錄下的庫 2285 [主]錯誤org.apache.sqoop.tool.BaseSqoopTool - 遇到錯誤創建數據庫管理器:java.io.IOException異常:沒有經理的連接字符串:

我缺少的東西?任何幫助表示讚賞。

感謝, 阿布舍克

回答

3

當您使用shell(例如Bash或ZSH)執行Sqoop時,您需要手動轉義參數,以便shell不會改變它們。在您的示例中,您已將JDBC URL放入引號中,以便分號不會被解釋爲命令的結尾。由於Oozie不使用shell來調用Sqoop,所以這些轉義字符無效。因此,您應該刪除在oozie工作流程中爲shell引入的轉義。

E.g:

<command>import --connect jdbc:sqlserver://host;username=sqoopimport;password=password;database=db1 --table t1--target-dir /user/root/sqoop-import/tmp/t1</command> 

同時請注意,建議使用參數--username和--password而不是JDBC URL名稱相同的屬性。

0

嘗試在你的論點給予--driver com.microsoft.jdbc.sqlserver.SQLServerDriver>另外,還要確保你有存在於類路徑中的MS SQL Server JDBC罐子。