通過oozie的Sqoop工作

我已經創建了一個名爲TeamMemsImportJob的sqoop作業，它基本上將數據從sql server拉入配置單元。我可以通過UNIX命令行中運行以下命令執行sqoop作業：如果我創建一個有實際瓢導入命令的Oozie的工作通過oozie的Sqoop工作

sqoop job –exec TeamMemsImportJob

，它貫穿罰款。但是，如果我創建了Oozie的工作，並貫穿它的sqoop工作，我得到以下錯誤：

oozie job -config TeamMemsImportJob.properties -run 

>>> Invoking Sqoop command line now >>> 

4273 [main] WARN org.apache.sqoop.tool.SqoopTool – $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration. 
4329 [main] INFO org.apache.sqoop.Sqoop – Running Sqoop version: 1.4.4.2.1.1.0-385 
5172 [main] ERROR org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage – Cannot restore job: TeamMemsImportJob 
5172 [main] ERROR org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage – (No such job) 
5172 [main] ERROR org.apache.sqoop.tool.JobTool – I/O error performing job operation: java.io.IOException: Cannot restore missing job TeamMemsImportJob 
at org.apache.sqoop.metastore.hsqldb.HsqldbJobStorage.read(HsqldbJobStorage.java:256) 
at org.apache.sqoop.tool.JobTool.execJob(JobTool.java:198)

它看起來好像它無法找到工作。但我可以看到下面的工作

[[email protected] ~]# sqoop job –list 
Warning: /usr/lib/sqoop/../accumulo does not exist! Accumulo imports will fail. 
Please set $ACCUMULO_HOME to the root of your Accumulo installation. 
14/06/25 08:12:08 INFO sqoop.Sqoop: Running Sqoop version: 1.4.4.2.1.1.0-385 
Available jobs: 
TeamMemsImportJob

我該如何解決？

來源

2014-06-25 Colman

如果我們看到日誌，我們可以看到它找不到存儲的作業。

由於您使用的是本地hsql數據庫。

要使Sqoop作業可以跨其他系統使用，您應該配置其他數據庫，例如可以被所有系統訪問的mysql。

從技術文檔

Running sqoop-metastore launches a shared HSQLDB database instance on the current machine. Clients can connect to this metastore and create jobs which can be shared between users for execution

The location of the metastore’s files on disk is controlled by the sqoop.metastore.server.location property in conf/sqoop-site.xml. This should point to a directory on the local filesystem.

The metastore is available over TCP/IP. The port is controlled by the sqoop.metastore.server.port configuration parameter, and defaults to 16000.

Clients should connect to the metastore by specifying sqoop.metastore.client.autoconnect.url or --meta-connect with the value jdbc:hsqldb:hsql://:/sqoop. For example, jdbc:hsqldb:hsql://metaserver.example.com:16000/sqoop.

This metastore may be hosted on a machine within the Hadoop cluster, or elsewhere on the network.

可以檢查該數據庫是從其他系統訪問。

來源

2014-07-15 00:58:21 user2230605

您必須使用--meta-connect標誌，同時創建作業以創建自定義Sqoop Metastore數據庫，以便Oozie可以訪問。

sqoop \ 
job \ 
--meta-connect \ 
"jdbc:hsqldb:file:/on/server/not/hdfs/sqoop-metastore/sqoop-meta.db;shutdown=true" \ 
--create \ 
jobName \ 
-- \ 
import \ 
--connect jdbc:oracle:thin:@server:port:sid \ 
--username username \ 
--password-file /path/on/hdfs/server.password \ 
--table TABLE \ 
--incremental append \ 
--check-column ID \ 
--last-value "0" \ 
--target-dir /path/on/hdfs/TABLE

當你需要執行的工作，你可以從Oozie的做到這一點的常規方式，但一定要包括--meta-connect來指示作業的存儲位置。

來源

2016-04-19 11:49:45 kfkhalili

通過oozie的Sqoop工作

回答

相關問題