Sqoop導出錯誤 - 原因：org.apache.hadoop.mapreduce.lib.input.InvalidInputException：輸入路徑不存在

Question

我正在開發一個java程序。

java程序將數據從配置單元導出到mysql。

首先，我寫的代碼

ProcessBuilder pb = new ProcessBuilder("sqoop-export", "export", 
     "--connect",    "jdbc:mysql://localhost/mydb", 
     "--hadoop-home", "/home/yoonhok/development/hadoop-1.1.1", 
     "--table",     "mytable", 
     "--export-dir",   "/user/hive/warehouse/tbl_2", 
     "--username",   "yoonhok", 
     "--password",   "1234"); 

try { 
    Process p = pb.start(); 
    if (p.waitFor() != 0) { 
     System.out.println("Error: sqoop-export failed."); 
     return false; 
    } 
} catch (IOException e) { 
    e.printStackTrace(); 
} catch (InterruptedException e) { 
    e.printStackTrace(); 
} 

它完美。

但我學到了一種在java中使用sqoop的新方法。

Sqoop還不支持客戶端API。

所以我添加sqoop lib和只寫Sqoop.run（）

第二，我用新的方式重新編寫代碼。

String[] str = {"export", 
    "--connect",    "jdbc:mysql://localhost/mydb", 
    "--hadoop-home", "/home/yoonhok/development/hadoop-1.1.1", 
    "--table",     "mytable", 
    "--export-dir",   "/user/hive/warehouse/tbl_2", 
    "--username",   "yoonhok", 
    "--password",   "1234" 
}; 

if (Sqoop.runTool(str) == 1) { 
    System.out.println("Error: sqoop-export failed."); 
    return false; 
} 

但它沒有運行。

我得到了錯誤......

13/02/14 16:17:09 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 
13/02/14 17:43:12 WARN sqoop.ConnFactory: $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration. 
13/02/14 16:17:09 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 
13/02/14 16:17:09 INFO tool.CodeGenTool: Beginning code generation 
13/02/14 16:17:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_2` AS t LIMIT 1 
13/02/14 16:17:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_2` AS t LIMIT 1 
13/02/14 16:17:09 INFO orm.CompilationManager: HADOOP_HOME is /home/yoonhok/development/hadoop-1.1.1 
Note: /tmp/sqoop-yoonhok/compile/45dd1a113123726796a4ed4ce10c9110/tbl_2.java uses or overrides a deprecated API. 
Note: Recompile with -Xlint:deprecation for details. 
13/02/14 16:17:10 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-yoonhok/compile/45dd1a113123726796a4ed4ce10c9110/tbl_2.jar 
13/02/14 16:17:10 INFO mapreduce.ExportJobBase: Beginning export of tbl_2 
13/02/14 16:17:10 WARN mapreduce.ExportJobBase: Input path file:/user/hive/warehouse/tbl_2 does not exist 
13/02/14 16:17:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
13/02/14 16:17:11 INFO mapred.JobClient: Cleaning up the staging area file:/tmp/hadoop-yoonhok/mapred/staging/yoonhok314601126/.staging/job_local_0001 
13/02/14 16:17:11 ERROR security.UserGroupInformation: PriviledgedActionException as:yoonhok cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/user/hive/warehouse/tbl_2 
13/02/14 16:17:11 ERROR tool.ExportTool: Encountered IOException running export job: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/user/hive/warehouse/tbl_2 

我看到$SQOOP_CONF_DIR has not been set in the environment.

，所以我說

SQOOP_CONF_DIR = /家庭/ yoonhok /開發/ sqoop-1.4.2 .bin__hadoop-1.0.0/conf

in the

的/ etc /環境

，然後再試一次，但錯誤......

13/02/14 16:17:09 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 
13/02/14 16:17:09 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 
13/02/14 16:17:09 INFO tool.CodeGenTool: Beginning code generation 
13/02/14 16:17:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_2` AS t LIMIT 1 
13/02/14 16:17:09 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_2` AS t LIMIT 1 
13/02/14 16:17:09 INFO orm.CompilationManager: HADOOP_HOME is /home/yoonhok/development/hadoop-1.1.1 
Note: /tmp/sqoop-yoonhok/compile/45dd1a113123726796a4ed4ce10c9110/tbl_2.java uses or overrides a deprecated API. 
Note: Recompile with -Xlint:deprecation for details. 
13/02/14 16:17:10 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-yoonhok/compile/45dd1a113123726796a4ed4ce10c9110/tbl_2.jar 
13/02/14 16:17:10 INFO mapreduce.ExportJobBase: Beginning export of tbl_2 
13/02/14 16:17:10 WARN mapreduce.ExportJobBase: Input path file:/user/hive/warehouse/tbl_2 does not exist 
13/02/14 16:17:11 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
13/02/14 16:17:11 INFO mapred.JobClient: Cleaning up the staging area file:/tmp/hadoop-yoonhok/mapred/staging/yoonhok314601126/.staging/job_local_0001 
13/02/14 16:17:11 ERROR security.UserGroupInformation: PriviledgedActionException as:yoonhok cause:org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/user/hive/warehouse/tbl_2 
13/02/14 16:17:11 ERROR tool.ExportTool: Encountered IOException running export job: org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: file:/user/hive/warehouse/tbl_2 

我認爲，出口-dir是問題。

我使用「/ user/hive/warehouse/tbl_2」。

而當我運行「hadoop fs -ls/user/hive/warehouse /」時，表「tbl_2」存在。

我認爲

「輸入路徑不存在：文件：/用戶/蜂巢/倉儲/ tbl_2」 也不行。

「輸入路徑不存在：hdfs：/ user/hive/warehouse/tbl_2」可以。

但我不知道如何解決它。

---

好吧，在我得到提示之前。

我編輯'出口目錄'

--export-dir hdfs://localhost:9000/user/hive/warehouse/tbl_2 

但是......這是錯誤... TT

13/02/15 15:17:20 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 
13/02/15 15:17:20 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. 
13/02/15 15:17:20 INFO tool.CodeGenTool: Beginning code generation 
13/02/15 15:17:20 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_2` AS t LIMIT 1 
13/02/15 15:17:20 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `tbl_2` AS t LIMIT 1 
13/02/15 15:17:20 INFO orm.CompilationManager: HADOOP_HOME is /home/yoonhok/development/hadoop-1.1.1/libexec/.. 
Note: /tmp/sqoop-yoonhok/compile/697590ee9b90c022fb8518b8a6f1d86b/tbl_2.java uses or overrides a deprecated API. 
Note: Recompile with -Xlint:deprecation for details. 
13/02/15 15:17:22 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-yoonhok/compile/697590ee9b90c022fb8518b8a6f1d86b/tbl_2.jar 
13/02/15 15:17:22 INFO mapreduce.ExportJobBase: Beginning export of tbl_2 
13/02/15 15:17:22 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
13/02/15 15:17:23 INFO input.FileInputFormat: Total input paths to process : 1 
13/02/15 15:17:23 INFO input.FileInputFormat: Total input paths to process : 1 
13/02/15 15:17:23 INFO mapred.JobClient: Cleaning up the staging area file:/tmp/hadoop-yoonhok/mapred/staging/yoonhok922915382/.staging/job_local_0001 
13/02/15 15:17:23 ERROR security.UserGroupInformation: PriviledgedActionException as:yoonhok cause:java.io.FileNotFoundException: File /user/hive/warehouse/tbl_2/000000_0 does not exist. 
13/02/15 15:17:23 ERROR tool.ExportTool: Encountered IOException running export job: java.io.FileNotFoundException: File /user/hive/warehouse/tbl_2/000000_0 does not exist. 

當我檢查HDFS，

hadoop fs -ls /user/hive/warehouse/tbl_2 

或

hadoop fs -ls hdfs://localhost:9000/user/hive/warehouse/tbl_2 

該文件存在。

-rw-R - R-- 1個yoonhok超羣14029022 2013年2月15日12點十六分/用戶/蜂巢/倉庫/ tbl_2/000000_0

我嘗試在外殼命令終端

sqoop-export --connect jdbc:mysql://localhost/detector --table tbl_2 --export-dir hdfs://localhost:9000/user/hive/warehouse/tbl_2 --username yoonhok --password 1234 

這是工作。

什麼問題？

我不知道。

你能幫我嗎？

Answer 1

您需要加載並提供您的Hadoop配置文件。默認情況下，它們是從classpath中讀取的，但是您可以通過Configuration.setDefaultResource（沒有保證）覆蓋它。