在HDP 2.3.2沙箱中通過Sqoop導入表時出錯

Question

我想在HDP 2.3.2沙箱中的Hive中導入70 + GB表，但我已經在SQL Server和沙箱之間建立了連接，但是，試圖導入表使用以下命令：

sudo -u hdfs sqoop import --connect "jdbc:sqlserver://XX.XX.XX.XX;database=XX;username=XX;password=XX" --table XX --split-by ID --target-dir "/user/hdfs/Kunal/2" --hive-import -- --schema dbo 

但它給我下面的錯誤

Error: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.UnsupportedOperationException: Java Runtime Environment (JRE) version 1.7 is not supported by this driver. Use the sqljdbc4.jar class library, which provides support for JDBC 4.0. 
    at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:167) 
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:76) 
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:136) 
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:749) 
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) 
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at javax.security.auth.Subject.doAs(Subject.java:415) 
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) 
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) 
Caused by: java.lang.RuntimeException: java.lang.UnsupportedOperationException: Java Runtime Environment (JRE) version 1.7 is not supported by this driver. Use the sqljdbc4.jar class library, which provides support for JDBC 4.0. 
    at org.apache.sqoop.mapreduce.db.DBInputFormat.getConnection(DBInputFormat.java:220) 
    at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:165) 
    ... 9 more 

Answer 1

選項1：要使用單一映射器（-m 1），注意單 - 符號。但整個70GB將在單線程中讀取，您可能會面臨完成，最好寫入一個hdfs文件。

選項2：使用--split-by與稀疏分佈式，--split-by用於分割工作單元的表格列。防爆。 emp表中的Employee_id將是唯一且稀疏分佈的。

參考：http://sqoop.apache.org/docs/1.4.6/SqoopUserGuide.html最新sqoop用戶指南。