1. 首頁
  2. 問答
  3. Sqoop-Hive導入

Sqoop-Hive導入

2016-04-15 180 views
0

我在通過sqoop將rdbms的數據導入配置單元時面臨一個問題。請不要將其標記爲重複Sqoop-Hive導入

我正在將數據從oracle 12c db導入到Hive中。數據是在hdfs裏面創建的，但是hive表沒有被創建，即一個文件夾在我導入的表的名字裏面/ user/hive/warehouse裏創建，但是當我進入hive編輯器並檢查這個表時不顯示我。

但是，如果我在導入數據（create table ..）之前創建了此表的模式，現在如果通過運行sqoop導入來導入數據，則也可以在hdfs和hive表中找到數據。

下面是我在做什麼

1.正開始

usermod -a -G supergroup venkat 
    export SQOOP_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop 
    export HIVE_HOME=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hive 
    export HADOOP_CLASSPATH=/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/ojdbc7.jar:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/hive/lib/* 
    export HADOOP_USER_NAME=hdfs 
    export PATH=$PATH:$HIVE_HOME/bin
細節

2.運行sqoop導入

sqoop import --connect jdbc:oracle:thin:@bigdatadev2:1521/orcl --username BDD1 --password oracle123 --table EMP -m 1 --hive-import --hive-table emp

以下是日誌：

16/04/15 18:10:10 DEBUG orm.CompilationManager: Could not rename /tmp/sqoop-root/compile/c6516678b105260186a4b5daf7552fcb/EMP.java to /root/./EMP.java 
    16/04/15 18:10:10 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-root/compile/c6516678b105260186a4b5daf7552fcb/EMP.jar 
    16/04/15 18:10:10 DEBUG orm.CompilationManager: Scanning for .class files in directory: /tmp/sqoop-root/compile/c6516678b105260186a4b5daf7552fcb 
    16/04/15 18:10:10 DEBUG orm.CompilationManager: Got classfile: /tmp/sqoop-root/compile/c6516678b105260186a4b5daf7552fcb/EMP.class -> EMP.class 
    16/04/15 18:10:10 DEBUG orm.CompilationManager: Finished writing jar file /tmp/sqoop-root/compile/c6516678b105260186a4b5daf7552fcb/EMP.jar 
    16/04/15 18:10:10 DEBUG manager.OracleManager$ConnCache: Got cached connection for jdbc:oracle:thin:@bigdatadev2:1521/orcl/BDD1 
    16/04/15 18:10:10 INFO manager.OracleManager: Time zone has been set to GMT 
    16/04/15 18:10:10 DEBUG manager.OracleManager$ConnCache: Caching released connection for jdbc:oracle:thin:@bigdatadev2:1521/orcl/BDD1 
    16/04/15 18:10:10 DEBUG manager.OracleManager$ConnCache: Got cached connection for jdbc:oracle:thin:@bigdatadev2:1521/orcl/BDD1 
    16/04/15 18:10:10 INFO manager.OracleManager: Time zone has been set to GMT 
    16/04/15 18:10:10 DEBUG manager.OracleManager$ConnCache: Caching released connection for jdbc:oracle:thin:@bigdatadev2:1521/orcl/BDD1 
    16/04/15 18:10:10 INFO mapreduce.ImportJobBase: Beginning import of EMP 
    16/04/15 18:10:10 DEBUG util.ClassLoaderStack: Checking for existing class: EMP 
    16/04/15 18:10:10 DEBUG util.ClassLoaderStack: Attempting to load jar through URL: jar:file:/tmp/sqoop-root/compile/c6516678b105260186a4b5daf7552fcb/EMP.jar!/ 
    16/04/15 18:10:10 DEBUG util.ClassLoaderStack: Previous classloader is [email protected] 
    16/04/15 18:10:10 DEBUG util.ClassLoaderStack: Testing class in jar: EMP 
    16/04/15 18:10:10 DEBUG util.ClassLoaderStack: Loaded jar into current JVM: jar:file:/tmp/sqoop-root/compile/c6516678b105260186a4b5daf7552fcb/EMP.jar!/ 
    16/04/15 18:10:10 DEBUG util.ClassLoaderStack: Added classloader for jar /tmp/sqoop-root/compile/c6516678b105260186a4b5daf7552fcb/EMP.jar: [email protected] 
    16/04/15 18:10:10 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar 
    16/04/15 18:10:10 DEBUG db.DBConfiguration: Securing password into job credentials store 
    16/04/15 18:10:10 DEBUG manager.OracleManager$ConnCache: Got cached connection for jdbc:oracle:thin:@bigdatadev2:1521/orcl/BDD1 
    16/04/15 18:10:10 INFO manager.OracleManager: Time zone has been set to GMT 
    16/04/15 18:10:10 DEBUG manager.OracleManager$ConnCache: Caching released connection for jdbc:oracle:thin:@bigdatadev2:1521/orcl/BDD1 
    16/04/15 18:10:10 DEBUG mapreduce.DataDrivenImportJob: Using table class: EMP 
    16/04/15 18:10:10 DEBUG mapreduce.DataDrivenImportJob: Using InputFormat: class com.cloudera.sqoop.mapreduce.db.OracleDataDrivenDBInputFormat 
    16/04/15 18:10:11 INFO Configuration.deprecation: mapred.map.tasks is deprecated. Instead, use mapreduce.job.maps 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/sqoop-1.4.6-cdh5.5.1.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/ojdbc6.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/sqoop-1.4.6-cdh5.5.1.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/sqoop-1.4.6-cdh5.5.1.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/parquet-jackson.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/avro-mapred-hadoop2.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/parquet-avro.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/xz-1.0.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/jackson-annotations-2.3.1.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/commons-compress-1.4.1.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/fastutil-6.3.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/ojdbc6.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/ant-eclipse-1.0-jvm1.2.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/jackson-databind-2.3.1.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/avro.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/kite-data-mapreduce.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/kite-data-hive.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/kite-hadoop-compatibility.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/jackson-core-asl-1.8.8.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/commons-io-1.4.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/parquet-encoding.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/kite-data-core.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/hsqldb-1.8.0.10.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/parquet-column.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/parquet-common.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/ojdbc7.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/commons-jexl-2.1.1.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/paranamer-2.3.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/commons-logging-1.1.3.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/parquet-hadoop.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/parquet-format.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/slf4j-api-1.7.5.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/opencsv-2.3.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/jackson-mapper-asl-1.8.8.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/snappy-java-1.0.4.1.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/commons-codec-1.4.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/jackson-core-2.3.1.jar 
    16/04/15 18:10:11 DEBUG mapreduce.JobBase: Adding to job classpath: file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/lib/sqoop/lib/ant-contrib-1.0b3.jar 
    16/04/15 18:10:11 INFO client.RMProxy: Connecting to ResourceManager at bigdata/10.103.25.39:8032 
    16/04/15 18:10:19 DEBUG db.DBConfiguration: Fetching password from job credentials store 
    16/04/15 18:10:19 INFO db.DBInputFormat: Using read commited transaction isolation 
    16/04/15 18:10:19 DEBUG db.DataDrivenDBInputFormat: Creating input split with lower bound '1=1' and upper bound '1=1' 
    16/04/15 18:10:19 INFO mapreduce.JobSubmitter: number of splits:1 
    16/04/15 18:10:20 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1460530815947_0198 
    16/04/15 18:10:21 INFO impl.YarnClientImpl: Submitted application application_1460530815947_0198 
    16/04/15 18:10:21 INFO mapreduce.Job: The url to track the job: http://bigdata:8088/proxy/application_1460530815947_0198/ 
    16/04/15 18:10:21 INFO mapreduce.Job: Running job: job_1460530815947_0198 
    16/04/15 18:10:30 INFO mapreduce.Job: Job job_1460530815947_0198 running in uber mode : false 
    16/04/15 18:10:30 INFO mapreduce.Job: map 0% reduce 0% 
    16/04/15 18:10:38 INFO mapreduce.Job: map 100% reduce 0% 
    16/04/15 18:10:39 INFO mapreduce.Job: Job job_1460530815947_0198 completed successfully 
    16/04/15 18:10:39 INFO mapreduce.Job: Counters: 30 
     File System Counters 
      FILE: Number of bytes read=0 
      FILE: Number of bytes written=137941 
      FILE: Number of read operations=0 
      FILE: Number of large read operations=0 
      FILE: Number of write operations=0 
      HDFS: Number of bytes read=87 
      HDFS: Number of bytes written=12 
      HDFS: Number of read operations=4 
      HDFS: Number of large read operations=0 
      HDFS: Number of write operations=2 
     Job Counters 
      Launched map tasks=1 
      Other local map tasks=1 
      Total time spent by all maps in occupied slots (ms)=4157 
      Total time spent by all reduces in occupied slots (ms)=0 
      Total time spent by all map tasks (ms)=4157 
      Total vcore-seconds taken by all map tasks=4157 
      Total megabyte-seconds taken by all map tasks=2128384 
     Map-Reduce Framework 
      Map input records=3 
      Map output records=3 
      Input split bytes=87 
      Spilled Records=0 
      Failed Shuffles=0 
      Merged Map outputs=0 
      GC time elapsed (ms)=56 
      CPU time spent (ms)=2240 
      Physical memory (bytes) snapshot=213250048 
      Virtual memory (bytes) snapshot=2180124672 
      Total committed heap usage (bytes)=134742016 
     File Input Format Counters 
      Bytes Read=0 
     File Output Format Counters 
      Bytes Written=12 
    16/04/15 18:10:39 INFO mapreduce.ImportJobBase: Transferred 12 bytes in 28.3024 seconds (0.424 bytes/sec) 
    16/04/15 18:10:39 INFO mapreduce.ImportJobBase: Retrieved 3 records. 
    16/04/15 18:10:39 DEBUG util.ClassLoaderStack: Restoring classloader: [email protected] 
    16/04/15 18:10:39 DEBUG hive.HiveImport: Hive.inputTable: EMP 
    16/04/15 18:10:39 DEBUG hive.HiveImport: Hive.outputTable: emp 
    16/04/15 18:10:39 DEBUG manager.OracleManager: Using column names query: SELECT t.* FROM EMP t WHERE 1=0 
    16/04/15 18:10:39 DEBUG manager.SqlManager: Execute getColumnInfoRawQuery : SELECT t.* FROM EMP t WHERE 1=0 
    16/04/15 18:10:39 DEBUG manager.OracleManager$ConnCache: Got cached connection for jdbc:oracle:thin:@bigdatadev2:1521/orcl/BDD1 
    16/04/15 18:10:39 INFO manager.OracleManager: Time zone has been set to GMT 
    16/04/15 18:10:39 DEBUG manager.SqlManager: Using fetchSize for next query: 1000 
    16/04/15 18:10:39 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM EMP t WHERE 1=0 
    16/04/15 18:10:39 DEBUG manager.SqlManager: Found column EMP_ID of type [12, 5, 0] 
    16/04/15 18:10:39 DEBUG manager.SqlManager: Found column EMP_NAME of type [12, 100, 0] 
    16/04/15 18:10:39 DEBUG manager.OracleManager$ConnCache: Caching released connection for jdbc:oracle:thin:@bigdatadev2:1521/orcl/BDD1 
    16/04/15 18:10:39 DEBUG hive.TableDefWriter: Create statement: CREATE TABLE IF NOT EXISTS `emp` (`EMP_ID` STRING, `EMP_NAME` STRING) COMMENT 'Imported by sqoop on 2016/04/15 18:10:39' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE 
    16/04/15 18:10:39 DEBUG hive.TableDefWriter: Load statement: LOAD DATA INPATH 'hdfs://bigdata:8020/user/hdfs/EMP' INTO TABLE `emp` 
    16/04/15 18:10:39 INFO hive.HiveImport: Loading uploaded data into Hive 
    16/04/15 18:10:39 DEBUG hive.HiveImport: Using in-process Hive instance. 
    16/04/15 18:10:39 DEBUG util.SubprocessSecurityManager: Installing subprocess security manager 

    Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/hive-common-1.1.0-cdh5.5.1.jar!/hive-log4j.properties 
    OK 
    Time taken: 0.782 seconds 
    Loading data to table default.emp 
    Table default.emp stats: [numFiles=1, numRows=0, totalSize=12, rawDataSize=0] 
    OK 
    Time taken: 0.676 seconds 
    You have new mail in /var/spool/mail/root 
    [[email protected] ~]#

3.當我運行下面sqoop導入命令，我收到以下錯誤（顯示日誌的最後幾行）

sqoop import --connect jdbc:oracle:thin:@bigdatadev2:1521/orcl --username BDD1 --password oracle123 --table EMP -m 1 --create-hive-table --hive-import --hive-table emp 

    16/04/15 18:17:02 INFO mapreduce.Job: The url to track the job: http://bigdata:8088/proxy/application_1460530815947_0200/ 
    16/04/15 18:17:02 INFO mapreduce.Job: Running job: job_1460530815947_0200 
    16/04/15 18:17:11 INFO mapreduce.Job: Job job_1460530815947_0200 running in uber mode : false 
    16/04/15 18:17:11 INFO mapreduce.Job: map 0% reduce 0% 
    16/04/15 18:17:17 INFO mapreduce.Job: map 100% reduce 0% 
    16/04/15 18:17:19 INFO mapreduce.Job: Job job_1460530815947_0200 completed successfully 
    16/04/15 18:17:19 INFO mapreduce.Job: Counters: 30 
     File System Counters 
      FILE: Number of bytes read=0 
      FILE: Number of bytes written=137940 
      FILE: Number of read operations=0 
      FILE: Number of large read operations=0 
      FILE: Number of write operations=0 
      HDFS: Number of bytes read=87 
      HDFS: Number of bytes written=12 
      HDFS: Number of read operations=4 
      HDFS: Number of large read operations=0 
      HDFS: Number of write operations=2 
     Job Counters 
      Launched map tasks=1 
      Other local map tasks=1 
      Total time spent by all maps in occupied slots (ms)=4067 
      Total time spent by all reduces in occupied slots (ms)=0 
      Total time spent by all map tasks (ms)=4067 
      Total vcore-seconds taken by all map tasks=4067 
      Total megabyte-seconds taken by all map tasks=2082304 
     Map-Reduce Framework 
      Map input records=3 
      Map output records=3 
      Input split bytes=87 
      Spilled Records=0 
      Failed Shuffles=0 
      Merged Map outputs=0 
      GC time elapsed (ms)=58 
      CPU time spent (ms)=2350 
      Physical memory (bytes) snapshot=221048832 
      Virtual memory (bytes) snapshot=2178715648 
      Total committed heap usage (bytes)=135790592 
     File Input Format Counters 
      Bytes Read=0 
     File Output Format Counters 
      Bytes Written=12 
    16/04/15 18:17:19 INFO mapreduce.ImportJobBase: Transferred 12 bytes in 21.9242 seconds (0.5473 bytes/sec) 
    16/04/15 18:17:19 INFO mapreduce.ImportJobBase: Retrieved 3 records. 
    16/04/15 18:17:19 DEBUG util.ClassLoaderStack: Restoring classloader: [email protected] 
    16/04/15 18:17:19 DEBUG hive.HiveImport: Hive.inputTable: EMP 
    16/04/15 18:17:19 DEBUG hive.HiveImport: Hive.outputTable: emp 
    16/04/15 18:17:19 DEBUG manager.OracleManager: Using column names query: SELECT t.* FROM EMP t WHERE 1=0 
    16/04/15 18:17:19 DEBUG manager.SqlManager: Execute getColumnInfoRawQuery : SELECT t.* FROM EMP t WHERE 1=0 
    16/04/15 18:17:19 DEBUG manager.OracleManager$ConnCache: Got cached connection for jdbc:oracle:thin:@bigdatadev2:1521/orcl/BDD1 
    16/04/15 18:17:19 INFO manager.OracleManager: Time zone has been set to GMT 
    16/04/15 18:17:19 DEBUG manager.SqlManager: Using fetchSize for next query: 1000 
    16/04/15 18:17:19 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM EMP t WHERE 1=0 
    16/04/15 18:17:19 DEBUG manager.SqlManager: Found column EMP_ID of type [12, 5, 0] 
    16/04/15 18:17:19 DEBUG manager.SqlManager: Found column EMP_NAME of type [12, 100, 0] 
    16/04/15 18:17:19 DEBUG manager.OracleManager$ConnCache: Caching released connection for jdbc:oracle:thin:@bigdatadev2:1521/orcl/BDD1 
    16/04/15 18:17:19 DEBUG hive.TableDefWriter: Create statement: CREATE TABLE `emp` (`EMP_ID` STRING, `EMP_NAME` STRING) COMMENT 'Imported by sqoop on 2016/04/15 18:17:19' ROW FORMAT DELIMITED FIELDS TERMINATED BY '\001' LINES TERMINATED BY '\012' STORED AS TEXTFILE 
    16/04/15 18:17:19 DEBUG hive.TableDefWriter: Load statement: LOAD DATA INPATH 'hdfs://bigdata:8020/user/hdfs/EMP' INTO TABLE `emp` 
    16/04/15 18:17:19 INFO hive.HiveImport: Loading uploaded data into Hive 
    16/04/15 18:17:19 DEBUG hive.HiveImport: Using in-process Hive instance. 
    16/04/15 18:17:19 DEBUG util.SubprocessSecurityManager: Installing subprocess security manager 

    Logging initialized using configuration in jar:file:/opt/cloudera/parcels/CDH-5.5.1-1.cdh5.5.1.p0.11/jars/hive-common-1.1.0-cdh5.5.1.jar!/hive-log4j.properties 
    FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. AlreadyExistsException(message:Table emp already exists)

4. 現在，當我創建表的架構「創建表emp ...'然後導入數據，我得到我的hdfs以及hive表中的數據。

僅當表格模式存在時才能使用sqoop嗎？它不能自己創建一個新表？

我是否在我的導入命令行中丟失了某些東西。請不要將其標記爲重複

來源

2016-04-15 Hardik Gupta

+0

但創建表腳本==> ** CREATE TABLE'emp'（'EMP_ID' STRING，'EMP_NAME' STRING）COMMENT '由sqoop上2016年4月15日十八時17分19秒進口' 行格式由'\ 001'終止的'\ 001'終結的界限字段存儲爲TEXTFILE **，執行時沒有任何錯誤。只需從配置單元shell啓動此命令並檢查表是否已創建 –

+0

是的，當我運行腳本CREATE TABLE emp（EMP_ID STRING，EMP_NAME STRING）時，表被創建。現在如果我運行我的sqoop導入它填充表中的數據 –

回答

0

通常這些錯誤是由於權限錯誤&錯誤配置引起的。考慮到這不是一個增量數據攝入，下面應該工作。

我已經添加了倉庫目錄&配置單元覆蓋選項。請在Hive &中刪除現有的emp表，執行下面的命令，用hdfs中的適當倉庫目錄替換path/to/dir。

sqoop import --connect jdbc:oracle:thin:@bigdatadev2:1521/orcl --username BDD1 --password oracle123 --table EMP -m 1 --warehouse-dir "path/to/dir" --hive-import --hive-overwrite --hive-table emp

來源

2016-04-17 15:22:43 gkc123

+0

嗨魯本，我運行命令，我給倉庫目錄/ user/hive/warehouse，因爲這是我的'emp'文件夾被創建的地方（沒有文件夾被創建在/ user/hdfs文件夾中）。但是仍然沒有創建表 –

+0

@HardikGupta/user/hive/warehouse是配置單元的倉庫目錄，通常還有sqoop的倉庫目錄。我的假設是，即使由於權限錯誤而執行「drop table emp」，表格也不會被刪除。您需要小心並在整個系統中使用相同的用戶以避免出現此類問題。首先，我會檢查表是否被丟棄;創建＆give/user/hdfs/data作爲warehouse-dir;使用hdfs用戶從命令行執行sqoop命令;爲emp表創建和提供特定的數據庫名稱。 – gkc123

相關問題

 相關問題