Hive上的重建索引失敗

Question

我正在使用HortonWorks沙盒探索Hive的一些功能，並使用Sqoop將表導入到其中。

由於它是一個相當大的表，我試圖在其中一列創建索引。它要求我添加我所做的「WITH DEFERRED REBUILD」選項，並創建索引。

但是當我試圖重建索引時，作業失敗並顯示以下日誌。

日誌

Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: java.lang.NullPointerException 
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:171) 
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137) 
    at org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:337) 
    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179) 
    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at javax.security.auth.Subject.doAs(Subject.java:415) 
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) 
    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171) 
    at org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167) 
    at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36) 
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) 
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) 
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) 
    at java.lang.Thread.run(Thread.java:745) 

Caused by: java.lang.RuntimeException: java.lang.NullPointerException 
    at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:192) 
    at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.<init>(TezGroupedSplitsInputFormat.java:131) 
    at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat.getRecordReader(TezGroupedSplitsInputFormat.java:97) 
    at org.apache.tez.mapreduce.lib.MRReaderMapred.setupOldRecordReader(MRReaderMapred.java:149) 
    at org.apache.tez.mapreduce.lib.MRReaderMapred.setSplit(MRReaderMapred.java:80) 
    at org.apache.tez.mapreduce.input.MRInput.initFromEventInternal(MRInput.java:614) 
    at org.apache.tez.mapreduce.input.MRInput.initFromEvent(MRInput.java:593) 
    at org.apache.tez.mapreduce.input.MRInputLegacy.checkAndAwaitRecordReaderInitialization(MRInputLegacy.java:141) 
    at org.apache.tez.mapreduce.input.MRInputLegacy.init(MRInputLegacy.java:109) 
    at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.getMRInput(MapRecordProcessor.java:370) 
    at org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.init(MapRecordProcessor.java:127) 
    at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:147) 
    ... 14 more 

Caused by: java.lang.NullPointerException 
    at org.apache.hadoop.hive.ql.io.HiveInputFormat.init(HiveInputFormat.java:270) 
    at org.apache.hadoop.hive.ql.io.HiveInputFormat.getRecordReader(HiveInputFormat.java:234) 
    at org.apache.hadoop.mapred.split.TezGroupedSplitsInputFormat$TezGroupedSplitsRecordReader.initNextRecordReader(TezGroupedSplitsInputFormat.java:189) 
    ... 25 more 

Answer 1

我遇到了同樣的問題。我發現了一個解決辦法

set hive.execution.engine=mr; 
alter index index_name on table_name rebuild; 
set hive.execution.engine=tez; 

運行重建使用MR沒有問題爲我工作的指標。我不確定TEZ爲什麼會失敗。

Answer 2

該問題與tez實際上在tez中配置錯誤的內存參數有關。

這就是爲什麼切換到MR解決問題。

關於tez內存的一篇很好的文章在調試這個問題時非常有用。

https://community.hortonworks.com/content/kbentry/14309/demystify-tez-tuning-step-by-step.html