1. 首頁
  2. 問答
  3. Hive給出SemanticException [Error 10014]：運行我的UDF時

Hive給出SemanticException [Error 10014]：運行我的UDF時

2017-04-14 83 views
1

我有一個配置單元UDF，它執行GeoIP查找。Hive給出SemanticException [Error 10014]：運行我的UDF時

public static Text evaluate(Text inputFieldName, Text option, 
     Text databaseFileName) { 

    String inputField, fieldOption, dbFileName, result = null; 

    inputField = inputFieldName.toString(); 
    fieldOption = option.toString(); 
    dbFileName = databaseFileName.toString(); 
    ExtractData eed = new ExtractData(); 
    try { 
     result = eed.ExtractDB(inputField, fieldOption, 
       dbFileName); 
    } catch (IOException e) { 
     e.printStackTrace(); 
    } catch (GeoIp2Exception e) { 
     e.printStackTrace(); 
    } 
    return new Text(result); 

}

然後我做了一個罐子出來的這一點，跑在蜂巢以下CLI

add jar /location_of_jar/MyUDF.jar; 
add file /user/riyan/GeoIP2-Enterprise.mmdb; 
create temporary function samplefunction as 'com.package.name.App'; 
select samplefunction('172.73.14.54','country_name','/user/riyan/GeoIP2-Enterprise.mmdb') AS result;

我經過GeoIP2-Enterprise.mmdb數據庫的位置到UDF。它在我的本地系統上正常工作。但是，當我做罐子了出來，並在命令行中運行它，它給了我一個錯誤說

FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments ''/user/riyan/GeoIP2-Enterprise.mmdb'': org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public static org.apache.hadoop.io.Text com.package.name.App.evaluate(org.apache.hadoop.io.Text,org.apache.hadoop.io.Text,org.apache.hadoop.io.Text) on object [email protected] of class com.package.name.App with arguments {172.73.14.54:org.apache.hadoop.io.Text, country_name:org.apache.hadoop.io.Text, /user/riyan/GeoIP.mmdb:org.apache.hadoop.io.Text} of size 3

我也試圖將參數從文本更改爲字符串，並且給了我同樣的異常。 有人能告訴我我做錯了什麼嗎？ 感謝

編輯：添加以下部分

我跑了它在蜂巢調試模式，並根據您的answre得到這個

FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments ''./GeoIP2-Enterprise.mmdb'': org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String com.package.name.App.evaluate(java.lang.String,java.lang.String,java.lang.String) on object [email protected] of class com.package.name.App with arguments {172.73.14.54:java.lang.String, countryCode:java.lang.String, ./GeoIP2-Enterprise.mmdb:java.lang.String} of size 3 
17/04/18 11:02:30 [main]: ERROR ql.Driver: FAILED: SemanticException [Error 10014]: Line 1:7 Wrong arguments ''./GeoIP2-Enterprise.mmdb'': org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String com.bankofamerica.gisds.App.evaluate(java.lang.String,java.lang.String,java.lang.String) on object [email protected] of class com.package.name.App with arguments {172.73.14.54:java.lang.String, countryCode:java.lang.String, ./GeoIP2-Enterprise.mmdb:java.lang.String} of size 3 
org.apache.hadoop.hive.ql.parse.SemanticException: Line 1:7 Wrong arguments ''./GeoIP2-Enterprise.mmdb'': org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String com.package.name.App.evaluate(java.lang.String,java.lang.String,java.lang.String) on object [email protected] of class com.package.name.App with arguments {172.73.14.54:java.lang.String, countryCode:java.lang.String, ./GeoIP2-Enterprise.mmdb:java.lang.String} of size 3 
     at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1184) 
     at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90) 
     at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94) 
     at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78) 
     at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132) 
     at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109) 
     at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:193) 
     at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:146) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:10422) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:10378) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3771) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:3550) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:8830) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8785) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9652) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9545) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genOPTree(SemanticAnalyzer.java:10018) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10029) 
     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9909) 
     at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:223) 
     at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:488) 
     at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1274) 
     at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1391) 
     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1203) 
     at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1193) 
     at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:220) 
     at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:172) 
     at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:383) 
     at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:775) 
     at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:693) 
     at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:628) 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:606) 
     at org.apache.hadoop.util.RunJar.run(RunJar.java:221) 
     at org.apache.hadoop.util.RunJar.main(RunJar.java:136) 
Caused by: org.apache.hadoop.hive.ql.exec.UDFArgumentException: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String com.package.name.App.evaluate(java.lang.String,java.lang.String,java.lang.String) on object [email protected] of class com.package.name.App with arguments {172.73.14.54:java.lang.String, countryCode:java.lang.String, ./GeoIP2-Enterprise.mmdb:java.lang.String} of size 3 
     at org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:171) 
     at org.apache.hadoop.hive.ql.plan.ExprNodeGenericFuncDesc.newInstance(ExprNodeGenericFuncDesc.java:233) 
     at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:959) 
     at org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1176) 
     ... 36 more 
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to execute method public java.lang.String com.package.name.App.evaluate(java.lang.String,java.lang.String,java.lang.String) on object [email protected] of class com.package.name.App with arguments {172.73.14.54:java.lang.String, countryCode:java.lang.String, ./GeoIP2-Enterprise.mmdb:java.lang.String} of size 3 
     at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:978) 
     at org.apache.hadoop.hive.ql.udf.generic.GenericUDFBridge.evaluate(GenericUDFBridge.java:182) 
     at org.apache.hadoop.hive.ql.udf.generic.GenericUDF.initializeAndFoldConstants(GenericUDF.java:168) 
     ... 39 more 
Caused by: java.lang.reflect.InvocationTargetException 
     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
     at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
     at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
     at java.lang.reflect.Method.invoke(Method.java:606) 
     at org.apache.hadoop.hive.ql.exec.FunctionRegistry.invoke(FunctionRegistry.java:954) 
     ... 41 more 
Caused by: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.node.ObjectNode.<init>(Lcom/fasterxml/jackson/databind/node/JsonNodeFactory;Ljava/util/Map;)V 
     at com.maxmind.db.Decoder.decodeMap(Decoder.java:285) 
     at com.maxmind.db.Decoder.decodeByType(Decoder.java:154) 
     at com.maxmind.db.Decoder.decode(Decoder.java:147) 
     at com.maxmind.db.Decoder.decodeMap(Decoder.java:281) 
     at com.maxmind.db.Decoder.decodeByType(Decoder.java:154) 
     at com.maxmind.db.Decoder.decode(Decoder.java:147) 
     at com.maxmind.db.Decoder.decode(Decoder.java:87) 
     at com.maxmind.db.Reader.<init>(Reader.java:132) 
     at com.maxmind.db.Reader.<init>(Reader.java:116) 
     at com.maxmind.geoip2.DatabaseReader.<init>(DatabaseReader.java:35) 
     at com.maxmind.geoip2.DatabaseReader.<init>(DatabaseReader.java:23) 
     at com.maxmind.geoip2.DatabaseReader$Builder.build(DatabaseReader.java:129) 
     at com.bankofamerica.gisds.ExtractEnterpriseData.ExtractEnterpriseDB(ExtractEnterpriseData.java:27) 
     at com.package.name.App.evaluate(App.java:73) 
     ... 46 more

來源

2017-04-14 Riyan Mohammed

+0

這個問題解決了嗎？ –

回答

1

，它看起來像你缺少一些依賴你JAR文件。你如何編譯包含UDF的項目？

可能缺少這一個在蜂房類路徑

<dependency> 
    <groupId>com.fasterxml.jackson.core</groupId> 
    <artifactId>jackson-databind</artifactId> 
    <version>2.1.4</version> 
</dependency>

隨着工作的周圍，你可以嘗試使用具有依賴性的罐子（不是一個好的做法這種情況下）編譯它，但至少我們知道，如果這是您的問題

<build> 
    <plugins> 
    <plugin> 
     <artifactId>maven-assembly-plugin</artifactId> 
     <configuration> 
     <archive> 
      <manifest> 
      </manifest> 
     </archive> 
     <descriptorRefs> 
      <descriptorRef>jar-with-dependencies</descriptorRef> 
     </descriptorRefs> 
     </configuration> 
    </plugin> 
    </plugins> 
</build>

的另一種選擇是加入這個依賴蜂房類路徑，然後再試一次

https://mvnrepository.com/artifact/com.fasterxml.jackson.core/jackson-databind/2.1.4

來源

2017-04-14 19:21:18 hlagos

+0

沒有工作....它仍然給我同樣的錯誤...！ –

+0

它執行你的代碼嗎？取決於配置單元版本的錯誤可能不是很清楚，你可以嘗試一些虛擬邏輯，如返回一個參數？ – hlagos

+0

我添加了一個新的虛擬評估函數，它只返回一個字符串。它的工作正常。 實際的評估函數在本地機器上運行良好...如果我在集羣上運行，它給了我上述錯誤 –

0

失敗的真正原因是由於下面的錯誤。

所致：java.lang.NoSuchMethodError：com.fasterxml.jackson.databind.node.ObjectNode（LCOM/fasterxml /傑克遜/數據綁定/節點/ JsonNodeFactory; Ljava/util的/地圖;）V

看起來你可能在類路徑中存在衝突的jar版本。

add -verbose：在啓動它之前添加-verbose：hs2類的jvm選項，然後查看哪個jar被拾取 - 如果舊的或者udf不需要，則移除該jar。

來源

2017-12-10 08:46:04

相關問題

 相關問題