如何在執行作業時將OracleDriver添加到hadoop類路徑中？

我正在使用hadoop處理來自數據庫的大量數據。我正在使用oracle的jdbc驅動程序連接到Oracle數據庫並執行處理。但是當我嘗試通過bin/hadoop與打包的JAR文件執行hadoop作業時，它顯示未找到OracleDriver類。我該如何解決？如何在執行作業時將OracleDriver添加到hadoop類路徑中？

$ bin/hadoop jar ~/hadoop1.jar name.hadoop.Hadoop1 ~/output

 

Exception in thread "main" java.lang.RuntimeException: Error in configuring object 
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93) 
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64) 
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) 
    at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:575) 
    at org.apache.hadoop.mapred.JobClient.writeOldSplits(JobClient.java:1051) 
    at org.apache.hadoop.mapred.JobClient.writeSplits(JobClient.java:1043) 
    at org.apache.hadoop.mapred.JobClient.access$700(JobClient.java:179) 
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:959) 
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:912) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at javax.security.auth.Subject.doAs(Subject.java:415) 
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1136) 
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:912) 
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:886) 
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1323) 
    at name.shahalpk.poc.hadoop.Hadoop1.main(Hadoop1.java:73) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:601) 
    at org.apache.hadoop.util.RunJar.main(RunJar.java:156) 
Caused by: java.lang.reflect.InvocationTargetException 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:601) 
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88) 
    ... 20 more 
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver 
    at org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:271) 
    ... 25 more 
Caused by: java.lang.ClassNotFoundException: oracle.jdbc.driver.OracleDriver 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:366) 
    at java.net.URLClassLoader$1.run(URLClassLoader.java:355) 
    at java.security.AccessController.doPrivileged(Native Method) 
    at java.net.URLClassLoader.findClass(URLClassLoader.java:354) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:423) 
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) 
    at java.lang.ClassLoader.loadClass(ClassLoader.java:356) 
    at java.lang.Class.forName0(Native Method) 
    at java.lang.Class.forName(Class.java:186) 
    at org.apache.hadoop.mapred.lib.db.DBConfiguration.getConnection(DBConfiguration.java:123) 
    at org.apache.hadoop.mapred.lib.db.DBInputFormat.configure(DBInputFormat.java:266) 
    ... 25 more

來源

2012-12-18 shahalpk

打印出你的異常？ –

@MohammodHossain更新！ – shahalpk

當你執行jar文件時發生異常，對嗎？ –

添加ojdbc5.jar到類路徑：

${JRE_HOME}\jre\lib\ext

注：

${JRE_HOME} means JRE(Java Runtime Environment) Installed Directory; Like below 
${JRE_HOME}=C:\Program Files\Java\jre6\

來源

2012-12-18 09:57:16

你的問題解決了嗎？ –

可以使用-libjars通用選項輕鬆實現這個多了，它也處理你的jar到集羣節點的分發：

$ bin/hadoop jar ~/hadoop1.jar name.hadoop.Hadoop1 -libjars ojdbc5.jar ~/output

這並假定你的主類（name.hadoop.Hadoop1）使用ToolRunner.run（）方法來啓動你的工作：

public class Hadoop1 extends Configured implements Tool { 
    public static void main(String args[]) throws Exception { 
    ToolRunner.run(new Hadoop1(), args); 
    } 

    public int run(String args[]) { 
    JobConf job = new JobConf(getConf()); 

    // rest of your job init code... 

    RunningJob rj = JobClient.runJob(job); 
    rj.waitForCompletion(); 
    return rj.isSuccessful() ? 0 : 1; 
    } 
}

（手動輸入代碼，表示歉意任何拼寫錯誤或編譯錯誤）

來源

2012-12-18 11:57:30

如何在執行作業時將OracleDriver添加到hadoop類路徑中？

回答

相關問題