hadoop版本不兼容

我在Hadoop 0.20.2版本上開發了一個帶reduce方聯接的MapReduce。所有的java類都編譯時沒有任何錯誤。我的程序正常運行。但是這個hadoop版本在測試虛擬機上。在我的真實集羣上，我有另一個版本的hadoop（hadoop 2.00-cdh4.1.2）。當我想編譯java類時，它不起作用。我得到100錯誤如下：hadoop版本不兼容

JoinMapper.java:8: package org.apache.hadoop.filecache does not exist 
import org.apache.hadoop.filecache.DistributedCache; 
           ^
JoinMapper.java:9: package org.apache.hadoop.fs does not exist 
import org.apache.hadoop.fs.Path; 
         ^
JoinMapper.java:10: package org.apache.hadoop.io does not exist 
import org.apache.hadoop.io.Text; 
         ^
JoinMapper.java:11: package org.apache.hadoop.io does not exist 
import org.apache.hadoop.io.*; 
^ 
JoinMapper.java:12: package org.apache.hadoop.mapred does not exist 
import org.apache.hadoop.mapred.FileInputFormat; 
          ^
JoinMapper.java:13: package org.apache.hadoop.mapred does not exist 
import org.apache.hadoop.mapred.FileOutputFormat; 
          ^
JoinMapper.java:14: package org.apache.hadoop.mapred does not exist 
import org.apache.hadoop.mapred.InputFormat; 
          ^
JoinMapper.java:15: package org.apache.hadoop.mapred does not exist 
import org.apache.hadoop.mapred.JobClient; 
          ^
JoinMapper.java:16: package org.apache.hadoop.mapred does not exist 
import org.apache.hadoop.mapred.JobConf; 
          ^
JoinMapper.java:17: package org.apache.hadoop.mapred does not exist 
import org.apache.hadoop.mapred.MapReduceBase; 
          ^
JoinMapper.java:18: package org.apache.hadoop.mapred does not exist 
import org.apache.hadoop.mapred.Mapper; 
          ^
JoinMapper.java:19: package org.apache.hadoop.mapred does not exist 
import org.apache.hadoop.mapred.OutputCollector; 
          ^
JoinMapper.java:20: package org.apache.hadoop.mapred does not exist 
import org.apache.hadoop.mapred.Reporter; 
          ^
JoinMapper.java:21: package org.apache.hadoop.mapred does not exist 
import org.apache.hadoop.mapred.TextOutputFormat; 
          ^
JoinMapper.java:22: package org.apache.hadoop.mapreduce.lib.input does not exist 
import org.apache.hadoop.mapreduce.lib.input.MultipleInputs; 
              ^
JoinMapper.java:23: package org.apache.hadoop.util does not exist 
import org.apache.hadoop.util.Tool; 
          ^
JoinMapper.java:24: package org.apache.hadoop.util does not exist 
import org.apache.hadoop.util.ToolRunner; 
          ^
JoinMapper.java:26: cannot find symbol 
symbol: class MapReduceBase 
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> { 
           ^
JoinMapper.java:26: cannot find symbol 
symbol: class Mapper 
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> { 
                 ^
JoinMapper.java:26: cannot find symbol 
symbol: class LongWritable 
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> { 
                   ^
JoinMapper.java:26: cannot find symbol 
symbol: class Text 
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> { 
                      ^
TextPair.java:2: package org.apache.hadoop.io does not exist 
import org.apache.hadoop.io.*; 
^ 
TextPair.java:4: cannot find symbol 
symbol: class WritableComparable 
public class TextPair implements WritableComparable<TextPair> { 
           ^
TextPair.java:4: interface expected here 
public class TextPair implements WritableComparable<TextPair> { 
               ^
JoinMapper.java:26: cannot find symbol 
symbol: class Text 
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> { 
                          ^
JoinMapper.java:26: interface expected here 
public class JoinMapper extends MapReduceBase implements Mapper<LongWritable, Text, TextPair, Text> { 
                  ^
NcdcRecordParser.java:2: package org.apache.hadoop.io does not exist 
import org.apache.hadoop.io.*;

我明白這個版本沒有我需要的軟件包。但我不知道如何解決我的問題。誰能幫我？

來源

2012-12-26 Tanya

什麼罐子已經在你的classpath中？ –

你是如何編譯的？（什麼工具，Maven？如果是這樣，你用什麼依賴關係來試着用2.00進行編譯？） –

對於maven，試試param -Dhadoop.profile = 23。另外，請參閱https://issues.apache.org/jira/browse/HBASE-4327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098596#comment-13098596

來源

2012-12-26 17:46:01 hudvin

據我所知，這些Java包仍然應該存在於最新版本的Hadoop中。

我認爲你的問題是，hadoop罐不在你的classpath中。當我嘗試在沒有它們的情況下編譯我的map reduce項目時，出現類似的錯誤。

如果您正在使用javac，嘗試這樣的事情（這使用路徑位置我的Ubuntu CDH4 dev的機器）：

javac -classpath /usr/lib/hadoop/*:/usr/lib/hadoop-0.20-mapreduce/*:/usr/lib/hadoop-hdfs/* JoinMapper.java TextPair.java NcdcRecordParser.java

來源

2012-12-26 17:59:23

謝謝..我昨天也是這樣做了。 – Tanya

的Apache的Hadoop做0.20之間及其內部包的主要的重新命名和0.23（也被稱爲2.0.0）。這包括將org.apache.hadoop.mapred重命名爲org.apache.hadoop.mapreduce。

Hadoop的Cloudera發行版具有單獨的Maven構件，您可以依靠它們與舊版本進行構建。工件名稱在CDH4 Maven page上列出;你想要版本2.0.0-mr1-cdh4.1.2的文物。

另請參閱Is it better to use the mapred or the mapreduce package to create a Hadoop Job?瞭解有關是否需要移植到較新的API的一些詳細信息。

來源

2012-12-26 18:48:46

hadoop版本不兼容

回答

相關問題