錯誤，同時批量加載hbase

我正在嘗試Hbase - bulkLoad通過Java MapReduce程序。我正在Eclipse中運行我的程序。錯誤，同時批量加載hbase

但我收到以下錯誤：

12/06/14 20:04:28 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId= 
12/06/14 20:04:28 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 
12/06/14 20:04:29 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same. 
12/06/14 20:04:29 WARN mapred.JobClient: No job jar file set. User classes may not be found. See JobConf(Class) or JobConf#setJar(String). 
12/06/14 20:04:29 INFO input.FileInputFormat: Total input paths to process : 1 
12/06/14 20:04:29 WARN snappy.LoadSnappy: Snappy native library not loaded 
12/06/14 20:04:29 INFO mapred.JobClient: Running job: job_local_0001 
12/06/14 20:04:29 INFO mapred.MapTask: io.sort.mb = 100 
12/06/14 20:04:29 INFO mapred.MapTask: data buffer = 79691776/99614720 
12/06/14 20:04:29 INFO mapred.MapTask: record buffer = 262144/327680 
12/06/14 20:04:29 WARN mapred.LocalJobRunner: job_local_0001 
java.lang.IllegalArgumentException: Can't read partitions file 
    at org.apache.hadoop.hbase.mapreduce.hadoopbackport.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:111) 
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) 
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117) 
    at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:560) 
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:639) 
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:323) 
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210) 
Caused by: java.io.FileNotFoundException: File _partition.lst does not exist. 
    at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:383) 
    at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251) 
    at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:776) 
    at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1424) 
    at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1419) 
    at org.apache.hadoop.hbase.mapreduce.hadoopbackport.TotalOrderPartitioner.readPartitions(TotalOrderPartitioner.java:296) 
    at org.apache.hadoop.hbase.mapreduce.hadoopbackport.TotalOrderPartitioner.setConf(TotalOrderPartitioner.java:82) 
    ... 6 more 
12/06/14 20:04:30 INFO mapred.JobClient: map 0% reduce 0% 
12/06/14 20:04:30 INFO mapred.JobClient: Job complete: job_local_0001 
12/06/14 20:04:30 INFO mapred.JobClient: Counters: 0

我GOOGLE了很多，但沒有找到任何解決辦法。

我試圖從控制檯運行此相同的程序和下面的錯誤來了：

hadoop jar /home/user/hbase-0.90.4-cdh3u2/lib/zookeeper-3.3.3-cdh3u2.jar /home/user/hadoop-0.20.2-cdh3u2/Test.jar BulkLoadHBase_1 /bulkLoad.txt /out 
Exception in thread "main" java.lang.NoSuchMethodException: org.apache.zookeeper.server.quorum.QuorumPeer.main([Ljava.lang.String;) 
    at java.lang.Class.getMethod(Class.java:1605) 
    at org.apache.hadoop.util.RunJar.main(RunJar.java:180)

我的代碼：

import java.io.IOException; 
import org.apache.hadoop.conf.Configuration; 
import org.apache.hadoop.fs.Path; 
import org.apache.hadoop.hbase.HBaseConfiguration; 
import org.apache.hadoop.hbase.client.Put; 
import org.apache.hadoop.hbase.io.ImmutableBytesWritable; 
import org.apache.hadoop.hbase.mapreduce.HFileOutputFormat; 
import org.apache.hadoop.hbase.mapreduce.PutSortReducer; 
import org.apache.hadoop.hbase.mapreduce.hadoopbackport.TotalOrderPartitioner; 
import org.apache.hadoop.hbase.util.Bytes; 
import org.apache.hadoop.io.*; 
import org.apache.hadoop.mapreduce.Job; 
import org.apache.hadoop.mapreduce.Mapper; 
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat; 
import org.apache.hadoop.mapreduce.lib.input.KeyValueTextInputFormat; 

public class BulkLoadHBase_1 { 

    public static class BulkLoadHBase_1Mapper 
      extends Mapper<Text, Text, ImmutableBytesWritable, Put>{ 

     public void map(Text key, Text value, Context context 
         ) throws IOException, InterruptedException { 

      System.out.println("KEY "+key.toString()); 
      System.out.println("VALUES : "+value); 
      System.out.println("Context : "+context); 

      ImmutableBytesWritable ibw = 
        new ImmutableBytesWritable(Bytes.toBytes(key.toString())); 

      String val = value.toString(); 
      byte[] b = Bytes.toBytes(val); 
      Put p = new Put(Bytes.toBytes(key.toString())); 

      p.add(Bytes.toBytes("cf"),Bytes.toBytes("c"),Bytes.toBytes(val)); 

      context.write(ibw, p); 
     } 
    } 

    public static void main(String[] args) throws Exception { 
     Configuration conf = new Configuration(); 

     Job job = new Job(conf, "bulk-load"); 

     job.setJarByClass(BulkLoadHBase_1.class); 
     job.setMapperClass(BulkLoadHBase_1Mapper.class); 

     job.setReducerClass(PutSortReducer.class); 
     job.setOutputKeyClass(ImmutableBytesWritable.class); 
     job.setOutputValueClass(Put.class); 
     job.setPartitionerClass(TotalOrderPartitioner.class); 
     job.setInputFormatClass(KeyValueTextInputFormat.class); 

     FileInputFormat.addInputPath(job, 
        new Path("/home/user/Desktop/bulkLoad.txt")); 
     HFileOutputFormat.setOutputPath(job, 
        new Path("/home/user/Desktop/HBASE_BulkOutput/"));  

     System.exit(job.waitForCompletion(true) ? 0 : 1); 
    } 
}

來源

2012-06-14 pradeep

的問題是：這個程序需要在分佈式模式中運行。並要求JAR應該發貨...

來源

2012-10-15 10:27:55 pradeep

你在分佈式模式下啓動的HBase？如果是該行：

org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:210)

在您的堆棧跟蹤顯示地圖降低作業在本地模式，而不是分佈模式運行。

另請注意，如果您要從控制檯內運行命令，您的輸入文件必須駐留在您的常規（例如NTFS或EXT3）文件系統上的hadoop文件系統上。

問候

來源

2012-06-21 08:25:04

有問題機智jar文件。 – pradeep

不，我在本地模式下運行... 但之後，我搖擺到分佈式模式...然後JAR概率進入圖片.. – pradeep

錯誤，同時批量加載hbase

回答

相關問題