如何使用Java Hadoop map reduce以降序對數據集中的列進行排序？

我的數據文件是：如何使用Java Hadoop map reduce以降序對數據集中的列進行排序？

Utsav Chatterjee Dangerous Soccer Coldplay 4 
Rodney Purtle Awesome Football Maroon5 3 
Michael Gross Amazing Basketball Iron Maiden 6 
Emmanuel Ezeigwe Cool Pool Metallica 5 
John Doe Boring Golf Linkin Park 8 
David Bekham Godlike Soccer Justin Beiber 89 
Abhishek Kumar Geek Cricket Abhishek Kumar 7 
Abhishek Singh Geek Cricket Abhishek Kumar 7

我想通過列號作爲參數調用時Hadoop的罐子，我需要整個數據集的基礎上按降序排列特定列進行排序。我可以通過將所需列設置爲映射器輸出中的鍵，以升序順序輕鬆完成此操作。但是，我無法按降序完成此操作。

我的映射器和減速代碼：

public static class Map extends Mapper<LongWritable,Text,Text,Text>{ 
     public static void map(LongWritable key, Text value, Context context) 
     throws IOException,InterruptedException 
     { 
      Configuration conf = context.getConfiguration(); 
      String param = conf.get("columnRef"); 
      int colref = Integer.parseInt(param); 
      String line = value.toString(); 
      String[] parts = line.split("\t"); 
      context.write(new Text(parts[colref]), value); 
      } 
     } 

    public static class Reduce extends Reducer<Text,Text,Text,Text>{ 
     public void reduce(Text key, Iterable<Text> value, Context context) 
     throws IOException,InterruptedException 
     { 
      for (Text text : value) { 
       context.write(text,null); 
      } 
     } 
    }

我比較類：

public static class sortComparator extends WritableComparator { 

     protected sortComparator() { 
      super(LongWritable.class, true); 
      // TODO Auto-generated constructor stub 
     } 

     @Override 
     public int compare(WritableComparable o1, WritableComparable o2) { 
      LongWritable k1 = (LongWritable) o1; 
      LongWritable k2 = (LongWritable) o2; 
      int cmp = k1.compareTo(k2); 
      return -1 * cmp; 
     } 

     }

我可能做錯事與比較。有人可以幫我從這裏出去嗎？當我運行這個時，選擇索引爲5的列（數字最後一列）作爲此類的基礎，我仍然按升序得到結果。

Driver類：

public static void main(String[] args) throws Exception { 

     Configuration conf= new Configuration(); 
     conf.set("columnRef", args[2]); 

     Job job = new Job(conf, "Sort"); 

     job.setJarByClass(Sort.class); 
     job.setMapperClass(Map.class); 
     job.setReducerClass(Reduce.class); 

     job.setOutputKeyClass(Text.class); 
     job.setOutputValueClass(Text.class); 
     job.setSortComparatorClass(DescendingKeyComparator.class); 
     job.setInputFormatClass(TextInputFormat.class); 
     job.setOutputFormatClass(TextOutputFormat.class); 

     Path outputPath = new Path(args[1]); 

     FileInputFormat.addInputPath(job, new Path(args[0])); 
     FileOutputFormat.setOutputPath(job, new Path(args[1])); 

     outputPath.getFileSystem(conf).delete(outputPath); 

     System.exit(job.waitForCompletion(true) ? 0 : 1); 
    }

如何可能能夠實現這個任務（降序）將是我非常有幫助的任何建議！感謝

來源

2016-03-05 Utsav Chatterjee

在你的驅動程序類，下面的代碼行： job.setSortComparatorClass(DescendingKeyComparator.class);

您已設置類作爲DescendingKeyComparator.class。將其設置爲sortComparator.class。它應該工作。

來源

2016-10-23 05:03:29 himi

如何使用Java Hadoop map reduce以降序對數據集中的列進行排序？

回答

相關問題