與JavaPairRDD實現的Hadoop Map作爲星火路

Question

我有一個RDD：

JavaPairRDD<Long, ViewRecord> myRDD 

它通過newAPIHadoopRDD方法創建的。我有我想要實現它在星火路一個已經存在的地圖功能：

LongWritable one = new LongWritable(1L); 

protected void map(Long key, ViewRecord viewRecord, Context context) 
    throws IOException ,InterruptedException { 

    String url = viewRecord.getUrl(); 
    long day = viewRecord.getDay(); 

    tuple.getKey().set(url); 
    tuple.getValue().set(day); 

    context.write(tuple, one); 
}; 

PS：元組來源於：

KeyValueWritable<Text, LongWritable> 

，並可以在這裏找到：TextLong.java

Answer 1

我不知道什麼元組是，但如果你只是想將記錄映射到關鍵字(url, day)和值1L的元組，你可以這樣做：

result = myRDD 
    .values() 
    .mapToPair(viewRecord -> { 
     String url = viewRecord.getUrl(); 
     long day = viewRecord.getDay(); 
     return new Tuple2<>(new Tuple2<>(url, day), 1L); 
    }) 


//java 7 style 
JavaPairRDD<Pair, Long> result = myRDD 
     .values() 
     .mapToPair(new PairFunction<ViewRecord, Pair, Long>() { 
         @Override 
         public Tuple2<Pair, Long> call(ViewRecord record) throws Exception { 
          String url = record.getUrl(); 
          Long day = record.getDay(); 

          return new Tuple2<>(new Pair(url, day), 1L); 
         } 
        } 
     );