2016-11-05 18 views
1

我有兩個數據集:
用戶:無法indentify我減速的bug加入代碼

Bobby 06 Amsterdam 
Sunny 07 Rotterdam 
Steven 08 Liverpool 
Jamie 23 Liverpool 
Macca 91 Liverpool 
Messi 10 Barcelona 
Pique 04 Barcelona 
Suarez 09 Barcelona 
Neymar 11 brazil 
Klopp 12 Liverpool 

userlogs:

Sunny NewPlayer 12.23.14.421 
Klopp Crazy 88.33.44.555 
Bobby NewPlayer 99.12.11.222 
Steven Captain 99.55.66.777 
Jamie Local 88.99.33.232 
Suarez Spain 77.55.66.444 

我想用減速機加入到加入這兩個數據集。 我寫我的班是這樣的:

MapperClass:

Public class MapperClass { 
    public static class UserMap extends Mapper<LongWritable, Text, Text, Text> { 
     @Override 
     protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { 
      String line  = value.toString(); 
      String[] tokens = line.split(" "); 
      String name  = tokens[0]; 
      String city  = tokens[2]; 
      context.write(new Text(name), new Text("UserFile" + "\t" + city)); 
     } 
    } 

    public static class UserLogs extends Mapper<LongWritable, Text, Text, Text> { 
     @Override 
     protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException { 
      String line  = value.toString(); 
      String[] tokens = line.split(" "); 
      String name  = tokens[0]; 
      String ip  = tokens[2]; 
      context.write(new Text(name), new Text("UserLogs" + "\t" + ip)); 
     } 
    } 
} 

減速機類:

public class ReducerClass extends Reducer<Text, Text, Text, Text>{ 
    @Override 
    public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException { 
     String city = null; 
     String ip = null; 
     for(Text t: values) { 
      String[] parts = t.toString().split("\t"); 
      if(parts[0].equals("UserFile")) { 
       city = parts[1]; 
      } 
      if(parts[0].equals("UserLogs")) { 
       ip = parts[1]; 
      } else { 
       ip = "IP Address not found"; 
      } 
     } 
     context.write(key, new Text(city + "\t" + ip)); 
    } 
} 

Driverclass:

public class MainClass { 
    public static void main(String[] args)throws IOException, InterruptedException, ClassNotFoundException { 
     Job job = new Job(); 
     job.setJarByClass(MainClass.class); 
     job.setOutputKeyClass(Text.class); 
     job.setReducerClass(ReducerClass.class); 
     job.setOutputValueClass(Text.class); 
     job.setInputFormatClass(TextInputFormat.class); 
     job.setOutputFormatClass(TextOutputFormat.class); 
     MultipleInputs.addInputPath(job, new Path(args[0]), TextInputFormat.class, UserMap.class); 
     MultipleInputs.addInputPath(job, new Path(args[1]), TextInputFormat.class, UserLogs.class); 
     FileOutputFormat.setOutputPath(job, new Path(args[2])); 

     System.exit(job.waitForCompletion(true)?0:1); 
    } 
} 

輸出應該是這樣的:

Bobby Amsterdam 99.12.11.222 
Sunny Rotterdam 12.23.14.421 
Klopp Liverpool 88.33.44.555 
Steven Liverpool 99.55.66.777 
Jamie Liverpool 88.99.33.232 
Suarez Barcelona 77.55.66.444 

相反即時得到的輸出是這樣的:

Bobby Amsterdam  IP Address not found 
Jamie Liverpool  88.99.33.232 
Klopp Liverpool  IP Address not found 
Macca Liverpool  IP Address not found 
Messi Barcelona  IP Address not found 
Neymar brazil IP Address not found 
Pique Barcelona  IP Address not found 
Steven Liverpool  99.55.66.777 
Suarez Barcelona  IP Address not found 
Sunny Rotterdam  12.23.14.421 

我不明白我做了什麼在這裏做了錯誤。 任何人都可以幫助我解決這個問題。 任何形式的幫助真的很感激。

回答

0

減速器發生錯誤,取決於values順序,它覆蓋IP地址。試試這個:

public class ReducerClass extends Reducer<Text, Text, Text, Text>{ 
    @Override 
    public void reduce(Text key, Iterable<Text> values, Context context) throws IOException, InterruptedException { 
     String city = null; 
     String ip = null; 
     for(Text t: values) { 
      String[] parts = t.toString().split("\t"); 
      if(parts[0].equals("UserFile")) { 
       city = parts[1]; 
      } else if(parts[0].equals("UserLogs")) { 
       ip = parts[1]; 
      } 
     } 
     if (ip != null && city != null) { 
      context.write(key, new Text(city + "\t" + ip)); 
     } 
    } 
} 
+0

更正它並像魅力一樣工作。非常感謝Mariusz。 – Sidhartha