FlatMap在Java中與星火2.1.0

Question

我試圖從Java 8火花2.1.0到flatMap與

2.2.0文檔顯示這個例子

JavaDStream<String> words = lines.flatMap(x -> Arrays.asList(x.split(" ")).iterator()); 

當我嘗試從2.1 .0我得到以下內容

Error:(31, 25) java: method flatMap in class org.apache.spark.rdd.RDD<T> cannot be applied to given types; 
required: scala.Function1<java.lang.String,scala.collection.TraversableOnce<U>>,scala.reflect.ClassTag<U> 
found: (x)->Array[...]tor() 
reason: cannot infer type-variable(s) U 
(actual and formal argument lists differ in length) 

給出這些版本的flatMap的正確方法是什麼？

Answer 1

下面的代碼適用於Spark 2.1.0。

JavaDStream<String> lines = messages.map(tuple -> tuple._2()); 
JavaDStream<String> words = lines.flatMap(x -> Arrays.asList(SPACE.split(x)).iterator()); 
JavaPairDStream<String, Integer> wordCounts = words.mapToPair(s -> new Tuple2<>(s, 1)) 
    .reduceByKey((i1, i2) -> i1 + i2); 

請檢查你的pom.xml火花依賴版本。如果你想參考Spark 2.1.0版本的例子，請轉到https://github.com/apache/spark/tree/branch-2.1/examples/src/main/java/org/apache/spark/examples/streaming