Spark列中的數組的每個值的映射

Question

我有以下格式的json數據集，每行一個條目。

{ "sales_person_name" : "John", "products" : ["apple", "mango", "guava"]} 
{ "sales_person_name" : "Tom", "products" : ["mango", "orange"]} 
{ "sales_person_name" : "John", "products" : ["apple", "banana"]} 
{ "sales_person_name" : "Steve", "products" : ["apple", "mango"]} 
{ "sales_person_name" : "Tom", "products" : ["mango", "guava"]} 

我想知道誰賣了最大的芒果等等。 因此，我想將文件加載到數據框，併爲每個事務發出陣列中每個產品值的（鍵，值）（產品，名稱）對。

var df = spark.read.json("s3n://sales-data.json") 
df.printSchema() 
root 
|-- sales_person_name: string (nullable = true) 
|-- products: array (nullable = true) 

var nameProductsMap = df.select("sales_person_name", "products").show() 
+-----------------+--------------------+ 
|sales_person_name| products   | 
+-----------------+--------------------+ 
|    John|[mango, apple,... | 
|    Tom|[mango, orange,... | 
|    John|[apple, banana... | 

var resultMap = df.select("products", "sales_person_name") 
        .map(r => (r(1), r(0))) 
        .show() //This is where I am stuck. 

我無法找出正確的方式爆炸（）行（0），並有它的所有行（1）值一次發射值。任何人都可以提出一種方法謝謝！

Answer 1

UPDATE

下面的代碼應該工作

import org.apache.spark.sql.functions.explode 
import scala.collection.mutable 

val resultMap = df.select(explode($"products"), $"sales_person_name") 


def counter(l: TraversableOnce[Any]) = { 
    val temp = mutable.Map[Any, Int]() 
    for (i <- l) { 
     if(temp.contains(i)) temp(i) += 1 
     else temp(i) = 1 
    } 
    temp 
} 

resultsMap.map(x => (x(0), Array(x(1)))). 
      reduceByKey(_ ++ _). 
      map { case (x,y) => (x, counter(y).toArray) } 

所得輸出：Array((banana,Array((John,1))), (guava,Array((Tom,1), (John,1))), (orange,Array((Tom,1))), (apple,Array((Steve,1), (John,2))), (mango,Array((Tom,2), (Steve,1), (John,1))))

Answer 2

val exploded = df.explode("products", "product") { a: mutable.WrappedArray[String] => a } 
val result = exploded.drop("products") 
result.show() 

打印：

+-----------------+-------+ 
|sales_person_name|product| 
+-----------------+-------+ 
|    John| apple| 
|    John| mango| 
|    John| guava| 
|    Tom| mango| 
|    Tom| orange| 
|    John| apple| 
|    John| banana| 
|   Steve| apple| 
|   Steve| mango| 
|    Tom| mango| 
|    Tom| guava| 
+-----------------+-------+