火花斯卡拉 - 合併多發性行成一個

Question

我有一個數據幀

|--id:string (nullable = true) 
|--ddd:struct (nullable = true) 
    |-- aaa: string (nullable = true) 
    |-- bbb: long(nullable = true) 
    |-- ccc: string (nullable = true) 
    |-- eee: long(nullable = true) 

我有這樣的

id  | ddd 
-------------------------- 
    1 | [hi,1,this,2] 
    2 | [hello,6,good,3] 
    1 | [hru,2,where,7] 
    3 | [in,4,you,1] 
    2 | [how,4,to,3] 

我想預期的O/P輸出：

id | ddd 
    -------------------- 
    1 | [hi,1,this,2],[hru,2,where,7] 
    2 | [hello,6,good,3],[how,4,to,3] 
    3 | [in,4,you,1] 

請幫忙

Answer 1

你可以collect_list如下

import org.apache.spark.sql.functions._ 
df.groupBy("id").agg(collect_list("ddd").as("ddd")) 

collect_set作品，以及

df.groupBy("id").agg(collect_set("ddd").as("ddd"))