如何變換數據集<Row>另一個

Question

我的數據集有一個名爲列「鍵（串），價值（長）」

像prefix.20171012.111.2222列項的值，和列值的9999價值

我想將數據集轉換成一個新的數據集，將colmun鍵拆分爲像「day，rt，item_id，value」這樣的其他數據集。

如何做到這一點，非常感謝

Answer 1

// input ds looks like this 
+--------+-----+ 
|  key|value| 
+--------+-----+ 
|20171011| 9999| 
+--------+-----+ 

//import the functions you need 
import org.apache.spark.sql.functions.{to_date, month, year, dayofmonth} 

// ds2 
val ds2 = ds.withColumn("date", to_date($"key", "yyyyMMdd")) 

// ds2.show() 
+--------+-----+----------+ 
|  key|value|  date| 
+--------+-----+----------+ 
|20171011| 9999|2017-10-11| 
+--------+-----+----------+ 

// ds3 
val ds3 = ds2.withColumn("Month", month($"date")) 
    .withColumn("Year", year($"date")) 
    .withColumn("Date", dayofmonth($"date")) 

// ds3.show() 
+--------+-----+----+-----+----+ 
|  key|value|Date|Month|Year| 
+--------+-----+----+-----+----+ 
|20171011| 9999| 11| 10|2017| 
+--------+-----+----+-----+----+