在寫入JSON的時候保留鍵值爲空的值

Question

我在用spark寫一個JSON文件。有一些鍵值爲null。這些在DataSet中顯示得很好，但是當我寫入文件時，密鑰被丟棄。我如何確保他們被保留？

代碼寫入文件：從源JSON數據的

ddp.coalesce(20).write().mode("overwrite").json("hdfs://localhost:9000/user/dedupe_employee"); 

部分：

"event_header": { 
     "accept_language": null, 
     "app_id": "App_ID", 
     "app_name": null, 
     "client_ip_address": "IP", 
     "event_id": "ID", 
     "event_timestamp": null, 
     "offering_id": "Offering", 
     "server_ip_address": "IP", 
     "server_timestamp": 1492565987565, 
     "topic_name": "Topic", 
     "version": "1.0" 
    } 

輸出：

"event_header": { 
     "app_id": "App_ID", 
     "client_ip_address": "IP", 
     "event_id": "ID", 
     "offering_id": "Offering", 
     "server_ip_address": "IP", 
     "server_timestamp": 1492565987565, 
     "topic_name": "Topic", 
     "version": "1.0" 
    } 

在上面的例子鍵accept_language，app_name和event_timestamp已被放棄。

Answer 1

顯然，火花不提供任何選項來處理空​​值。所以下面的自定義解決方案應該工作

import com.fasterxml.jackson.module.scala.DefaultScalaModule 
import com.fasterxml.jackson.module.scala.experimental.ScalaObjectMapper 
import com.fasterxml.jackson.databind.ObjectMapper 

case class EventHeader(accept_language:String,app_id:String,app_name:String,client_ip_address:String,event_id: String,event_timestamp:String,offering_id:String,server_ip_address:String,server_timestamp:Long,topic_name:String,version:String) 

val ds = Seq(EventHeader(null,"App_ID",null,"IP","ID",null,"Offering","IP",1492565987565L,"Topic","1.0")).toDS() 

val ds1 = ds.mapPartitions(records => { 
val mapper = new ObjectMapper with ScalaObjectMapper 
mapper.registerModule(DefaultScalaModule) 
records.map(mapper.writeValueAsString(_)) 
}) 

ds1.coalesce(1).write.text("hdfs://localhost:9000/user/dedupe_employee") 

這將產生輸出爲：

{"accept_language":null,"app_id":"App_ID","app_name":null,"client_ip_address":"IP","event_id":"ID","event_timestamp":null,"offering_id":"Offering","server_ip_address":"IP","server_timestamp":1492565987565,"topic_name":"Topic","version":"1.0"}