Hive在單個文件中拆分兩個相同的Jsons，以逗號分隔

Question

我有兩個Jsons，它們如下所示。我們的生產系統將多個Jsons聚合爲單個文件，如下所示。我的問題在於如何拆分由逗號分隔的單個jsons（i，e 1 & 2）並從文件中刪除3。 1,2和3之間的差爲3多年平均值具備的條件ID

1---{name:string, 
     actid: string, 
     data:array<struct>, 
     conditionId:string 
    }, 
2---{name:string, 
     actid: string, 
     data:array<struct>, 
     conditionId:string 
    }, 
3---{name:string, 
     actid:string, 
     data:array<struct> 
    } 

Answer 1

如果你有數據的文本文件作爲

1--- {name:string, actid: string, data:array, conditionId:string }, 2---{name:string, actid: string, data:array, conditionId:string}, 3---{name:string, actid:string, data:array} 

如果你正在尋找filter畸形的JSON作爲3和將文件保存爲有效json文件，那麼你可以在spark

val data = sc.textFile("path to the malformed input file") 
data.flatMap(line => line.split("---")) 
    .map(array => array.split(",")) 
    .filter(_.length > 3) 
    .map(array => conditions(array(0).split(":")(1), array(1).split(":")(1), array(2).split(":")(1), array(3).split(":")(1).replace("}", ""))) 
    .toDF() 
    .write.json("path to output json file") 

你應該甲肝使用如下代碼e爲有效的JSON文件作爲

{"name":"string","actid":" string","data":"array","conditionId":"string "} 
{"name":"string","actid":" string","data":"array","conditionId":"string"}