1
我正在從元素類型爲com.google.gson.JsonObject的RDD中讀取數據。嘗試將其轉換爲DataSet,但不知道如何做到這一點。Spark如何將RDD [JSONObject]轉換爲數據集
import com.google.gson.{JsonParser}
import org.apache.hadoop.io.LongWritable
import org.apache.spark.sql.{SparkSession}
object tmp {
class people(name: String, age: Long, phone: String)
def main(args: Array[String]): Unit = {
val spark = SparkSession.builder().master("local[*]").getOrCreate()
val sc = spark.sparkContext
val parser = new JsonParser();
val jsonObject1 = parser.parse("""{"name":"abc","age":23,"phone":"0208"}""").getAsJsonObject()
val jsonObject2 = parser.parse("""{"name":"xyz","age":33}""").getAsJsonObject()
val PairRDD = sc.parallelize(List(
(new LongWritable(1l), jsonObject1),
(new LongWritable(2l), jsonObject2)
))
val rdd1 =PairRDD.map(element => element._2)
import spark.implicits._
//How to create Dataset as schema People from rdd1?
}
}
即使試圖打印RDD1集元素拋出
object not serializable (class: org.apache.hadoop.io.LongWritable, value: 1)
- field (class: scala.Tuple2, name: _1, type: class java.lang.Object)
- object (class scala.Tuple2, (1,{"name":"abc","age":23,"phone":"0208"}))
基本上,我得到這個RDD [LongWritable,JsonParser]從BigQuery表,我要轉換爲數據集,所以我可以申請SQL進行改造。
我已經故意將第二條記錄中的電話留空,BigQuery不會爲該元素返回空值。
由於鄰省出新的火花背景下,我已經編輯我的問題,如果,讓更多的想法。 – xstack2000