scala.collection.mutable.ArrayBuffer不能轉換爲java.lang.Double中（星火）

Question

我有這樣一個數據幀：

root 
|-- midx: double (nullable = true) 
|-- future: array (nullable = true) 
| |-- element: struct (containsNull = true) 
| | |-- _1: long (nullable = false) 
| | |-- _2: long (nullable = false) 

使用此代碼，我試圖將其轉移到這樣的事情：

val T = withFfutures.where($"midx" === 47.0).select("midx","future").collect().map((row: Row) => 
     Row { 
     row.getAs[Seq[Row]]("future").map { case Row(e: Long, f: Long) => 
      (row.getAs[Double]("midx"), e, f) 
     } 
     } 
    ).toList 

root 
|-- id: double (nullable = true) 
|-- event: long (nullable = true) 
|-- future: long (nullable = true) 

因此，計劃是將（event，future）數組轉換爲以這兩個字段爲列的數據框。我試圖T傳送到這樣一個數據幀：

val schema = StructType(Seq(
    StructField("id", DoubleType, nullable = true) 
    , StructField("event", LongType, nullable = true) 
    , StructField("future", LongType, nullable = true) 
)) 

val df = sqlContext.createDataFrame(context.parallelize(T), schema) 

但是，當我一個，試圖尋找到df我得到這個錯誤：

java.lang.ClassCastException: scala.collection.mutable.ArrayBuffer cannot be cast to java.lang.Double 

Answer 1

過了一會兒，我發現了什麼是問題：首先，列中的數組結構應該被轉換爲Row。因此，構建最終數據幀的最終代碼應如下所示：

val T = withFfutures.select("midx","future").collect().flatMap((row: Row) => 
    row.getAs[Seq[Row]]("future").map { case Row(e: Long, f: Long) => 
     (row.getAs[Double]("midx") , e, f) 
    }.toList 
).toList 

val all = context.parallelize(T).toDF("id","event","future")