斯卡拉星火 - 調用createDataFrame

-1

當超載方法我嘗試從雙陣列像下面的Array（數組[數組[雙]）創建一個數據框：斯卡拉星火 - 調用createDataFrame

val points : ArrayBuffer[Array[Double]] = ArrayBuffer(
Array(0.19238990024216676, 1.0, 0.0, 0.0), 
Array(0.2864319929878242, 0.0, 1.0, 0.0), 
Array(0.11160349352921925, 0.0, 2.0, 1.0), 
Array(0.3659220026496052, 2.0, 2.0, 0.0), 
Array(0.31809629470827383, 1.0, 1.0, 1.0)) 

val x = Array("__1", "__2", "__3", "__4") 
val myschema = StructType(x.map(fieldName ⇒ StructField(fieldName, DoubleType, true))) 

points.map(e => Row(e(0), e(1), e(2), e(3))) 
val newDF = sqlContext.createDataFrame(points, myschema)

但得到這個錯誤：

<console>:113: error: overloaded method value createDataFrame with alternatives: 
(data: java.util.List[_],beanClass: Class[_])org.apache.spark.sql.DataFrame <and> 
(rdd: org.apache.spark.api.java.JavaRDD[_],beanClass: Class[_])org.apache.spark.sql.DataFrame <and> 
(rdd: org.apache.spark.rdd.RDD[_],beanClass: Class[_])org.apache.spark.sql.DataFrame <and> 
(rows: java.util.List[org.apache.spark.sql.Row],schema: org.apache.spark.sql.types.StructType)org.apache.spark.sql.DataFrame <and> 
(rowRDD: org.apache.spark.api.java.JavaRDD[org.apache.spark.sql.Row],schema: org.apache.spark.sql.types.StructType)org.apache.spark.sql.DataFrame <and> 
(rowRDD: org.apache.spark.rdd.RDD[org.apache.spark.sql.Row],schema: org.apache.spark.sql.types.StructType)org.apache.spark.sql.DataFrame 
cannot be applied to (scala.collection.mutable.ArrayBuffer[Array[Double]], org.apache.spark.sql.types.StructType) 
val newDF = sqlContext.createDataFrame(points, myschema)

我在互聯網上搜索，但無法找到如何解決它！所以如果有人對此有任何想法，請幫助我！

來源

2017-02-13 zock king

-1

這個工作對我來說：

import org.apache.spark.sql._ 
import org.apache.spark.sql.types._ 
import scala.collection.mutable.ArrayBuffer 

val sqlContext = new org.apache.spark.sql.SQLContext(sc) 

val points : ArrayBuffer[Array[Double]] = ArrayBuffer(
    Array(0.19238990024216676, 1.0, 0.0, 0.0), 
    Array(0.2864319929878242, 0.0, 1.0, 0.0), 
    Array(0.11160349352921925, 0.0, 2.0, 1.0), 
    Array(0.3659220026496052, 2.0, 2.0, 0.0), 
    Array(0.31809629470827383, 1.0, 1.0, 1.0)) 

val x = Array("__1", "__2", "__3", "__4") 
val myschema = StructType(x.map(fieldName ⇒ StructField(fieldName, DoubleType, true))) 

val rdd = sc.parallelize(points.map(e => Row(e(0), e(1), e(2), e(3)))) 
val newDF = sqlContext.createDataFrame(rdd, myschema) 

newDF.show

來源

2017-02-13 15:16:55 semsorock

爲什麼downvoted我會欣賞一個解釋。謝謝 – semsorock

謝謝！這適用於我！ –

接受ArrayBuffer[Array[Double]]實例的方法createDataFrame沒有超載。您撥打points.map的電話沒有被分配給任何內容，它將返回一個新實例，而不是在原地進行操作。嘗試：

val points : List[Array[Double]] = List(
    Seq(0.19238990024216676, 1.0, 0.0, 0.0), 
    Seq(0.2864319929878242, 0.0, 1.0, 0.0), 
    Seq(0.11160349352921925, 0.0, 2.0, 1.0), 
    Seq(0.3659220026496052, 2.0, 2.0, 0.0), 
    Seq(0.31809629470827383, 1.0, 1.0, 1.0)) 

val x = Array("__1", "__2", "__3", "__4") 
val myschema = StructType(x.map(fieldName ⇒ StructField(fieldName, DoubleType, true))) 

val newDF = sqlContext.createDataFrame(
    points.map(Row.fromSeq(_), myschema)

來源

2017-02-13 14:53:53 ImDarrenG

斯卡拉星火 - 調用createDataFrame

回答

相關問題