乾淨缺失值火花與聚合函數

Question

我想通過替換他們的意思來清除缺少的值。這個源代碼用於工作我不爲什麼，它現在不工作。任何幫助將不勝感激。 這裏是集我使用

RowNumber,Poids,Age,Taille,0MI,Hmean,CoocParam,LdpParam,Test2,Classe 
0,,72,160,5,,2.9421,,3,4 
1,54,70,,5,0.6301,2.7273,,3, 
2,,51,164,5,,2.9834,,3,4 
3,,74,170,5,0.6966,2.9654,2.3699,3,4 
4,108,62,,5,0.6087,2.7093,2.1619,3,4 

這裏我做了什麼

val spark = SparkSession.builder.master("local").appName("my-spark-app").getOrCreate() 

    val df = spark.read.option("header", true).option("inferSchema", true).format("com.databricks.spark.csv").load("C:/Users/mhattabi/Desktop/data_with_missing_values3.csv") 
    df.show(false) 
    var newDF = df 
    df.dtypes.foreach { x => 
     val colName = x._1 
     newDF = newDF.na.fill(df.agg(max(colName)).first()(0).toString, Seq(colName)) 
    } 
    newDF.show(false) 

下面是結果，什麼都沒有發生

initial_data 

    +---------+-----+---+------+---+------+---------+--------+-----+------+ 
    |RowNumber|Poids|Age|Taille|0MI|Hmean |CoocParam|LdpParam|Test2|Classe| 
    +---------+-----+---+------+---+------+---------+--------+-----+------+ 
    |0  |null |72 |160 |5 |null |2.9421 |null |3 |4  | 
    |1  |54 |70 |null |5 |0.6301|2.7273 |null |3 |null | 
    |2  |null |51 |164 |5 |null |2.9834 |null |3 |4  | 
    |3  |null |74 |170 |5 |0.6966|2.9654 |2.3699 |3 |4  | 
    |4  |108 |62 |null |5 |0.6087|2.7093 |2.1619 |3 |4  | 
    +---------+-----+---+------+---+------+---------+--------+-----+------+ 

    new_data 
    +---------+-----+---+------+---+------+---------+--------+-----+------+ 
    |RowNumber|Poids|Age|Taille|0MI|Hmean |CoocParam|LdpParam|Test2|Classe| 
    +---------+-----+---+------+---+------+---------+--------+-----+------+ 
    |0  |null |72 |160 |5 |null |2.9421 |null |3 |4  | 
    |1  |54 |70 |null |5 |0.6301|2.7273 |null |3 |null | 
    |2  |null |51 |164 |5 |null |2.9834 |null |3 |4  | 
    |3  |null |74 |170 |5 |0.6966|2.9654 |2.3699 |3 |4  | 
    |4  |108 |62 |null |5 |0.6087|2.7093 |2.1619 |3 |4  | 
    +---------+-----+---+------+---+------+---------+--------+-----+------+ 

我應該做的

Answer 1

您可以使用withColumn API和使用when功能在columns作爲

df.dtypes.foreach { x => 
     val colName = x._1 
     val fill = df.agg(max(col(s"`$colName`"))).first()(0).toString 
     newDF = newDF.withColumn(colName, when(col(s"`$colName`").isNull , fill).otherwise(col(s"`$colName`"))) 
    } 
    newDF.show(false) 

我希望這能解決您的問題，以檢查空值

Answer 2

這應該做：

var imputeDF = df 
df.dtypes.foreach { x => 
     val colName = x._1 
     newDF = newDF.na.fill(df.agg(max(colName)).first()(0).toString , Seq(colName)) } 

請注意，在scala中使用可變數據類型不是一個好習慣。

根據您的數據，您可以使用SQL連接或其他方法來用更合適的值替換空值。

Answer 3

如果你試圖取代與平均值的null值價值然後你計算mean和fill爲

import org.apache.spark.sql.functions.mean 


    val data = spark.read.option("header", true) 
      .option("inferSchema", true).format("com.databricks.spark.csv") 
      .load("data.csv") 

    //Calculate the mean for each column and create a map with its column name 
    //and use na.fill() method to replace null with that mean 

    data.na.fill(data.columns.zip(
      data.select(data.columns.map(mean(_)): _*).first.toSeq 
     ).toMap) 

我已經在本地測試了代碼並正常工作。

輸出：

+---------+-----+---+------+---+------------------+---------+------------------+-----+------+ 
|RowNumber|Poids|Age|Taille|0MI|    Hmean|CoocParam|   LdpParam|Test2|Classe| 
+---------+-----+---+------+---+------------------+---------+------------------+-----+------+ 
|  0| 81| 72| 160| 5|0.6451333333333333| 2.9421|2.2659000000000002| 3|  4| 
|  1| 54| 70| 164| 5|   0.6301| 2.7273|2.2659000000000002| 3|  4| 
|  2| 81| 51| 164| 5|0.6451333333333333| 2.9834|2.2659000000000002| 3|  4| 
|  3| 81| 74| 170| 5|   0.6966| 2.9654|   2.3699| 3|  4| 
|  4| 108| 62| 164| 5|   0.6087| 2.7093|   2.1619| 3|  4| 
+---------+-----+---+------+---+------------------+---------+------------------+-----+------+ 

希望這有助於！