火花查詢在地圖上

Question

我有以下架構

|-- Name: string (nullable = true) 
|-- Attendance: long (nullable = true) 
|-- Efficiency: map (nullable = true) 
| |-- key: string 
| |-- value: double (valueContainsNull = true) 

凡效率值的範圍從-1到+1，密鑰爲不同類別，例如體育，學術等鑲木文件的所有鍵我有多達20個不同的鑰匙。

我想獲取排名前100名的出勤率，其效率[關鍵]小於0. 我能夠爲一個關鍵點做到這一點。但我無法弄清楚我應該如何爲我的所有密鑰同時實現這一點。 代碼段一個鍵：

spark.sql("select Name,Attendance,Efficiency['Sports'] from data where Efficiency['Sports'] < 0 order by Attendance desc limit 100") 

在做了一些分析，我發現，我們需要爆炸我們的地圖。但是，每當我爆炸我的表中的行數增加，我無法獲取前100名。

一個鍵的樣本數據。實際的表有地圖而不是在這裏看到一列

+--------------------+------------------+-------------+       
|Name    |Attendance  |Efficiency[Sports]| 
+--------------------+------------------+-------------+ 
|A     |1000    |0.002  | 
|B     |365    |0.0   | 
|C     |1080    |0.193  | 
|D     |245    |-0.002  | 
|E     |1080    |-0.515  | 
|F     |905    |0.0   | 
|G     |900    |-0.001  | 

預期輸出：100名的名單上解決這個倒很樂於助人

每個鍵

+-----------------------+--------------+           
|Sports     |Academics  | 
+-----------------------+--------------+ 
|A      |A    | 
|B      |C    | 
|C      |D    | 
|D      |E    | 

任何幫助

感謝

Answer 1

我希望這是你在找什麼

import org.apache.spark.sql.functions._ 

//dummy data 
val d = Seq(
    ("a", 10, Map("Sports" -> -0.2, "Academics" -> 0.1)), 
    ("b", 20, Map("Sports" -> -0.1, "Academics" -> -0.1)), 
    ("c", 5, Map("Sports" -> -0.2, "Academics" -> 0.5)), 
    ("d", 15, Map("Sports" -> -0.2, "Academics" -> 0.0)) 
).toDF("Name", "Attendence", "Efficiency") 

//explode the map and get key value 
val result = d.select($"Name", $"Attendence", explode($"Efficiency")) 

//select value less than 0 and show 100 
result.select("*").where($"value".lt(0)) 
    .sort($"Attendence".desc) 
    .show(100) 

輸出：

+----+----------+---------+-----+ 
|Name|Attendence|key  |value| 
+----+----------+---------+-----+ 
|b |20  |Sports |-0.1 | 
|b |20  |Academics|-0.1 | 
|d |15  |Sports |-0.2 | 
|a |10  |Sports |-0.2 | 
|c |5   |Sports |-0.2 | 
+----+----------+---------+-----+ 

希望這有助於！

Answer 2

鑑於輸入數據幀作爲

+----+----------+-----------------------------------------+ 
|Name|Attendance|Efficiency        | 
+----+----------+-----------------------------------------+ 
|A |1000  |Map(Sports -> 0.002, Academics -> 0.002) | 
|B |365  |Map(Sports -> 0.0, Academics -> 0.0)  | 
|C |1080  |Map(Sports -> 0.193, Academics -> 0.193) | 
|D |245  |Map(Sports -> -0.002, Academics -> -0.46)| 
|E |1080  |Map(Sports -> -0.515, Academics -> -0.5) | 
|F |905  |Map(Sports -> 0.0, Academics -> 0.0)  | 
|G |900  |Map(Sports -> -0.001, Academics -> -0.0) | 
+----+----------+-----------------------------------------+ 

使用udf函數來迭代Map用於檢查小於零倍的值。這是可以做到如下

import org.apache.spark.sql.functions._ 
val isLessThan0 = udf((maps: Map[String, Double]) => maps.map(x => x._2 < 0).toSeq.contains(true)) 

df.withColumn("lessThan0", isLessThan0('Efficiency)) 
    .filter($"lessThan0" === true) 
    .orderBy($"Attendance".desc) 
    .drop("lessThan0") 
    .show(100, false) 

你將有輸出

+----+----------+-----------------------------------------+ 
|Name|Attendance|Efficiency        | 
+----+----------+-----------------------------------------+ 
|E |1080  |Map(Sports -> -0.515, Academics -> -0.5) | 
|G |900  |Map(Sports -> -0.001, Academics -> -0.0) | 
|D |245  |Map(Sports -> -0.002, Academics -> -0.46)| 
+----+----------+-----------------------------------------+