如何在Spark SQL中的爆炸字段上做GROUP BY？

Question

• 齊柏林0.6
• 星火1.6
• SQL

我試圖找到一些微博的前20名出現的詞。 filtered包含每個推文的單詞數組。以下：

select explode(filtered) AS words from tweettable 

列出了每個單詞你所期望的，但我要的是讓每個單詞的數量在所有的鳴叫，然後顯示這些排名前20位。下面的作品，但我需要爲此在SQL：

df.select(explode($"filtered").as("value")) 
    .groupBy("value") 
    .count() 
    .sort(desc("count")) 
    .show(20, false) 

我試圖GROUP BY上words，filtered，並且explode(filtered)但所有給了錯誤。

Answer 1

可以在FROM語句中使用subqueries：

SELECT value, count(*) AS count 
FROM (SELECT explode(filtered) AS value 
     FROM tweettable) AS temp 
GROUP BY value 
ORDER BY count DESC 

Answer 2

下面的代碼會給你全面的瞭解，以達到你所期待的東西。測試火花（1.6）

val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) 
import hiveContext.implicits._ 

val lst = List(Seq("Hello","Hyd","Hello","Mumbai"),Seq("Hello","Mumbai"),Seq("Hello","Delhi","Hello","Banglore")) 
case class Tweets(filtered: Seq[String]) 
val df = sc.parallelize(lst).map(x=>Tweets(x)).toDF 

import org.apache.spark.sql.functions.{explode} 
import org.apache.spark.sql.functions.count 
df.select(explode($"filtered").as("value")).groupBy("value").agg(count("*").alias("cnt")).orderBy('cnt.desc).show(20,false) 

或者，您可以使用窗口功能。

val hiveContext = new org.apache.spark.sql.hive.HiveContext(sc) 
import hiveContext.implicits._ 

val lst = List(Seq("Hello","Hyd","Hello","Mumbai"),Seq("Hello","Mumbai"),Seq("Hello","Delhi","Hello","Banglore")) 
case class Tweets(filtered: Seq[String]) 
val df = sc.parallelize(lst).map(x=>Tweets(x)).toDF 

import org.apache.spark.sql.expressions.Window 
import org.apache.spark.sql.functions._ 
val w = org.apache.spark.sql.expressions.Window.orderBy('cnt.desc) 

df.select(explode($"filtered").as("value")).groupBy("value").agg(count("*").alias("cnt")).withColumn("filteredrank", rank.over(w)).filter(col("filteredrank") <= 20).show()