pyspark中的聚合稀疏矢量

Question

我有一個Hive表，其中包含文本數據和與每個文檔關聯的一些元數據。看起來像這樣。

from pyspark.ml.feature import Tokenizer 
from pyspark.ml.feature import CountVectorizer 

df = sc.parallelize([ 
    ("1", "doc_1", "fruit is good for you"), 
    ("2", "doc_2", "you should eat fruit and veggies"), 
    ("2", "doc_3", "kids eat fruit but not veggies") 
]).toDF(["month","doc_id", "text"]) 


+-----+------+--------------------+ 
|month|doc_id|    text| 
+-----+------+--------------------+ 
| 1| doc_1|fruit is good for...| 
| 2| doc_2|you should eat fr...| 
| 2| doc_3|kids eat fruit bu...| 
+-----+------+--------------------+ 

我想按月計算單詞。按月

tokenizer = Tokenizer().setInputCol("text").setOutputCol("words") 
tokenized = tokenizer.transform(df) 

cvModel = CountVectorizer().setInputCol("words").setOutputCol("features").fit(tokenized) 
counted = cvModel.transform(tokenized) 

+-----+------+--------------------+--------------------+--------------------+ 
|month|doc_id|    text|    words|   features| 
+-----+------+--------------------+--------------------+--------------------+ 
| 1| doc_1|fruit is good for...|[fruit, is, good,...|(12,[0,3,4,7,8],[...| 
| 2| doc_2|you should eat fr...|[you, should, eat...|(12,[0,1,2,3,9,11...| 
| 2| doc_3|kids eat fruit bu...|[kids, eat, fruit...|(12,[0,1,2,5,6,10...| 
+-----+------+--------------------+--------------------+--------------------+ 

現在我想組和返回的東西，看起來像： 到目前爲止，我已經採取了CountVectorizer方法

month word count 
1  fruit 1 
1  is  1 
... 
2  fruit 2 
2  kids 1 
2  eat 2 
... 

我怎麼能這樣做呢？

Answer 1

Vector聚合沒有內置機制，但您不需要在這裏。一旦你已經符號化數據你可以explode和彙總：

from pyspark.sql.functions import explode 

(counted 
    .select("month", explode("words").alias("word")) 
    .groupBy("month", "word") 
    .count()) 

如果您更喜歡限制結果爲vocabulary只需添加一個過濾器：

from pyspark.sql.functions import col 

(counted 
    .select("month", explode("words").alias("word")) 
    .where(col("word").isin(cvModel.vocabulary)) 
    .groupBy("month", "word") 
    .count())