用pySpark計算用戶事件之間的平均時間

Question

我有一個由「Events」，「Time」，「UserId」組成的日誌文件。

+------------+----------------+---------+ 
| Events |  Time  | UserId | 
+------------+----------------+---------+ 
| ClickA  | 7/6/16 10:00am | userA | 
+------------+----------------+---------+ 
| ClickB  | 7/6/16 12:00am | userA | 
+------------+----------------+---------+ 

我想爲每個用戶計算事件之間的平均時間。你們如何解決這個問題？ 在傳統的編程環境中，我會經歷用戶的每個事件，並計算事件n和n-1之間的時間增量，將此值添加到數組A.然後，我將計算每個值的平均值A. 如何使用Spark做到這一點？

Answer 1

忽略日期解析它看起來像一個窗口功能，然後進行簡單的聚集，從而大致一份工作，你需要這樣的事：

import org.apache.spark.sql.expressions.Window 
import org.apache.spark.sql.functions.{lag, avg} 

val df = Seq(
    ("ClickA", "2016-06-07 10:00:00", "UserA"), 
    ("ClickB", "2016-06-07 12:00:00", "UserA") 
).toDF("events", "time", "userid").withColumn("time", $"time".cast("timestamp")) 

val w = Window.partitionBy("userid").orderBy("time") 

// Difference between consecutive events in seconds 
val diff = $"time".cast("long") - lag($"time", 1).over(w).cast("long") 

df.withColumn("diff", diff).groupBy("userid").agg(avg($"diff"))