在Spark中獲取上個星期一

Question

我正在使用Spark 2.0和Python API。

我有一個類型爲DateType（）的列的數據框。我想向包含最近星期一的數據框添加一列。

我能做到這一點是這樣的：

reg_schema = pyspark.sql.types.StructType([ 
    pyspark.sql.types.StructField('AccountCreationDate', pyspark.sql.types.DateType(), True), 
    pyspark.sql.types.StructField('UserId', pyspark.sql.types.LongType(), True) 
]) 
reg = spark.read.schema(reg_schema).option('header', True).csv(path_to_file) 
reg = reg.withColumn('monday', 
    pyspark.sql.functions.when(pyspark.sql.functions.date_format(reg.AccountCreationDate,'E') == 'Mon', 
     reg.AccountCreationDate).otherwise(
    pyspark.sql.functions.when(pyspark.sql.functions.date_format(reg.AccountCreationDate,'E') == 'Tue', 
     pyspark.sql.functions.date_sub(reg.AccountCreationDate, 1)).otherwise(
    pyspark.sql.functions.when(pyspark.sql.functions.date_format(reg.AccountCreationDate, 'E') == 'Wed', 
     pyspark.sql.functions.date_sub(reg.AccountCreationDate, 2)).otherwise(
    pyspark.sql.functions.when(pyspark.sql.functions.date_format(reg.AccountCreationDate, 'E') == 'Thu', 
     pyspark.sql.functions.date_sub(reg.AccountCreationDate, 3)).otherwise(
    pyspark.sql.functions.when(pyspark.sql.functions.date_format(reg.AccountCreationDate, 'E') == 'Fri', 
     pyspark.sql.functions.date_sub(reg.AccountCreationDate, 4)).otherwise(
    pyspark.sql.functions.when(pyspark.sql.functions.date_format(reg.AccountCreationDate, 'E') == 'Sat', 
     pyspark.sql.functions.date_sub(reg.AccountCreationDate, 5)).otherwise(
    pyspark.sql.functions.when(pyspark.sql.functions.date_format(reg.AccountCreationDate, 'E') == 'Sun', 
     pyspark.sql.functions.date_sub(reg.AccountCreationDate, 6)) 
     ))))))) 

然而，這似乎是一個大量的代碼的東西，應該是相當簡單的。有沒有更簡潔的方式來做到這一點？

Answer 1

您可以使用next_day確定下一個日期並減去一週。所需的功能可以如下輸入：

from pyspark.sql.functions import next_day, date_sub 

又如：

def previous_day(date, dayOfWeek): 
    return date_sub(next_day(date, "monday"), 7) 

最後一個例子：

from pyspark.sql.functions import to_date 

df = sc.parallelize([ 
    ("2016-10-26",) 
]).toDF(["date"]).withColumn("date", to_date("date")) 

df.withColumn("last_monday", previous_day("date", "monday")) 

隨着結果：

+----------+-----------+ 
|  date|last_monday| 
+----------+-----------+ 
|2016-10-26| 2016-10-24| 
+----------+-----------+