使用兩列火花蟒蛇

Question

鍵我有我的csv文件4列和多行。

Date(MM/DD/YY) Arr_Dep  Dom_Int    Num_Fl 
01/01/15 0:00 Arrival  Domestic   357 
03/01/15 0:00 Arrival  International  269 
06/01/15 0:00 Departure Domestic   82 
08/01/15 0:00 Departure International  5 
05/01/16 0:00 Arrival  Domestic   44 
06/01/16 0:00 Arrival  Domestic   57 
07/01/16 0:00 Departure International  51 
08/01/16 0:00 Departure International  40 
08/01/17 0:00 Arrival  Domestic   1996 
10/01/17 0:00 Departure International  21 

我必須根據航班是抵達還是出發，找到特定年份每月的平均航班數。所以輸出我期待爲上述輸入：

2015, arrival, 313 
2015, departure, 44 
2016, arrival, 51 
2016, departure, 46 
2017, arrival, 1996 
2017, departure, 21 

我現在面臨的問題，我怎麼應該包括在我的地圖功能在我的鑰匙，即Arr_Dep和日期列兩列，最終減少它得到平均。 我寫了下面的腳本爲止。不確定如何繼續

from pyspark import SparkContext 
from operator import add 
import sys 

sc = SparkContext(appName="example") 
input_file = sys.argv[1] 
lines = sc.textFile(input_file) 
first = lines.map(lambda x : ((x.split(",")[0].split(" ")[0][5:]).encode('ascii','ignore'), int(x.split(",")[-1]), x.split(",")[1])) 
second = first.filter(lambda x : "Arrival" in x[1] or "Departure" in x[1]) 
third = second.map(lambda x : (x[0],x[1])) 
result = third.reduceByKey("Not sure how to calculate average") 
output = result.collect() 
for v in sorted(output, key = lambda x:x[0]): 
    print '%s, %s' % (v[0], v[1]) 

我不確定上述腳本。我是新來的火花和蟒蛇。任何想法如何繼續？

Answer 1

最好是使用SQL API：

from pyspark.sql.functions import * 

df = spark.read.options(inferSchema=True, header=True).csv(input_file) 
df\ 
    .groupBy(year(to_date("Date(MM/DD/YY)", "MM/dd/yyH:mm")).alias("year"), "Arr_Dep")\ 
    .avg("Num_Fl")