如何使用ARRAY_AGG（）聚合函數在豬或蜂房

Question

我有如下數據：

================================================================ 
session_id      screen_name screen_launch_time 
================================================================ 
990004916946605-1404157897784 screen1  1404157898275 
990004916946605-1404157897784 screen2  1404157898337 
990004947764274-1435162269418 screen1  1435162274044 
990004947764274-1435162269418 screen3  1435162274081 

我想用一個array_agg函數來獲得在下面的格式我的數據：

========================================================= 
session_id      screen_flow   count 
========================================================= 
990004916946605-1404157897784 screen1->screen2 1 
990004947764274-1435162269418 screen1->screen3 1 

有沒有人試過編寫UDAF或python腳本來實現array_agg函數中使用的邏輯？

請分享您的想法。

Answer 1

只由session_id組成，concat screen_name，並計算每組的記錄數。如果您不想製作brickhouse罐子，您可以使用collect_list()而不是collect()（但我不推薦它）。

查詢：

add jar /path/to/jars/brickhouse-0.7.1.jar; 
create temporary function collect as "brickhouse.udf.collect.CollectUDAF"; 

select session_id, screen_flow 
    , count(*) count 
from (
    select session_id 
    , concat_ws('->', collect(screen_name)) screen_flow 
    from db.table 
    group by session_id) x 
group by session_id, screen_flow 

輸出：

990004916946605-1404157897784 screen1->screen2 1 
990004947764274-1435162269418 screen1->screen3 1 

Answer 2

輸入： -

990004916946605-1404157897784,screen1,1404157898275 
990004916946605-1404157897784,screen2,1404157898337 
990004947764274-1435162269418,screen1,1435162274044 
990004947764274-1435162269418,screen3,1435162274081 

下面是豬風格答案..

records = LOAD '/user/user/inputfiles/session_id.txt' USING PigStorage(',') AS (session_id:chararray,screen_name:chararray,screnn_launch_time:chararray); 

rec_grped = GROUP records BY session_id; 

rec_each = FOREACH rec_grped 
        { 
         rec_inner_each = FOREACH records GENERATE screen_name; 

         GENERATE group as session_id, REPLACE(BagToString(rec_inner_each),'_','-->') as screen_flow, 1 as cnt; 
}; 

dump rec_each; 

輸出： -

990004916946605-1404157897784 screen1-->screen2 1 
990004947764274-1435162269418 screen1-->screen3 1