如何篩選空的輸出文件？

Question

這是我的豬腳本。

data = load 's3a://sessionlog/2016-05-28/' using SegmentationDataLoader() as (cookie:chararray,tags_and_pageref:map[]); 

tags_data = foreach data generate cookie, tags_and_pageref#'tags' as score_tag_bag; 

flattened_data = FOREACH tags_data GENERATE cookie, FLATTEN(score_tag_bag) as score_tag; 

converted_flattened_data = FOREACH flattened_data GENERATE cookie, (long)score_tag#'score' as score, score_tag#'tag' as tag; 

-- dump converted_flattened_data; 

-- tuple_data = FOREACH flattened_data GENERATE cookie, TOTUPLE(tags) as tag_tuple; 

-- splited_data = FOREACH flattened_data GENERATE cookie, TOKENIZE(tags) as score_tags:bag{t1:(),t2:()}; 

grouped_data = group converted_flattened_data by (cookie,tag); 

acc_data = foreach grouped_data generate group.cookie as cookie, group.tag as tag,SUM(converted_flattened_data.score) as score; 

pageref_data = foreach data generate cookie, tags_and_pageref#'pageref' as pageref_bag; 

flattened_pageref_data = FOREACH pageref_data GENERATE cookie, FLATTEN(pageref_bag) as score_tag; 

filtered_data = FILTER flattened_pageref_data BY score_tag is not null and not IsEmpty(score_tag); 

store acc_data into 'segmentation/2016-05-28/4' using PigStorage(','); 

store filtered_data into 'pagerefdata/2016-05-28/4' using PigStorage(','); 

但是pagerefdata的輸出全部是空文件。我如何過濾它，因爲它們都是空的，我不想要任何輸出。

在此先感謝。

Answer 1

不知道我在這裏完全理解了你的問題，但是從底部開始的3行似乎試圖通過測試null來過濾空白行。您是否嘗試過以下方法？：

FILTER flattened_pageref_data BY SIZE(TRIM(score_tag)) == 0;