聯合使用豬列使用兩個文件

Question

我想使用豬聯合/合併兩個文件。但是，這是一個不同於一般工會的統一體。以下是我的文件（H *的文件頭）：

F1 : 
h1,h2,h3,h4 
a01,a02,a03,a04 
a11,a12,a13,a14 

F2 : 
h3,h4,h5,h6 
a23,a24,b01,b02 
a33,a34,b11,b12 

輸出結果必須是這些文件的聯盟這樣的：

FR : 
h1,h2,h3,h4,h5,h6 
a01,a02,a03,a04,, 
a11,a12,a13,a14,, 
,,a23,a24,b01,b02 
,,a33,a34,b11,b12 

還有一個困難是我想使它通用所以它適用於動態數量的通用列。目前有兩個共同的列，它可以有3個或1個共同的列，或者甚至沒有共同的列。例如：

F1 : 
h1,h2,h3,h4 
a1,a2,a3,a4 

F2 
h5,h6,h7,h8 
b1,b2,b3,b4 

FR 
a1,a2,a3,a4 
,,,,b1,b2,b3,b4 

任何提示/幫助是可觀的。

Answer 1

這裏是你如何能做到這靜態：

F1full = FOREACH F1 GENERATE h1,h2,h3,h4, NULL as h5, NULL as h6; 
F2full = FOREACH F2 GENERATE NULL as h1,NULL as h2,h3,h4, h5, h6; 

FR = F1full UNION F2full; 

豬也不是很靈活的，所以我不認爲這是可以動態生成這種/爲通用情況。

如果你想爲普通情況下的解決方案，你可以使用一種語言如Python建立一個基於存儲的表格/文件的元數據所需要的命令。

Answer 2

我試圖解決使用以下方法的問題：

1) Load both of the files. 
2) Add counter to generate a unique field (ID). 
3) Start the counter for file B where counter for A ended. 
4) Cogroup both files with common columns, including counteer. 
5) Take all group columns in a different schema. 
6) Generate uncommon columns from both files, along with the counter. 
7) First join uncommon columns from file A with group columns on counter. 
8) Join the result of step 7 with uncommon columns from file B on counter. 

以下是豬腳本做同樣的。由於這個腳本是通用的，我已經提到了在運行腳本之前需要的所有參數。

-- Parameters required : $file1_path, $file2_path, $file1_schema, $file2_schema, $COUNT_A (number of rows in file A), $CMN_COLUMN_A (common columns in A), $CMN_COLUMN_B, $UNCMN_COLUMN_A(Unique columns in file A), $UNCMN_COLUMN_B. 
A = LOAD '$file1_path' USING org.apache.pig.piggybank.storage.CSVExcelStorage('~', 'NO_MULTILINE', 'NOCHANGE', 'SKIP_INPUT_HEADER') as ($file1_schema); 
B = LOAD '$file2_path' USING org.apache.pig.piggybank.storage.CSVExcelStorage('~', 'NO_MULTILINE', 'NOCHANGE', 'SKIP_INPUT_HEADER') as ($file2_schema); 

RANK_A = RANK A; 
RANK_B = RANK B; 
COUNT_RANK_B = FOREACH RANK_B GENERATE ($0+(long)'$COUNT_A') as rank_B, $1 ..; 

COGRP_RANK_AB = COGROUP RANK_A BY($CMN_COLUMN_A), COUNT_RANK_B BY ($CMN_COLUMN_B); 

CMN_COGRP_RANK_AB = FOREACH COGRP_RANK_AB GENERATE FLATTEN(group) AS ($CMN_COLUMN_A); 
UNCMN_RB = FOREACH COUNT_RANK_B GENERATE $UNCMN_COLUMN_B; 

JOIN_CMN_UNCMN_A = JOIN CMN_COGRP_RANK_AB BY(rank_A) LEFT OUTER, UNCMN_RA by rank_A; 
JOIN_CMN_UNCMN_B = JOIN JOIN_CMN_UNCMN_A BY(CMN_COGRP_RANK_AB::rank_A) LEFT OUTER, UNCMN_RB by rank_B; 

STORE FINAL_DATA INTO '$store_path' USING org.apache.pig.piggybank.storage.CSVExcelStorage('~', 'NO_MULTILINE', 'UNIX', 'WRITE_OUTPUT_HEADER');