Unix通過awk使用正則表達式加入兩個文件

Question

我有一個文件（lookup.txt），其中包含一個由正則表達式列表組成的查找表，包含相應的數據（類別和句點）。例如

INTERNODE|household/bills/broadband|monthly 
ORIGIN ENERGY|household/bills/electricity|quarterly 
TELSTRA.*BILL|household/bills/phone|quarterly 
OPTUS|household/bills/mobile|quarterly 
SKYPE|household/bills/skype|non-periodic 

我有一個包含的費用列表，另一個文件（data.txt中），例如：

2009-10-31,cc,-39.9,INTERNODE BROADBAND 
2009-10-31,cc,-50,ORIGIN ENERGY 543546 
2009-10-31,cc,-68,INTERNODE BROADBAND EXCESS CHARGES 
2009-10-31,cc,-90,TELSTRA MOBILE BILL 
2009-11-02,cc,-320,TELSTRA HOME BILL 
2009-11-03,cc,-22.96,DICK SMITH 
2009-11-03,cc,-251.24,BUNNINGS 
2009-11-04,cc,-4.2,7-ELEVEN 

我想加入這兩者結合起來，由此的data.txt文件比賽的第4列lookup.txt文件第一列的正則表達式。

所以輸出將是：

2009-10-31,cc,-39.9,INTERNODE BROADBAND,household/bills/broadband,monthly 
2009-10-31,cc,-50,ORIGIN ENERGY 543546,household/bills/electricity,quarterly 
2009-10-31,cc,-68,INTERNODE BROADBAND EXCESS CHARGES,household/bills/broadband,monthly 
2009-10-31,cc,-90,TELSTRA MOBILE BILL,household/bills/phone,quarterly 
2009-11-02,cc,-320,TELSTRA HOME BILL,household/bills/phone,quarterly 
2009-11-03,cc,-22.96,DICK SMITH 
2009-11-03,cc,-251.24,BUNNINGS 
2009-11-04,cc,-4.2,7-ELEVEN 

我該使用bash的循環，遍歷查找，做裏grep和使用SED增加額外的列來達到的，但它是非常緩慢的。所以想知道是否有更快的方法來做這件事，說使用awk。

任何幫助，將不勝感激。

Answer 1

$ awk -F'|' 'FNR==NR{a[$1]=$2","$3;next}{m=split($0,b,",");for(i in a){if(b[4]~i){print $0","a[i];next}}}1' lookup file 
2009-10-31,cc,-39.9,INTERNODE BROADBAND,household/bills/broadband,monthly 
2009-10-31,cc,-50,ORIGIN ENERGY 543546,household/bills/electricity,quarterly 
2009-10-31,cc,-68,INTERNODE BROADBAND EXCESS CHARGES,household/bills/broadband,monthly 
2009-10-31,cc,-90,TELSTRA MOBILE BILL,household/bills/phone,quarterly 
2009-11-02,cc,-320,TELSTRA HOME BILL,household/bills/phone,quarterly 
2009-11-03,cc,-22.96,DICK SMITH 
2009-11-03,cc,-251.24,BUNNINGS 
2009-11-04,cc,-4.2,7-ELEVEN 

Answer 2

你可以做到這一點在Python：

#!/usr/bin/python 
import csv, re 
lookup = [] 
with open('lookup.txt') as f: 
    for rec in csv.reader(f, delimiter='|'): 
     lookup.append((re.compile(rec[0]), rec[1:])) 
with open('data.txt') as f: 
    for rec in csv.reader(f, delimiter=','): 
     for rexp, fields in lookup: 
      if rexp.match(rec[3]): 
       rec.extend(fields) 
       break 
     print ','.join(rec) 

爲您的文件lookup.txt和data.txt返回，在不到0.3秒以下：

2009-10-31,cc,-39.9,INTERNODE BROADBAND,household/bills/broadband,monthly 
2009-10-31,cc,-50,ORIGIN ENERGY 543546,household/bills/electricity,quarterly 
2009-10-31,cc,-68,INTERNODE BROADBAND EXCESS CHARGES,household/bills/broadband,monthly 
2009-10-31,cc,-90,TELSTRA MOBILE BILL,household/bills/phone,quarterly 
2009-11-02,cc,-320,TELSTRA HOME BILL,household/bills/phone,quarterly 
2009-11-03,cc,-22.96,DICK SMITH 
2009-11-03,cc,-251.24,BUNNINGS 
2009-11-04,cc,-4.2,7-ELEVEN 

Answer 3

你可以在Perl做到這一點。 Perl（或Python）的優點是他們有用於處理CSV文件的庫。你的例子很簡單，但如果雙引號內有逗號，會發生什麼？或者utf8呢？等等。

這個標準的Perl庫是Text:CSV_XS。然而，它有點冗長，我更喜歡Parse::CSV這是一個圍繞Text :: CSV_XS的包裝。

#!/usr/bin/perl 

use strict; 
use warnings; 
use Parse::CSV; 

my %lookup; 
my $l = Parse::CSV->new(file => "lookup.txt", sep_char => '|'); 
while (my $row = $l->fetch) { 
    my $key = qr/$row->[0]/; 
    $lookup{$key} = [$row->[1,]]; 
} 

my $d = Parse::CSV->new(file => "data.txt"); 
while (my $row = $d->fetch) { 
    foreach my $regex (keys %lookup) { 
     if ($row->[3] =~ $regex) { 
     push @$row, @{$lookup{$regex}}; 
     last; 
     } 
    } 
    print join(",", @$row), "\n"; 
} 

Answer 4

如果你沒有正則表達式，你可以使用join。 lookup.txt有多少個正則表達式？如果只是那個，只需展開並放棄該功能即可。

Answer 5

awk的設計目的是一次處理一條記錄中的單一數據流，因此它不適用於此項工作。這將是Perl或其他語言的十分鐘練習，它更多地面向通用編程。

如果您一心想用awk完成所有工作，請編寫一個腳本以從處理數據的查找文件生成第二個awk腳本，然後運行第二個腳本。