awk與讀線的比較線條

Question

我有兩個文件，一個帶17k線，另一個帶4k線。我想將位置115與第二個文件中每行的位置125進行比較，如果匹配，則將第一個文件中的整行寫入新文件。我想出了一個解決方案，我使用'cat $ filename |'來讀取文件同時閱讀LINE'。但需要大約8分鐘才能完成。還有沒有其他方式像使用'awk'來減少這個過程時間。

我的代碼

cat $filename | while read LINE 
do 
    #read 115 to 125 and then remove trailing spaces and leading zeroes 
    vid=`echo "$LINE" | cut -c 115-125 | sed 's,^ *,,; s, *$,,' | sed 's/^[0]*//'` 
    exist=0 
    #match vid with entire line in id.txt 
    exist=`grep -x "$vid" $file_dir/id.txt | wc -l` 
    if [[ $exist -gt 0 ]]; then 
    echo "$LINE" >> $dest_dir/id.txt 
    fi 
done 

Answer 1

這是怎麼回事：

FNR==NR {      # FNR == NR is only true in the first file 

    s = substr($0,115,10)  # Store the section of the line interested in 
    sub(/^\s*/,"",s)   # Remove any leading whitespace 
    sub(/\s*$/,"",s)   # Remove any trailing whitespace 

    lines[s]=$0    # Create array of lines 
    next      # Get next line in first file 
} 
{        # Now in second file 
    for(i in lines)   # For each line in the array 
     if (i~$0) {   # If matches the current line in second file 
      print lines[i]  # Print the matching line from file1 
      next    # Get next line in second file 
     } 
} 

保存到一個腳本script.awk和像運行：

$ awk -f script.awk "$filename" "${file_dir}/id.txt" > "${dest_dir}/id.txt" 

這仍將b因爲對於第二個文件中的每一行，您需要查看第一個中約50％的獨特行（假設大多數行確實匹配）。如果您可以確認第二個文件中的行與子字符串完全匹配，則可以顯着提高此功能。

---

對於全行一致，這應該是更快：

FNR==NR {      # FNR == NR is only true in the first file 

    s = substr($0,115,10)  # Store the section of the line interested in 
    sub(/^\s*/,"",s)   # Remove any leading whitespace 
    sub(/\s*$/,"",s)   # Remove any trailing whitespace 

    lines[s]=$0    # Create array of lines 
    next      # Get next line in first file 
} 
($0 in lines) {     # Now in second file 
    print lines[$0]  # Print the matching line from file1 
}