grep從一個巨大的日誌文件中的大量模式

Question

我有一個shell腳本，每小時通過cron作業調用，並搜索星號日誌，併爲我提供了以原因31結束的調用的唯一ID。

while read ref 
do 
cat sample.log | grep "$ref" | grep 'got hangup request, cause 31' | grep -o 'C-[0-9a-z][0-9a-z][0-9a-z][0-9a-z][0-9a-z][0-9a-z][0-9a-z][0-9a-z]' >> cause_temp.log 
done < callref.log 

問題是，while循環太慢，爲了準確性，我已經包含了4個while循環來執行各種檢查。

callref.log文件由呼叫標識符值組成，每小時它將有大約50-90,000個值，腳本需要大約45-50分鐘才能完成執行並向我發送報告。

如果我能夠縮短循環的執行時間，那將會非常有幫助。由於sample.log文件的大小約爲20 GB，並且對於每個循環打開文件並執行搜索，因此我認爲while循環是此處的瓶頸。

也做了研究，發現一些有用的鏈接像 Link 1Link 2

但解決方案建議我不能執行或者不知道怎麼樣。任何建議都會有所幫助。謝謝

由於sample.log包含敏感信息，我將無法共享任何日誌，但以下是我從互聯網上獲取的一些示例日誌。

Dec 16 18:02:04 asterisk1 asterisk[31774]: NOTICE[31787]: chan_sip.c:11242 in handle_request_register: Registration from '"503"<sip:503@192.168.1.107>' failed for '192.168.1.137' - Wrong password 
Dec 16 18:03:13 asterisk1 asterisk[31774]: NOTICE[31787]: chan_sip.c:11242 in handle_request_register: Registration from '"502"<sip:502@192.168.1.107>' failed for '192.168.1.137' - Wrong password 
Dec 16 18:04:49 asterisk1 asterisk[31774]: NOTICE[31787]: chan_sip.c:11242 in handle_request_register: Registration from '"1737245082"<sip:1737245082@192.168.1.107>' failed for '192.168.1.137' - Username/auth name mismatch 
Dec 16 18:04:49 asterisk1 asterisk[31774]: NOTICE[31787]: chan_sip.c:11242 in handle_request_register: Registration from '"100"<sip:100@192.168.1.107>' failed for '192.168.1.137' - Username/auth name mismatch 
Jun 27 18:09:47 host asterisk[31774]: ERROR[27910]: chan_zap.c:10314 setup_zap: Unable to register channel '1-2' 
Jun 27 18:09:47 host asterisk[31774]: WARNING[27910]: loader.c:414 __load_resource: chan_zap.so: load_module failed, returning -1 
Jun 27 18:09:47 host asterisk[31774]: WARNING[27910]: loader.c:554 load_modules: Loading module chan_zap.so failed! 

文件callref.log由行的列表，它看起來像的 -

C-001ec22d 
C-001ec23d 
C-001ec24d 
C-001ec31d 
C-001ec80d 

此外，上面的期望輸出而循環如下所示C-001ec80d

另外我的主關心的是讓while循環運行得更快。就像加載數組中的callref.log的所有值一樣，如果可能的話，同時在sample.log的一次傳遞中搜索所有值。

Answer 1

我花了一天時間來構建一個測試框架並測試不同命令的變體，我認爲你已經有了最快的一個。

這導致我認爲如果你想獲得更好的性能，你應該看看日誌消化框架，比如ossec（你的日誌樣本來自哪裏）或許是splunk。這些可能對您的願望太笨拙。或者，您應該考慮設計和構建更適合解析的java/C/perl/awk中的某些內容。

更頻繁地運行您現有的腳本也將有所幫助。

祝你好運！如果你喜歡，我可以把我做的工作放在一邊，然後張貼在這裏，但我認爲它太過矯枉過正。

按要求; CalFuncs.sh：在大多數我的腳本

#!/bin/bash 

LOGDIR="/tmp" 
LOG=$LOGDIR/CalFunc.log 
[ ! -d "$LOGDIR" ] && mkdir -p $(dirname $LOG) 

SSH_OPTIONS="-o StrictHostKeyChecking=no -q -o ConnectTimeout=15" 
SSH="ssh $SSH_OPTIONS -T" 
SCP="scp $SSH_OPTIONS" 
SI=$(basename $0) 

Log() { 
    echo "`date` [$SI] $@" >> $LOG 
} 

Run() { 
    Log "Running '$@' in '`pwd`'" 
    $@ 2>&1 | tee -a $LOG 
} 

RunHide() { 
    Log "Running '$@' in '`pwd`'" 
    $@ >> $LOG 2>&1 
} 

PrintAndLog() { 
    Log "$@" 
    echo "$@" 
} 

ErrorAndLog() { 
    Log "[ERROR] $@ " 
    echo "$@" >&2 
} 

showMilliseconds(){ 
    date +%s 
} 

runMethodForDuration(){ 
    local startT=$(showMilliseconds) 
    $1 
    local endT=$(showMilliseconds) 
    local totalT=$((endT-startT)) 
    PrintAndLog "that took $totalT seconds to run $1" 
    echo $totalT 
} 

genCallRefLog.sh庫我源 - 根據參數

#!/bin/bash 
#Script to make 80000 sequential lines of callref.log this should suffice for a POC 
if [ -z "$1" ] ; then 
    echo "genCallRefLog.sh requires an integer of the number of lines to pump out of callref.log" 
    exit 1 
fi 
file="callref.log" 
[ -f "$file" ] && rm -f "$file" # del file if exists 
i=0 #put start num in here 
j="$1" #put end num in here 
echo "building $j lines of callref.log" 
for (( a=i ; a < j; a++ )) 
do 
    printf 'C-%08x\n' "$a" >> $file 
done 

genSampleLog.sh生成虛擬樣本生成虛構callref.log大小。日誌大小取決於參數

#!/bin/bash 
#Script to make 80000 sequential lines of callref.log this should suffice for a POC 
if [ -z "$1" ] ; then 
    echo "genSampleLog.sh requires an integer of the number of lines to pump out of sample.log" 
    exit 1 
fi 
file="sample.log" 
[ -f "$file" ] && rm -f "$file" # del file if exists 
i=0 #put start num in here 
j="$1" #put end num in here 
echo "building $j lines of sample.log" 
for (( a=i ; a < j; a++ )) 
do 
    printf 'Dec 16 18:02:04 asterisk1 asterisk[31774]: NOTICE[31787]: C-%08x got hangup request, cause 31\n' "$a" >> $file 
done 

最後是我使用的實際測試腳本。通常我會將構建腳本註釋掉，因爲它們只需在更改日誌大小時運行即可。我通常一次只能運行一個測試功能並記錄結果。

test.sh

#!/bin/bash 
source "./CalFuncs.sh" 

targetLogFile="cause_temp.log" 
Log "Starting" 

checkTargetFileSize(){ 
    expectedS="$1" 
    hasS=$(cat $targetLogFile | wc -l) 
    if [ "$expectedS" != "$hasS" ] ; then 
    ErrorAndLog "Got $hasS but expected $expectedS, when inspecting $targetLogFile" 
    exit 244 
    fi 
} 

standard(){ 
    iter=0 
    while read ref 
    do 
    cat sample.log | grep "$ref" | grep 'got hangup request, cause 31' | grep -o 'C-[0-9a-z][0-9a-z][0-9a-z][0-9a-z][0-9a-z][0-9a-z][0-9a-z][0-9a-z]' >> $targetLogFile 
    done < callref.log 
} 

subStandardVarient(){ 
    iter=0 
    while read ref 
    do 
    cat sample.log | grep 'got hangup request, cause 31' | grep -o "$ref" >> $targetLogFile 
    done < callref.log 
} 

newFunction(){ 
    grep -f callref.log sample.log | grep 'got hangup request, cause 31' >> $targetLogFile 
} 

newFunction4(){ 
    grep 'got hangup request, cause 31' sample.log | grep -of 'callref.log'>> $targetLogFile 
} 

newFunction5(){ 
    #splitting grep 
    grep 'got hangup request, cause 31' sample.log > /tmp/somefile 
    grep -of 'callref.log' /tmp/somefile >> $targetLogFile 
} 

newFunction2(){ 
    iter=0 

    while read ref 
    do 
    ((iter++)) 
    echo "$ref" | grep 'got hangup request, cause 31' | grep -of 'callref.log' >> $targetLogFile 
    done < sample.log 
} 

newFunction3(){ 
    iter=0 
    pat="" 
    while read ref 
    do 
    if [[ "$pat." != "." ]] ; then 
     pat="$pat|" 
    fi 
    pat="$pat$ref" 
    done < callref.log 
    # Log "Have pattern $pat" 
    while read ref 
    do 
    ((iter++)) 
    echo "$ref" | grep 'got hangup request, cause 31' | grep -oP "$pat" >> $targetLogFile 
    done < sample.log 
    #grep: regular expression is too large 
} 

[ -f "$targetLogFile" ] && rm -f "$targetLogFile" 

numLines="100000" 
Log "testing algorithms with $numLines in each log file." 

setupCallRef(){ 
    ./genCallRefLog.sh $numLines 
} 

setupSampleLog(){ 
    ./genSampleLog.sh $numLines 
} 

setupCallRef 
setupSampleLog 

runMethodForDuration standard > /dev/null 
checkTargetFileSize "$numLines" 
[ -f "$targetLogFile" ] && rm -f "$targetLogFile" 
runMethodForDuration subStandardVarient > /dev/null 
checkTargetFileSize "$numLines" 
[ -f "$targetLogFile" ] && rm -f "$targetLogFile" 
runMethodForDuration newFunction > /dev/null 
checkTargetFileSize "$numLines" 
# [ -f "$targetLogFile" ] && rm -f "$targetLogFile" 
# runMethodForDuration newFunction2 > /dev/null 
# checkTargetFileSize "$numLines" 
# [ -f "$targetLogFile" ] && rm -f "$targetLogFile" 
# runMethodForDuration newFunction3 > /dev/null 
# checkTargetFileSize "$numLines" 
# [ -f "$targetLogFile" ] && rm -f "$targetLogFile" 
# runMethodForDuration newFunction4 > /dev/null 
# checkTargetFileSize "$numLines" 
[ -f "$targetLogFile" ] && rm -f "$targetLogFile" 
runMethodForDuration newFunction5 > /dev/null 
checkTargetFileSize "$numLines" 

由上述可知，現有的方法總是比任何我想出了更快。我認爲有人關心優化它。

Answer 2

既然你不能產生足夠的樣本日誌請求時，再次進行測試，甚至，我颳起了一些測試材料自己：

$ cat callref.log 
a 
b 
$ cat sample.log 
a 1 
b 2 
c 1 

用awk：

$ awk 'NR==FNR {    # hash callrefs 
    a[$1] 
    next 
} 
{       # check callrefs from sample records and output when match 
    for(l in a) 
     if($0 ~ l && $0 ~ 1) # 1 is the static string you look for along a callref 
      print l 
}' callref.log sample.log 
a 1 

HTH