自定義神交模式的日誌

Question

因此，這裏是我的日誌樣本：

23:28:32.226 WARN [MsgParser:ListProc-Q0:I5] Parsing error 
Error mapping the fieldAdditional Information: 

    at com.authentic.mapper.parsing.LengthVar.readBytes(LengthVar.java:178) 
    at com.authentic.mapper.parsing.GrpLengthVar.read(GrpLengthVar.java:96) 
    at com.authentic.mapper.parsing.GrpLengthVar.read(GrpLengthVar.java:119) 
    at com.authentic.mapper.parsing.MsgParser.processReadEnumeration(MsgParser.java:339) 
    at com.authentic.mapper.parsing.MsgParser.parseIncomingMessageBody(MsgParser.java:295) 
    at com.authentic.mapper.MapperMgr.parseMsg(MapperMgr.java:1033) 
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.parseMessage(AbstractConnectionHandler.java:4408) 
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.plainMessageReceivedEvent(AbstractConnectionHandler.java:2031) 
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.messageReceivedEvent(AbstractConnectionHandler.java:1911) 
    at com.authentic.architecture.interchange.accesspoint.SocketConnectionHandler.messageReceivedEvent(SocketConnectionHandler.java:801) 
    at com.authentic.architecture.interchange.accesspoint.SocketConnectionHandler.messageReceivedEvent(SocketConnectionHandler.java:282) 
    at com.authentic.architecture.interchange.accesspoint.SocketConnectionHandler.messageReceivedEvent(SocketConnectionHandler.java:261) 
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.processEventQueue(AbstractConnectionHandler.java:4110) 
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler.access$100(AbstractConnectionHandler.java:320) 
    at com.authentic.architecture.interchange.accesspoint.AbstractConnectionHandler$ConnectionHandlerRunner.execute(AbstractConnectionHandler.java:416) 
    at com.authentic.architecture.actions.ListProcessor.suspend(ListProcessor.java:1130) 
    at com.authentic.architecture.actions.ListProcessor.run(ListProcessor.java:775) 
    at java.lang.Thread.run(Unknown Source) 
Caused by: java.lang.NumberFormatException: For input string: "^123" 
    at java.lang.NumberFormatException.forInputString(Unknown Source) 
    at java.lang.Integer.parseInt(Unknown Source) 
    at java.lang.Integer.parseInt(Unknown Source) 
    at com.authentic.mapper.parsing.LengthVar.readBytes(LengthVar.java:170) 
    ... 17 more 

我要解析此登錄到以下字段：時間戳，日誌級別，記錄器，味精，堆棧跟蹤。

我已經使用的多過濾器：

multiline { 
pattern => "%{TIME:timestamp}" 
negate => true 
what => 「previous」 
} 

和i的神交濾波器所用的模式：

match=>{"message"=>"%{TIME:timestamp} %{LOGLEVEL:loglevel} \s*\[%{DATA:logger}\]\s*%{GREEDYDATA:msg}\n*(?<stacktrace>(.|\r|\n)*)"} 

我有http://grokconstructor.appspot.com/do/match檢查它。但爲stacktrace字段得到了這個匹配錯誤。

請提出一些建議。 在此先感謝。

Answer 1

如果要匹配整個堆棧跟蹤，則需要多行篩選器。這多過濾器應爲你工作：

codec => multiline { 
     pattern => "^%{TIME} " 
     negate => true 
     what => previous 
    } 

說明：每個行沒有開始時間戳（如23：28：32.226）將被regocnized作爲前行的一部分。有關處理多線的請參見docs。

現在你的模式。以下工作對我來說：

%{TIME:timestamp} %{LOGLEVEL:loglevel} \[%{DATA:logger}\] %{GREEDYDATA:message}\n(?<stacktrace>(.|\r|\n)*) 

漂亮的自我解釋，我希望： 擺脫括號，例如[和]與\[和\]，\n匹配換行符。還要注意條目之間的空格。

關於如何匹配包括換行符在內的所有內容，最後一部分（堆棧跟蹤）也參見this question。

---

一套完整的配置可能是這個樣子：

input { 
    file { 
    path => "/var/log/yourlog.log" 
    start_position => "beginning" 
    codec => multiline { 
     pattern => "^%{TIME} " 
     negate => true 
     what => previous 
    } 
    } 
} 
filter { 
    grok { 
    match => [ "message", "%{TIME:timestamp} %{LOGLEVEL:loglevel} \[%{DATA:logger}\] %{GREEDYDATA:message}\n(?<stacktrace>(.|\r|\n)*)" ] 
    } 
} 

---

結果上http://grokconstructor.appspot.com：