Aggregate logstash filter config

Question

我的目標是將基於pId的事件組合到logstash中。但是我發現具有相同pId的事件不會被合併爲一個事件。我無法看到添加aggregation.Please幫助

登錄看起來像這樣之後的任何變化：

June 1st 2017, 11:51:26.992 {id} {pId} ClassName:methodName:99 [DEBUG] - Received request: 
June 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] - Id: abbababcajdfbjasndflsdlf 
June 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] - unique id: AAAAA 
June 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] Total time: 12 

這裏是我的配置：

filter { 
grok{ 
match => { "message" => "%{DATESTAMP:log_timestamp} %{DATA:id} %{DATA:pId} %{DATA:ClassName} [%{LOGLEVEL:severity}] - %{GREEDYDATA:message}" } 
} 
if [message] =~ /Received request:/ { 
aggregate { 
task_id => "%{pId}" 
code => "map['message'] = event['message']" 
map_action => "create" 
} 
} 
else if [message] =~ /Total time:^/ { 
aggregate { 
task_id => "%{pId}" 
code => "map['new_message'] = event['message'];event['new_message'] = map['new_message']" 
map_action => "update" 
end_of_task => true 
timeout => 120 
} 
} 
else { 
aggregate { 
task_id => "%{pId}" 
code => "map['new_message'] = event['message'];event['new_message'] = map['new_message']" 
map_action => "update" 
} 
} 
} 

Answer 1

骨料是這些過濾器的一個可確實很難得到正確的。在很大程度上，因爲Logstash是通過將螺栓設計爲並行處理管道而設計的，因此每個調用過濾器堆棧的aggregate對於管道來說都是唯一的，並且您不能確定所有事件都將通過相同的管道運行。開箱即用。

如果使用-w 1參數運行logstash以通過單個管道強制所有內容，則會出現此問題。

在這種情況下，我建議改爲使用input上的multiline編解碼器代替。這將所有的loglines集中在一個事件中，您可以稍後在過濾器階段進行解析。當然，這假定每個這些多行事件同時被放入，並且不會多路複用。如果你得到多路複用，那麼聚合就需要失去你的並行性。

input { 
    file { 
    path => "/var/log/app/debug_logs.log" 
    codec => multiline { 
     pattern => "Received request:" 
     negate => true 
     what => previous 
    } 
    } 
} 

這將搜索不符合您Received request:正則表達式，並將它們附加到上線的事件。當它看到Received request:它將開始一個新的事件。您的filter {}階段將會看到這個

message => "June 1st 2017, 11:51:26.992 {id} {pId} ClassName:methodName:99 [DEBUG] - Received request:\nJune 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] - Id: abbababcajdfbjasndflsdlf\nJune 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] - unique id: AAAAA\nJune 1st 2017, 11:51:26.993 {id} {pId} ClassName:methodName:100 [DEBUG] Total time: 12" 

哪個在並行上下文中更容易操作。