Logstash魷魚日期無效格式

我正在一個服務器上運行ELK堆棧，在該服務器上我集成了Squid access.log文件（組合格式也可以獲得用戶代理）。它正確地顯示在Kibana所有日誌和我logstash.stdout文件顯示條目正確：Logstash魷魚日期無效格式

{ 
    "message" => "192.168.2.26 - - [06/Apr/2016:19:08:51 +0200] \"GET https://r2---sn-4gxx-25gs.googlevideo.com/videoplayback/id/o-ABKSIYjx-CBdsfnjHVg6926-k9evTCAai6SfBJ7AGoNA/itag/311/source/youtube/govp/slices%3D0-713,52432612-53212629/gosq/63/file/seg.ts?dnc=1&cpn=yH1AZUBmkDfs_UGh&sgovp=clen=186211243;dur=1154.849;itag=298;lmt=1459931359321727;gir=yes&ip=88.177.83.67&ratebypass=yes&mt=1459962265&playlist_type=DVR&ms=au&ipbits=0&pl=12&mv=m&mm=31&expire=1459984025&upn=egKraIHQnU8&pfa=1&keepalive=yes&fexp=9408541,9416137,9416891,9417405,9420452,9422542,9422596,9426927,9427015,9427902,9428269,9428333,9428398,9428567,9428769,9430014,9430058,9431045,9431614,9431850,9431976,9432026,9432429,9432631,9433548&sver=3&sparams=hls_chunk_host,id,initcwndbps,ip,ipbits,itag,mm,mn,ms,mv,pfa,pl,playlist_type,ratebypass,requiressl,sgovp,source,expire&signature=C55E2BDA79965F9062CB2306C7B082776587104A.3BACE1898A53453B99CF50CA230E127E8EC095E8&requiressl=yes&initcwndbps=682500&hls_chunk_host=r2---sn-4gxx-25gs.googlevideo.com&mn=sn-4gxx-25gs&key=yt6&dur=5.067 HTTP/1.1\" 200 350062 \"-\" \"AppleCoreMedia/1.0.0.13B143 (iPad; U; CPU OS 9_1 like Mac OS X; fr_fr)\" TCP_MISS_ABORTED:ORIGINAL_DST", 
    "@version" => "1", 
"@timestamp" => "2016-04-06T17:08:51.000Z", 
     "path" => "/var/log/squid/access.log", 
     "host" => "secbox.localdomain.local", 
     "type" => "squid", 
    "clientip" => "192.168.2.26", 
     "ident" => "-", 
     "auth" => "-", 
    "timestamp" => "06/Apr/2016:19:08:51 +0200", 
     "verb" => "GET", 
    "request" => "https://r2---sn-4gxx-25gs.googlevideo.com/videoplayback/id/o-ABKSIYjx-CBdsfnjHVg6926-k9evTCAai6SfBJ7AGoNA/itag/311/source/youtube/govp/slices%3D0-713,52432612-53212629/gosq/63/file/seg.ts?dnc=1&cpn=yH1AZUBmkDfs_UGh&sgovp=clen=186211243;dur=1154.849;itag=298;lmt=1459931359321727;gir=yes&ip=88.177.83.67&ratebypass=yes&mt=1459962265&playlist_type=DVR&ms=au&ipbits=0&pl=12&mv=m&mm=31&expire=1459984025&upn=egKraIHQnU8&pfa=1&keepalive=yes&fexp=9408541,9416137,9416891,9417405,9420452,9422542,9422596,9426927,9427015,9427902,9428269,9428333,9428398,9428567,9428769,9430014,9430058,9431045,9431614,9431850,9431976,9432026,9432429,9432631,9433548&sver=3&sparams=hls_chunk_host,id,initcwndbps,ip,ipbits,itag,mm,mn,ms,mv,pfa,pl,playlist_type,ratebypass,requiressl,sgovp,source,expire&signature=C55E2BDA79965F9062CB2306C7B082776587104A.3BACE1898A53453B99CF50CA230E127E8EC095E8&requiressl=yes&initcwndbps=682500&hls_chunk_host=r2---sn-4gxx-25gs.googlevideo.com&mn=sn-4gxx-25gs&key=yt6&dur=5.067", 
"httpversion" => "1.1", 
    "response" => "200", 
     "bytes" => "350062", 
    "referrer" => "\"-\"", 
     "agent" => "\"AppleCoreMedia/1.0.0.13B143 (iPad; U; CPU OS 9_1 like Mac OS X; fr_fr)\"", 
     "tags" => [ 
    [0] "squid" 
]

}

然而，有關日期錯誤logstash.log文件聲稱：

"error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"06/Apr/2016:18:51:04 +0200\" is malformed at \"/Apr/2016:18:51:04 +0200\""}}}}, :level=>:warn}

我/etc/logstash/conf.d/squid.conf文件中定義如下：

input { 
    file { 
    type => "squid" 
    path => [ "/var/log/squid/access.log" ] 
    } 
} 

filter { 
    if [type] == "squid" { 
    grok { 
     match => { "message" => "%{COMBINEDAPACHELOG}" } 
     add_tag => ["squid"] 
    } 
    date { 
     match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] 
    } 
    } 
} 

output { 
    elasticsearch { hosts => localhost } 
    stdout { codec => rubydebug } 
}

我不明白爲什麼我的日誌似乎被logstash正確解析了這條錯誤消息。

有什麼想法？預先感謝您的意見。

來源

2016-04-06 Sebastien Damaye

解析（grok）正在工作，但日期過濾器失敗。它似乎認爲你的輸入和模式在第一個斜槓後不匹配，但[調試器]（https://staging-jodadebug.herokuapp.com/）喜歡它。 –

雖然已經很長時間了，以爲答案會幫助那些正在面對這個問題的人。 Logstash date invalid format

網址給我的答案，使用變異做時間格式發生變異的取代{ GSUB => [ 「時間戳」，「/」，「 - 」] } 日期{ 匹配=> [「timestamp」，「your format」] }

來源

2016-12-27 07:07:31

Logstash魷魚日期無效格式

回答

相關問題