2013-07-08 63 views
2

我試圖設置logstash來解析並將日誌傳送到另一臺服務器。幾乎每一次神交無法解析日誌,它會導致以下錯誤:Logstash:GROK導致`線程看門狗超時'

{:timestamp=>"2013-07-08T02:20:07.390000-0400", 
:message=>"thread watchdog timeout", 
:thread=>#<Thread:0x133b92c run>, 
:backtrace=>["file:/opt/logstash/logstash.jar!/logstash/filterworker.rb:46:in `backtrace'", 
"file:/opt/logstash/logstash.jar!/logstash/filterworker.rb:46:in `run'", 
"file:/opt/logstash/logstash.jar!/logstash/agent.rb:785:in `each'", 
"file:/opt/logstash/logstash.jar!/logstash/agent.rb:785:in `run_filter'", 
"file:/opt/logstash/logstash.jar!/logstash/agent.rb:492:in `run_with_config'"], 
:thread_watchdog=>2013-07-08 02:19:57 -0400, 
:age=>10.006, 
:cutoff=>10, 
:state=>{:event=>#<LogStash::Event:0xa02ea9 @data={"@source"=>"file://clstaging12//home/xyz/xyz.com/apps/logs/mailerjob_log_2013_07_04.txt", 
"@tags"=>[], 
"@fields"=>{}, 
"@timestamp"=>"2013-07-08T06:19:50.114Z", 
"@source_host"=>"clstaging12", 
"@source_path"=>"//home/xyz/xyz.com/apps/logs/mailerjob_log_2013_07_04.txt", 
"@message"=>"PID:31730 2013-07-04T13:59:03-05:00 DEBUG :[property_listing_contact.php] Inside getBrokersContactInfoReceivedUserCount of the PropertyListing", 
"@type"=>"zend_log"}, 
@cancelled=false>, 
:filter=><LogStash::Filters::Grok type=>"zend_log", 
patterns_dir=>["/home/xyz/xyz.com/conf/patterns"], 
pattern=>["%{ZEND_LOG}"], 
match=>{"@message"=>["%{ZEND_LOG}"]}, 
tag_on_failure=>["_grokparsefailure"]>}, 
:level=>:fatal} 

模式Zend_Log進行是:

ZEND_LOG_SIGNATURE (?:IP\:(?<clientip>(?:\%ip\%|%{IP})))?PID\:%{NUMBER:pid} 
ZEND_CONTEXT [^\]]+ 
ZEND_LOG %{ZEND_LOG_SIGNATURE} %{TIMESTAMP_ISO8601:timestamp} %{WORD:level} \:\[%{ZEND_CONTEXT:context}\] %{GREEDYDATA:message} 

這將導致logstash代理每隔幾分鐘崩潰,使得它幾乎無法使用。 我看了很多現有的JIRA提交的錯誤,但沒有任何運氣。這裏有幾個環節:

https://logstash.jira.com/browse/LOGSTASH-508

https://logstash.jira.com/browse/LOGSTASH-525

https://logstash.jira.com/browse/LOGSTASH-765

回答

2

我一直沒能徹底解決這個問題,但經過進一步的調查,我發現,發生錯誤時神交不能以匹配整個線路,即它仍在等待更多輸入。

這可能完全是因爲我試圖解析的日誌是多行的,而且我使用了一個貪婪的正則表達式。我通過使我的正則表達式更強類型來解決這個問題。