Logstash不發貨我的日誌文件的尾部，直到我終止logstash過程

Question

我有Logstash問題，我開始

bin/logstash -f logstash.conf 

的過程，它直到我按ctrl正常，但不是從文件中運行航海日誌+ c終止進程，只有它發送數據到elasticsearch。

我的問題是爲什麼我必須終止進程，開始將所有收集的數據發送到彈性？

logstash.conf：

input { 
    file { 
    type => logs 
    path => "/home/admin/logs/*" 
    start_position => beginning 
    sincedb_path => "/home/admin/sincedb" 
    ignore_older => 0 
    codec => multiline { 
     pattern => "^[0-2][0-3]:[0-5][0-9].*" 
     negate => "true" 
     what => "previous" 
    } 
    } 
} 
filter { 
grok { 
    match => { 
     message => "%{NOTSPACE:date}\t+%{INT:done}\t+%{INT:idnumber}\t+SiteID=%{INT:SiteID};DateFrom=%{NOTSPACE:DateFrom};DateTo=%{NOTSPACE:DateTo};RoomCode=%{INT:RoomCode};RatePlanRoomID=%{INT:RatePlanRoomID};DaySetupIDs:%{NOTSPACE:DaySetupIDs};RatePlanID=%{INT:RatePlanID};RatePlanCode=%{INT:RatePlanCode};Calculation=%{WORD:Calculation};IsClosed=%{INT:IsClosed};BaseOccupancy=%{INT:BaseOccupancy};MaxOccupancy=%{INT:MaxOccupancy};MinLOS=%{INT:MinLOS};IsDirtyRate=%{INT:IsDirtyRate};IsDirtyAvail=%{INT:IsDirtyAvail};BasePrice=%{NOTSPACE:BasePrice};ExtraPriceAdult=%{NOTSPACE:ExtraPrice};Currency=%{WORD:Currency};Inventory=%{INT:Inventory};Reservations=%{INT:Reservations};\n+%{GREEDYDATA:Request}\n+%{GREEDYDATA:Response}" 
     } 
} 
grok { 
    match => { 
     path => "%{GREEDYDATA:pp}/%{INT:filedate}_%{INT:fileid}_%{INT:ChannelID}_%{GREEDYDATA:action}_%{INT:isdone}\.bin" 
     } 
} 
    mutate { 
     lowercase => [ "action" ] 
    } 
} 
output { 
    stdout { 
    codec => rubydebug 
    } 
    elasticsearch { 
     hosts => ["localhost:9200"] 
     index => "crs-%{SiteID}" 
    } 
} 

Logstash：

15:25:10.496 [[main]-pipeline-manager] INFO logstash.pipeline - Pipeline main started 
15:25:10.632 [Api Webserver] INFO logstash.agent - Successfully started Logstash API endpoint {:port=>9600} 

然後我等待，等待 - 什麼，直到我殺的過程：

^C15:28:33.530 [SIGINT handler] WARN logstash.runner - SIGINT received. Shutting down the agent. 
15:28:33.544 [LogStash::Runner] WARN logstash.agent - stopping pipeline {:id=>"main"} 
{ 
       "date" => "11:48:49", 
       "pp" => "/home/admin/logs", 
     "BasePrice" => "270.00", 
      "filedate" => "115151", 
      "idnumber" => "106274275", 
      "IsClosed" => "0", 
       "type" => "logs", 
       "path" => "/home/admin/logs/115151_00_3_SavePrice_1.bin", 
     "RatePlanID" => "13078", 
     "MaxOccupancy" => "0", 
      "Currency" => "PLN", 
      "@version" => "1", 
       "host" => "xxxx", 
     "Reservations" => "0", 
      "action" => "saveprice", 
     "Calculation" => "N", 
    "BaseOccupancy" => "0", 
      "isdone" => "1", 
     "RatePlanCode" => "975669", 
      "fileid" => "00", 
      "MinLOS" => "1", 
      "SiteID" => "1709", 
    "RatePlanRoomID" => "61840", 
     "ExtraPrice" => ";ExtraPriceChild=", 
      "Request" => "<?xml version=\"1.0\"?><request>xxx", 
     "ChannelID" => "3", 
       "done" => "1", 
     "DaySetupIDs" => "0=39355996", 
     "IsDirtyRate" => "1", 
       "tags" => [ 
     [0] "multiline" 
    ], 
      "Response" => "<ok></ok>", 
     "IsDirtyAvail" => "1", 
     "@timestamp" => 2016-12-29T15:28:33.801Z, 
      "RoomCode" => "28114102", 
      "DateFrom" => "2016-12-05", 
     "Inventory" => "3", 
      "DateTo" => "2016-12-05" 
} 

Answer 1

添加auto_flush_interval幫我

codec => multiline { 
    auto_flush_interval => 1 
    pattern => "^[0-2][0-3]:[0-5][0-9].*" 
    negate => "true" 
    what => "previous" 
} 

Answer 2

，如果你改變你的sincedb_path因爲什麼如下：

sincedb_path => "/dev/null" 

除此之外，您的conf上的file插件的屬性似乎是現貨。希望這個SO聽起來很方便。據我從官方文檔瞭解，sincedb_path只需要一個目錄logstash有註冊表的寫入權限。默認情況下Logstash會保留所有記錄在$ HOME/.sincedb *。

如果你不想在上面的方法繼續前進，你總是可以清除.sincedb文件的缺省目錄中，然後嘗試重新解析您的logstash的conf。更有可能logstash會自動獲取更改。

編輯：

我想我發現了你的問題，你就錯過了報價start_position，這應該是這樣的：

start_position => "beginning"