配置Rsyslog（Docker-> TCP-> Rsyslog-> ElasticSearch）

Question

我是rsyslog，遠程日誌記錄和elasticsearch的新手。

我配置了一個python腳本（從docker容器運行）通過TCP將日誌記錄發送到$ HOST：$ PORT。

我已經安裝了rsyslog，模塊mmnormalize和模塊omelasticsearch。

現在我想了解我的rsyslog.conf（在主機上）應該如何使用elasticsearch收集日誌（來自172.17.0.0/16）。

謝謝！

Answer 1

這裏是我解決了這個問題：

# /etc/rsyslog.d/docker.rb 
version=2 
# My sample record 
# [Apr 25 12:00]$CONTAINER_HOSTNAME:INFO:Package.Module.Sub-Module:Hello World 
# 
# Here there is the rule to parse the log records into trees 
rule=:[%date:char-to:]%]%hostname:char-to::%:%level:char-to::%:%file:char-to::%:%message:rest% 
# 
# alternative to set date field in rfc3339 format 
# rule=:[%date:date-rfc3339%]%hostname:char-to::%:%level:char-to::%:%file:char-to::%:%message:rest% 

---

# /etc/rsyslog.conf 
module(load="mmnormalize") 
module(load="omelasticsearch") 
module(load="imtcp") 

# apply to log records coming from address:port the specified rule 
input(type="imtcp" 
     address="127.0.0.1" # $HOST 
     port="514"   # $PORT 
     ruleset="docker-rule") 

# define the rule in two actions; parsing the log record into a tree with 
# root $! ($!son-node!grandson-node...) and adding to the elasticsearch index 
# 'docker-logs' the parsed tree, but in a JSON format (specified in a template) 
ruleset(name="docker-rule"){ 
    action(type="mmnormalize" 
      rulebase="/etc/rsyslog.d/docker.rb" 
      useRawMsg="on" 
      path="$!") 
    action(type="omelasticsearch" 
      template="docker-template" 
      searchIndex="docker-logs" 
      bulkmode="on" 
      action.resumeretrycount="-1") 
} 

# define the template: 
# 'constants' are simply putting into the record JSON delimiters as '{' or ',' 
# 'properties' are simply putting the values of the parsed tree into fields 
# named in the previous constant statements through 'value="..."' 
# the result is a JSON record like: 
# { "@timestamp":"foo", 
# "hostname":"bar", 
# "level":"foo", 
# "file":"bar", 
# "message":"foo" 
# } 
template(name="docker-template" type="list"){ 
    constant(value="{") 
     constant(value="\"@timestamp\":") 
      constant(value="\"") 
       # because kibana would use '$!date' as string not as date 
       # that is the only field not from the parsed tree 
       property(name="timereported" dateFormat="rfc3339") 
      constant(value="\"") 
     constant(value=",") 
     constant(value="\"hostname\":") 
      constant(value="\"") 
       property(name="$!hostname") 
      constant(value="\"") 
     constant(value=",") 
     constant(value="\"level\":") 
      constant(value="\"") 
       property(name="$!level") 
      constant(value="\"") 
     constant(value=",") 
     constant(value="\"file\":") 
      constant(value="\"") 
       property(name="$!file") 
      constant(value="\"") 
     constant(value=",") 
     constant(value="\"message\":") 
      constant(value="\"") 
       property(name="$!message") 
      constant(value="\"") 
    constant(value="}") 
} 

下一個安裝kibana可以「配置索引模式」，簡單的設置：「指標名稱或圖案」，以「泊塢窗日誌「和」時間字段名稱「設置爲」@timestamp「

請注意，日誌源（172.17.0.0/16）不受控制;如果正確解析，則發送到$ HOST：$ PORT的每個日誌記錄都將被插入到elasticsearch索引中。