如何配置FileBeat和Logstash在Elasticsearch中添加XML文件？

Question

我是初學者。我自己的問題是配置FileBeat和Logstash以在CentOS 7上的Elasticsearch中添加XML文件。 我已經安裝了最新版本的filebeat，logstash，elasticsearch和Kibana，並在插件「elasticsearch-head」中獨立顯示內彈性搜索。而測試我安裝，我已經成功地從CentOS系統（在/ var/log/messages中）添加簡單的日誌文件，看看它裏面elasticsearch頭插件（6指數和26個碎片）： This is a viex of my elasticsearch-head plug-in

和現在，下一步是從XML文件添加日誌。閱讀完文檔後，我配置了filebeat和logstash。所有的服務都在運行，我嘗試命令「touch /mes/AddOf.xml」來嘗試激活filebeat事件，並將日誌轉發到logstash（AddOf.xml是我的日誌文件）。

我的XML數據結構是這樣的一個日誌事件：

<log4j:event logger="ServiceLogger" timestamp="1494973209812" level="INFO" thread="QueueWorker_1_38a0fec5-7c7f-46f5-a87a-9134fff1b493"> 
    <log4j:message>Traitement du fichier \\ifs-app-01\Interfaces_MES\AddOf\ITF_MES_01_01_d2bef200-3a85-11e7-1ab5-9a50967946c3.xml</log4j:message> 
    <log4j:properties> 
     <log4j:data name="log4net:HostName" value="MES-01" /> 
     <log4j:data name="log4jmachinename" value="MES-01" /> 
     <log4j:data name="log4net:Identity" value="" /> 
     <log4j:data name="log4net:UserName" value="SOFRADIR\svc_mes_sf" /> 
     <log4j:data name="LogName" value="UpdateOperationOf" /> 
     <log4j:data name="log4japp" value="MES_SynchroService.exe" /> 
    </log4j:properties> 
    <log4j:locationInfo class="MES_SynchroService.Core.FileManager" method="TraiteFichier" file="C:\src\MES_PROD\MES_SynchroService\Core\FileManager.cs" line="47" /> 
</log4j:event> 

我filebeat構造是這樣的（/etc/filebeat/filebeat.yml）：

filebeat.prospectors: 

# Each - is a prospector. Most options can be set at the prospector level, so 
# you can use different prospectors for various configurations. 
# Below are the prospector specific configurations. 
- input_type: log 

    # Paths that should be crawled and fetched. Glob based paths. 
    paths: 
    - /mes/*.xml 

    document_type: message 

    ### Multiline options 

    # Mutiline can be used for log messages spanning multiple lines. This is common 
    # for Java Stack Traces or C-Line Continuation 

    # The regexp Pattern that has to be matched. The example pattern matches all lines starting with [ 
    multiline.pattern: ^<log4j:event 

    # Defines if the pattern set under pattern should be negated or not. Default is false. 
    multiline.negate: true 

    # Match can be set to "after" or "before". It is used to define if lines should be append to a pattern 
    # that was (not) matched before or after or as long as a pattern is not matched based on negate. 
    # Note: After is the equivalent to previous and before is the equivalent to to next in Logstash 
    multiline.match: after 

#================================ Outputs ===================================== 

# Configure what outputs to use when sending the data collected by the beat. 
# Multiple outputs may be used. 

#----------------------------- Logstash output -------------------------------- 
output.logstash: 
    # The Logstash hosts 
    hosts: ["localhost:5044"] 

    # Optional SSL. By default is off. 
    # List of root certificates for HTTPS server verifications 
    ssl.certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"] 

    # Certificate for SSL client authentication 
    #ssl.certificate: "/etc/pki/client/cert.pem" 

    # Client Certificate Key 
    #ssl.key: "/etc/pki/client/cert.key" 

我的輸入logstash配置（/etc/logstash/conf.d/01-beats-input.conf）：

input { 
beats { 
port => 5044 
ssl => true 
ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt" 
ssl_key => "/etc/pki/tls/private/logstash-forwarder.key" 
} 
} 

我的過濾器logstash configuratio N（/etc/logstash/conf.d/01-beats-filter.conf）：

filter 
{ 
    xml 
    { 
     source => "message" 
     xpath => 
     [ 
      "/log4j:event/log4j:message/text()", "messageMES" 
     ] 
     store_xml => true 
     target => "doc" 
    } 
} 

我的輸出logstash配置（/etc/logstash/conf.d/01-beats-output.conf）：

output { 
elasticsearch { 
hosts => ["localhost:9200"] 
sniffing => true 
manage_template => false 
index => "mes_log" 
document_type => "%{[@metadata][type]}" 
} 
} 

但是當我嘗試的命令「觸摸/mes/AddOf.xml」，或手動添加一個事件日誌中AddOf.xml，我沒有看到一個新的索引與事件從elasticsearch XML文件中記錄。

我已經看到logstash的XML插件的文檔（here），但我現在不需要如果我需要安裝某些東西？或者，也許我沒有爲filebeat發送日誌到logstash做正確的事情？

我非常積極地參與ELK堆棧學習。預先感謝您的專業知識和幫助。我會很感激 ！ :)

Answer 1

在你filebeat配置，爲multiline.pattern可能是正則表達式應該是單引號：

multiline.pattern: '^<log4j:event'