rsyslog性能優化

應該做些什麼來設置rsyslog以獲得最佳性能？rsyslog性能優化

我們可以允許一些項目在服務器崩潰或丟失時丟失。
我們將把日誌保存到MySQL DB中。
我們希望能夠處理每秒至少100次的日誌寫入，延遲時間爲0.001 - 0.005秒。
我們正在從PHP應用程序寫入日誌。

謝謝你的幫助。

2013-04-09 Victor Smirnov

我們剛剛經歷了一次使用MongoDB作爲數據庫的類似練習，所以我將記錄我們所做的並希望它對您有所幫助。

這是我們第一次使用rsyslog，因此花費了一點努力找到正確的文檔並將所有內容組合在一起。最後，我們的測試驅動程序（我們使用SoapUI）能夠通過php web服務獲得1000 TPS，該服務使用rsyslog寫入事務的摘要記錄。

我們發現下面的文章得到了我們開始說：

的概述是，你能rsyslog現在的隊列基礎設施，當使寫入傳入郵件到磁盤守護進程的內存隊列已滿。在我們的例子中，我們啓用了$ ActionQueueSaveOnShutdown，這聽起來像你不需要。然後，您將配置rsyslog規則集來解析傳入的消息並將它們傳遞給MySQL的輸出處理程序。最後，你的php腳本將使用openlog（）和syslog（）來寫任何你想記錄的數據。哦，我們還必須從源代碼編譯rsyslog，以啓用json/mongo插件，這本身就是一個練習。我們在Ubuntu 12.04上使用rsyslog 7.4.5。

我當然不是rsyslog的專家，但可以給你我們的配置文件和代碼作爲起點。再次，他們是爲MongoDB的，希望它給你一個想法做什麼和在哪裏改變你的實現的東西。

祝你好運！

/etc/rsyslog.conf：

$ModLoad imuxsock # provides support for local system logging 
$ModLoad imklog # provides kernel logging support (previously done by rklogd) 

# Load modules for MongoDB integration: json parser and MongoDB output driver 
module(load="mmjsonparse") 
module(load="ommongodb") 

# Use traditional timestamp format. 
# To enable high precision timestamps, comment out the following line. 
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat 

# Filter duplicated messages 
$RepeatedMsgReduction on 

# Set the default permissions for all log files. 
$FileOwner syslog 
$FileGroup adm 
$FileCreateMode 0640 
$DirCreateMode 0755 
$Umask 0022 
$PrivDropToUser syslog 
$PrivDropToGroup syslog 

# Where to place spool files 
$WorkDirectory /var/spool/rsyslog 

# use queue to decouple the db writes from default message handling 
# From http://www.rsyslog.com/doc/rsyslog_high_database_rate.html 
$MainMsgQueueFileName mainq  # set file name for main queue, also enables disk mode 
$ActionQueueType LinkedList  # use asynchronous processing 
$ActionQueueFileName mongodbq # set file name for mongo db queue, enables disk mode 
$ActionResumeRetryCount -1  # infinite retries on insert failure 
$ActionQueueSaveOnShutdown on # write all queue data to disk when rsyslogd is 
           # terminated (default is off) 

# Include all config files in /etc/rsyslog.d/ 
$IncludeConfig /etc/rsyslog.d/*.conf

/etc/rsyslog.d/10-mongo.conf：

input(type="imuxsock" socket="/dev/log") 

template(name="mongodblocal" type="subtree" subtree="$!") 

# use json parser for all "local0" facility messages, 
# if parsed successfully run the template to load the 
# message into the MongoDB database. 
if $syslogfacility-text == 'local0' then { 
     action(type="mmjsonparse") 
     if $parsesuccess == "OK" then { 
       # set some local vars that are appended onto the 
       # document that's written to MongoDB 
       set $!time = $timestamp; 
       set $!sys = $hostname; 
       set $!procid = $syslogtag; 
       set $!syslog_fac = $syslogfacility; 
       set $!syslog_sever = $syslogpriority; 
       set $!pid = $procid; 
       action(type="ommongodb" server="127.0.0.1" db="test" collection="syslog" template="mongodblocal") 
     } 
}

/etc/rsyslog.d/50-default.conf ：注意：這會禁用默認處理的「local0」消息。

# First some standard log files. Log by facility. 
auth,authpriv.*   /var/log/auth.log 

# don't write "local0" messages to syslog, 
# as they're processed using ommongodb (see 10-mongo.conf) 
*.*;local0,auth,authpriv.none -/var/log/syslog 

kern.*    -/var/log/kern.log 
mail.*    -/var/log/mail.log 

# Logging for the mail system. Split it up so that 
# it is easy to write scripts to parse these files. 
mail.err   /var/log/mail.err 

# Logging for INN news system. 
news.crit   /var/log/news/news.crit 
news.err   /var/log/news/news.err 
news.notice   -/var/log/news/news.notice 

# Emergencies are sent to everybody logged in. 
*.emerg        :omusrmsg:*

PHP的Web服務相關的來電：

// open syslog, include the process ID and open the connection to the logger 
// immediately, and use a user defined logging mechanism Local0 
openlog($SCRIPT_NAME, LOG_PID | LOG_NDELAY, LOG_LOCAL0); 
// note: calling closelog() is optional, and we don't use it 

... 
// construct $doc, which is what will be logged, change this as appropriate 
// for your implementation; here $ary_headers is the request's HTTP headers, 
// and $request/$response are what was posted/returned 
$doc = array("headers" => $ary_headers 
      ,"request" => $request 
      ,"response" => $response 
      ); 
... 

// write the log entry to syslog, where it queues it and writes it to MongoDB 
// NOTE: need the '@cee: ' prefix so the rsyslog json parser will process it 
// See: http://www.rsyslog.com/doc/rsyslog_conf_modules.html/mmjsonparse.html 

// JSON_BIGINT_AS_STRING = Encodes large integers as their original string value. 
// JSON_NUMERIC_CHECK = Encodes numeric strings as numbers. 
// JSON_UNESCAPED_SLASHES = Don't escape "/". 

syslog(LOG_INFO, '@cee: ' . json_encode($doc, JSON_BIGINT_AS_STRING | JSON_NUMERIC_CHECK | JSON_UNESCAPED_SLASHES));

來源

2013-10-29 18:02:03 Alan

這也許會有幫助，儘管它是關於Elasticsearch，而不是MySQL的： http://blog.sematext.com/2014/01/20/rsyslog-8-1-elasticsearch-output-performance/

但它說明了如何rsyslog現在的作品與隊列的基本知識和線程，並且您有一個可以從頭開始的新型配置。加上送你鏈接rsyslog現在的約隊列，規則集等

有關新型配置格式的詳細信息的文件，這是一個很好的資源：http://www.rsyslog.com/doc/rainerscript.html

來源

2014-01-25 10:21:49

rsyslog性能優化

回答

相關問題