Python中，要使用日誌旋轉和壓縮

Question

登錄任何人都可以提出一個方法在Python做記錄：每天

• 日誌旋轉
• 壓縮日誌時，他們正在旋轉
• 可選 - 刪除最早的日誌文件保存的自由空間X MB
• 可選 - SFTP日誌文件服務器

感謝您的任何答覆， 弗雷德

Answer 1

• 日誌每天旋轉：使用日誌的TimedRotatingFileHandler
• 壓縮：設置encoding='bz2'參數。 （注意這個「技巧」只適用於Python2，'Python2'不再被認爲是Python3的編碼。）
• 可選 - 刪除最舊的日誌文件以保留XMB的可用空間。 您可以（間接）使用RotatingFileHandler來安排此操作。通過設置maxBytes參數，日誌文件在達到一定大小時會翻轉。通過設置參數backupCount，您可以控制保留多少次翻車。這兩個參數一起允許您控制日誌文件消耗的最大空間。您也可以繼承TimeRotatingFileHandler以將此行爲也納入其中。

---

只是爲了好玩，這裏是你怎麼能繼承TimeRotatingFileHandler。當您運行下面的腳本時，它會將日誌文件寫入/tmp/log_rotate*。

對於time.sleep（如0.1）的值較小，日誌文件填滿很快，達到maxBytes限制，然後再翻轉。

對於較大的time.sleep（如1.0），日誌文件緩慢填滿，未達到maxBytes限制，但當達到定時間隔（10秒）時，它們會翻轉。

下面的所有代碼來自logging/handlers.py。我只需以最簡單的方式將RotatingFileHandler與TimeRotatingFileHandler網格化即可。

import time 
import re 
import os 
import stat 
import logging 
import logging.handlers as handlers 


class SizedTimedRotatingFileHandler(handlers.TimedRotatingFileHandler): 
    """ 
    Handler for logging to a set of files, which switches from one file 
    to the next when the current file reaches a certain size, or at certain 
    timed intervals 
    """ 

    def __init__(self, filename, maxBytes=0, backupCount=0, encoding=None, 
       delay=0, when='h', interval=1, utc=False): 
     handlers.TimedRotatingFileHandler.__init__(
      self, filename, when, interval, backupCount, encoding, delay, utc) 
     self.maxBytes = maxBytes 

    def shouldRollover(self, record): 
     """ 
     Determine if rollover should occur. 

     Basically, see if the supplied record would cause the file to exceed 
     the size limit we have. 
     """ 
     if self.stream is None:     # delay was set... 
      self.stream = self._open() 
     if self.maxBytes > 0:     # are we rolling over? 
      msg = "%s\n" % self.format(record) 
      # due to non-posix-compliant Windows feature 
      self.stream.seek(0, 2) 
      if self.stream.tell() + len(msg) >= self.maxBytes: 
       return 1 
     t = int(time.time()) 
     if t >= self.rolloverAt: 
      return 1 
     return 0 


def demo_SizedTimedRotatingFileHandler(): 
    log_filename = '/tmp/log_rotate' 
    logger = logging.getLogger('MyLogger') 
    logger.setLevel(logging.DEBUG) 
    handler = SizedTimedRotatingFileHandler(
     log_filename, maxBytes=100, backupCount=5, 
     when='s', interval=10, 
     # encoding='bz2', # uncomment for bz2 compression 
    ) 
    logger.addHandler(handler) 
    for i in range(10000): 
     time.sleep(0.1) 
     logger.debug('i=%d' % i) 

demo_SizedTimedRotatingFileHandler() 

Answer 2

除了unutbu的回答：下面是如何修改TimedRotatingFileHandler以使用zip文件進行壓縮。

import logging 
import logging.handlers 
import zipfile 
import codecs 
import sys 
import os 
import time 
import glob 


class TimedCompressedRotatingFileHandler(logging.handlers.TimedRotatingFileHandler): 
    """ 
    Extended version of TimedRotatingFileHandler that compress logs on rollover. 
    """ 
    def doRollover(self): 
     """ 
     do a rollover; in this case, a date/time stamp is appended to the filename 
     when the rollover happens. However, you want the file to be named for the 
     start of the interval, not the current time. If there is a backup count, 
     then we have to get a list of matching filenames, sort them and remove 
     the one with the oldest suffix. 
     """ 

     self.stream.close() 
     # get the time that this sequence started at and make it a TimeTuple 
     t = self.rolloverAt - self.interval 
     timeTuple = time.localtime(t) 
     dfn = self.baseFilename + "." + time.strftime(self.suffix, timeTuple) 
     if os.path.exists(dfn): 
      os.remove(dfn) 
     os.rename(self.baseFilename, dfn) 
     if self.backupCount > 0: 
      # find the oldest log file and delete it 
      s = glob.glob(self.baseFilename + ".20*") 
      if len(s) > self.backupCount: 
       s.sort() 
       os.remove(s[0]) 
     #print "%s -> %s" % (self.baseFilename, dfn) 
     if self.encoding: 
      self.stream = codecs.open(self.baseFilename, 'w', self.encoding) 
     else: 
      self.stream = open(self.baseFilename, 'w') 
     self.rolloverAt = self.rolloverAt + self.interval 
     if os.path.exists(dfn + ".zip"): 
      os.remove(dfn + ".zip") 
     file = zipfile.ZipFile(dfn + ".zip", "w") 
     file.write(dfn, os.path.basename(dfn), zipfile.ZIP_DEFLATED) 
     file.close() 
     os.remove(dfn) 

Answer 3

另一種方式來壓縮旋轉過程中的日誌文件（在Python 3.3新）使用BaseRotatingHandler（和所有繼承）類屬性肩例如：

import gzip 
import os 
import logging 
import logging.handlers 

class GZipRotator: 
    def __call__(self, source, dest): 
     os.rename(source, dest) 
     f_in = open(dest, 'rb') 
     f_out = gzip.open("%s.gz" % dest, 'wb') 
     f_out.writelines(f_in) 
     f_out.close() 
     f_in.close() 
     os.remove(dest) 

logformatter = logging.Formatter('%(asctime)s;%(levelname)s;%(message)s') 
log = logging.handlers.TimedRotatingFileHandler('debug.log', 'midnight', 1, backupCount=5) 
log.setLevel(logging.DEBUG) 
log.setFormatter(logformatter) 
log.rotator = GZipRotator() 

logger = logging.getLogger('main') 
logger.addHandler(log)  
logger.setLevel(logging.DEBUG) 

.... 

更多，你可以看到here。

Answer 4

要複製的文件，GZIP複製的文件（使用劃時代的時間），然後在不打亂記錄模塊的方式清理掉現有的文件：

import gzip 
import logging 
import os 
from shutil import copy2 
from time import time 

def logRoll(logfile_name): 
    log_backup_name = logfile_name + '.' + str(int(time())) 
    try: 
     copy2(logfile_name, log_backup_name) 
    except IOError, err: 
     logging.debug(' No logfile to roll') 
     return 
    f_in = open(log_backup_name, 'rb') 
    f_out = gzip.open(log_backup_name + '.gz', 'wb') 
    f_out.writelines(f_in) 
    f_out.close() 
    f_in.close() 
    os.remove(log_backup_name) 
    f=open(logfile_name, 'w') 
    f.close() 

Answer 5

我想這是爲時已晚加入派對，但這是我所做的。我創建了一個繼承logging.handlers.RotatingFileHandler類的新類，並在移動它之前添加了幾行gzip文件。

https://github.com/rkreddy46/python_code_reference/blob/master/compressed_log_rotator.py

#!/usr/bin/env python 

# Import all the needed modules 
import logging.handlers 
import sys 
import time 
import gzip 
import os 
import shutil 
import random 
import string 

__version__ = 1.0 
__descr__ = "This logic is written keeping in mind UNIX/LINUX/OSX platforms only" 


# Create a new class that inherits from RotatingFileHandler. This is where we add the new feature to compress the logs 
class CompressedRotatingFileHandler(logging.handlers.RotatingFileHandler): 
    def doRollover(self): 
     """ 
     Do a rollover, as described in __init__(). 
     """ 
     if self.stream: 
      self.stream.close() 
     if self.backupCount > 0: 
      for i in range(self.backupCount - 1, 0, -1): 
       sfn = "%s.%d.gz" % (self.baseFilename, i) 
       dfn = "%s.%d.gz" % (self.baseFilename, i + 1) 
       if os.path.exists(sfn): 
        # print "%s -> %s" % (sfn, dfn) 
        if os.path.exists(dfn): 
         os.remove(dfn) 
        os.rename(sfn, dfn) 
      dfn = self.baseFilename + ".1.gz" 
      if os.path.exists(dfn): 
       os.remove(dfn) 
      # These two lines below are the only new lines. I commented out the os.rename(self.baseFilename, dfn) and 
      # replaced it with these two lines. 
      with open(self.baseFilename, 'rb') as f_in, gzip.open(dfn, 'wb') as f_out: 
       shutil.copyfileobj(f_in, f_out) 
      # os.rename(self.baseFilename, dfn) 
      # print "%s -> %s" % (self.baseFilename, dfn) 
     self.mode = 'w' 
     self.stream = self._open() 

# Specify which file will be used for our logs 
log_filename = "/Users/myname/Downloads/test_logs/sample_log.txt" 

# Create a logger instance and set the facility level 
my_logger = logging.getLogger() 
my_logger.setLevel(logging.DEBUG) 

# Create a handler using our new class that rotates and compresses 
file_handler = CompressedRotatingFileHandler(filename=log_filename, maxBytes=1000000, backupCount=10) 

# Create a stream handler that shows the same log on the terminal (just for debug purposes) 
view_handler = logging.StreamHandler(stream=sys.stdout) 

# Add all the handlers to the logging instance 
my_logger.addHandler(file_handler) 
my_logger.addHandler(view_handler) 

# This is optional to beef up the logs 
random_huge_data = "".join(random.choice(string.ascii_letters) for _ in xrange(10000)) 

# All this code is user-specific, write your own code if you want to play around 
count = 0 
while True: 
    my_logger.debug("This is the message number %s" % str(count)) 
    my_logger.debug(random_huge_data) 
    count += 1 
    if count % 100 == 0: 
     count = 0 
     time.sleep(2) 

Answer 6

我認爲最好的選擇將是使用目前執行的TimedRotatingFileHandler和重命名日誌文件，以旋轉的版本只是壓縮它後：

import zipfile 
import os 
from logging.handlers import TimedRotatingFileHandler 


class TimedCompressedRotatingFileHandler(TimedRotatingFileHandler): 
    """ 
    Extended version of TimedRotatingFileHandler that compress logs on rollover. 
    """ 
    def find_last_rotated_file(self): 
     dir_name, base_name = os.path.split(self.baseFilename) 
     file_names = os.listdir(dir_name) 
     result = [] 
     prefix = '{}.20'.format(base_name) # we want to find a rotated file with eg filename.2017-12-12... name 
     for file_name in file_names: 
      if file_name.startswith(prefix) and not file_name.endswith('.zip'): 
       result.append(file_name) 
     result.sort() 
     return result[0] 

    def doRollover(self): 
     super(TimedCompressedRotatingFileHandler, self).doRollover() 

     dfn = self.find_last_rotated_file() 
     dfn_zipped = '{}.zip'.format(dfn) 
     if os.path.exists(dfn_zipped): 
      os.remove(dfn_zipped) 
     with zipfile.ZipFile(dfn_zipped, 'w') as f: 
      f.write(dfn, dfn_zipped, zipfile.ZIP_DEFLATED) 
     os.remove(dfn)