如何使用python將日誌文件拆分爲多個csv文件

Question

我對python和編碼一般都很陌生，所以對於任何愚蠢的問題都提前抱歉。我的程序需要根據關鍵字'MYLOG'將現有的日誌文件分成幾個* .csv文件（run1，.csv，run2.csv，...）。如果出現關鍵字，則應開始將兩個所需列複製到新文件中，直到關鍵字再次出現。完成後，需要有與關鍵字一樣多的csv文件。

---

53.2436  EXP  MYLOG: START RUN specs/run03_block_order.csv 
53.2589  EXP  TextStim: autoDraw = None 
53.2589  EXP  TextStim: autoDraw = None 
55.2257  DATA Keypress: t 
57.2412  DATA Keypress: t 
59.2406  DATA Keypress: t 
61.2400  DATA Keypress: t 
63.2393  DATA Keypress: t 
... 
89.2314  EXP  MYLOG: START BLOCK scene [specs/run03_block01.csv] 
89.2336  EXP  Imported specs/run03_block01.csv as conditions 
89.2339  EXP  Created sequence: sequential, trialTypes=9 
... 

---

[編輯]：每個文件輸出（運行* .CSV）應該是這樣的：

onset  type 
53.2436  EXP  
53.2589  EXP  
53.2589  EXP  
55.2257  DATA  
57.2412  DATA  
59.2406  DATA  
61.2400  DATA  
... 

---

該程序創建儘可能多的運行*。 csv根據需要，但我不能在我的新文件中存儲所需的列。完成後，我所得到的都是空的csv文件。如果我將計數器變量轉換爲== 1，它會創建一個包含所需列的大文件。

再次感謝！

import csv 

QUERY = 'MYLOG' 

with open('localizer.log', 'rt') as log_input: 
i = 0 

for line in log_input: 

    if QUERY in line: 
     i = i + 1 

     with open('run' + str(i) + '.csv', 'w') as output: 
      reader = csv.reader(log_input, delimiter = ' ') 
      writer = csv.writer(output) 
      content_column_A = [0] 
      content_column_B = [1] 

      for row in reader: 
       content_A = list(row[j] for j in content_column_A) 
       content_B = list(row[k] for k in content_column_B) 
       writer.writerow(content_A) 
       writer.writerow(content_B) 

Answer 1

看代碼有幾件事情是可能是錯誤的：

1. 的CSV讀者應該採取一個文件處理程序，不是一個單一的線。
2. 閱讀器分隔符不應該是單個空格字符，因爲它看起來像日誌中的實際分隔符是可變數量的多個空格字符。
3. 循環邏輯似乎有點關閉，使文件/行/行有點混亂。

您看到的可能是像下面的代碼（待澄清的問題）：

import csv 
NEW_LOG_DELIMITER = 'MYLOG' 

def write_buffer(_index, buffer): 
    """ 
    This function takes an index and a buffer. 
    The buffer is just an iterable of iterables (ex a list of lists) 
    Each buffer item is a row of values. 
    """ 
    filename = 'run{}.csv'.format(_index) 
    with open(filename, 'w') as output: 
     writer = csv.writer(output) 
     writer.writerow(['onset', 'type']) # adding the heading 
     writer.writerows(buffer) 

current_buffer = [] 
_index = 1 

with open('localizer.log', 'rt') as log_input: 
    for line in log_input: 
     # will deal ok with multi-space as long as 
     # you don't care about the last column 
     fields = line.split()[:2] 
     if not NEW_LOG_DELIMITER in line or not current_buffer: 
      # If it's the first line (the current_buffer is empty) 
      # or the line does NOT contain "MYLOG" then 
      # collect it until it's time to write it to file. 
      current_buffer.append(fields) 
     else: 
      write_buffer(_index, current_buffer) 
      _index += 1 
      current_buffer = [fields] # EDIT: fixed bug, new buffer should not be empty 
    if current_buffer: 
     # We are now out of the loop, 
     # if there's an unwritten buffer then write it to file. 
     write_buffer(_index, current_buffer) 

Answer 2

你可以用熊貓來簡化這個問題。

導入熊貓並讀入日誌文件。

import pandas as pd 

df = pd.read_fwf('localizer2.log', header=None) 
df.columns = ['onset', 'type', 'event'] 
df.set_index('onset', inplace=True) 

設置標誌，在第三欄== 'MYLOG'

df['flag'] = 0 
df.loc[df.event.str[:5] == 'MYLOG', 'flag'] = 1 
df.flag = df['flag'].cumsum() 

保存每次運行作爲一個獨立的運行* .csv文件

for i in range(1, df.flag.max()+1): 
    df.loc[df.flag == i, 'event'].to_csv('run{0}.csv'.format(i)) 

編輯： 看起來你格式與我原先假定的不同。更改爲使用pd.read_fwf。我的localizer.log文件是原始數據的複製和粘貼，希望這對您有用。我原來的帖子假設它沒有標題。如果確實有標題，請刪除header=None和df.columns = ['onset', 'type', 'event']。