比較兩個CSV文件並搜索相似的項目

Question

所以我有兩個CSV文件，我試圖比較並獲得類似項目的結果。 第一個文件，hosts.csv如下所示：

Path Filename Size Signature 
C:\  a.txt  14kb
D:\  b.txt  99kb 678910 
C:\  c.txt  44kb 111213 

第二個文件，masterlist.csv如下所示：

Filename Signature 
b.txt  678910 
x.txt  111213 
b.txt  777777 
c.txt  999999 

正如你所看到的行不匹配和masterlist .csv總是大於hosts.csv文件。我想要搜索的唯一部分是簽名部分。我知道這看起來是這樣的：

hosts[3] == masterlist[1] 

我在找，這將使我類似如下（基本hosts.csv文件與新的結果列）的解決方案：

Path Filename Size Signature RESULTS 
C:\  a.txt  14kbNOT FOUND in masterlist 
D:\  b.txt  99kb 678910  FOUND in masterlist (row 1) 
C:\  c.txt  44kb 111213  FOUND in masterlist (row 2) 

我搜索了這些帖子，發現類似於這個here，但我不太瞭解它，因爲我還在學習python。

編輯使用Python 2.6

Answer 1

編輯：雖然我的解決方案正常工作，看看下面的Martijn的回答更高效的解決方案。

你可以找到Python CSV模塊here的文檔。

你要找什麼是這樣的：

import csv 

f1 = file('hosts.csv', 'r') 
f2 = file('masterlist.csv', 'r') 
f3 = file('results.csv', 'w') 

c1 = csv.reader(f1) 
c2 = csv.reader(f2) 
c3 = csv.writer(f3) 

masterlist = list(c2) 

for hosts_row in c1: 
    row = 1 
    found = False 
    for master_row in masterlist: 
     results_row = hosts_row 
     if hosts_row[3] == master_row[1]: 
      results_row.append('FOUND in master list (row ' + str(row) + ')') 
      found = True 
      break 
     row = row + 1 
    if not found: 
     results_row.append('NOT FOUND in master list') 
    c3.writerow(results_row) 

f1.close() 
f2.close() 
f3.close() 

Answer 2

Python的CSV和收藏模塊，具體OrderedDict，是真正有用的在這裏。你想使用OrderedDict來保存鍵的順序等。你不需要，但它很有用！

import csv 
from collections import OrderedDict 


signature_row_map = OrderedDict() 


with open('hosts.csv') as file_object: 
    for line in csv.DictReader(file_object, delimiter='\t'): 
     signature_row_map[line['Signature']] = {'line': line, 'found_at': None} 


with open('masterlist.csv') as file_object: 
    for i, line in enumerate(csv.DictReader(file_object, delimiter='\t'), 1): 
     if line['Signature'] in signature_row_map: 
      signature_row_map[line['Signature']]['found_at'] = i 


with open('newhosts.csv', 'w') as file_object: 
    fieldnames = ['Path', 'Filename', 'Size', 'Signature', 'RESULTS'] 
    writer = csv.DictWriter(file_object, fieldnames, delimiter='\t') 
    writer.writer.writerow(fieldnames) 
    for signature_info in signature_row_map.itervalues(): 
     result = '{0} FOUND in masterlist {1}' 
     # explicit check for sentinel 
     if signature_info['found_at'] is not None: 
      result = result.format('', '(row %s)' % signature_info['found_at']) 
     else: 
      result = result.format('NOT', '') 
     payload = signature_info['line'] 
     payload['RESULTS'] = result 

     writer.writerow(payload) 

下面是使用測試CSV文件的輸出：

Path Filename  Size Signature  RESULTS 
C:\  a.txt 14kbNOT FOUND in masterlist 
D:\  b.txt 99kb 678910 FOUND in masterlist (row 1) 
C:\  c.txt 44kb 111213 FOUND in masterlist (row 2) 

請原諒的錯位，他們是製表符分隔:)

Answer 3

的csv模塊就派上用場了在解析的CSV文件。但爲了好玩，我只是將輸入分割爲空格來獲取數據。

只解析數據，爲masterlist.csv中的數據構建一個dict，簽名爲鍵，行號爲值。現在，對於hosts.csv的每一行，我們可以查詢dict，並確定masterlist.csv中是否存在相應的條目，如果是，那麼在哪一行。

#! /usr/bin/env python 

def read_data(filename): 
     input_source=open(filename,'r') 
     input_source.readline() 
     return [line.split() for line in input_source] 

if __name__=='__main__': 
     hosts=read_data('hosts.csv') 
     masterlist=read_data('masterlist.csv') 
     master=dict() 
     for index,data in enumerate(masterlist): 
       master[data[-1]]=index+1 
     for row in hosts: 
       try: 
         found="FOUND in masterlist (row %s)"%master[row[-1]] 
       except KeyError: 
         found="NOT FOUND in masterlist" 
       line=row+[found] 
       print "%s %s %s %s %s"%tuple(line) 

Answer 4

由srgerg的答案是非常低效的，因爲它運行在二次時間;這裏是一個線性時間溶液代替，使用Python 2.6兼容的語法：

import csv 

with open('masterlist.csv', 'rb') as master: 
    master_indices = dict((r[1], i) for i, r in enumerate(csv.reader(master))) 

with open('hosts.csv', 'rb') as hosts: 
    with open('results.csv', 'wb') as results:  
     reader = csv.reader(hosts) 
     writer = csv.writer(results) 

     writer.writerow(next(reader, []) + ['RESULTS']) 

     for row in reader: 
      index = master_indices.get(row[3]) 
      if index is not None: 
       message = 'FOUND in master list (row {})'.format(index) 
      else: 
       message = 'NOT FOUND in master list' 
      writer.writerow(row + [message]) 

這將產生一個字典，從masterlist.csv映射簽名行號第一。字典中的查找需要一定的時間，使得第hosts.csv行上的第二個循環與masterlist.csv中的行數無關。更不用說代碼更簡單了。