產生多個進程來編寫不同的文件Python

Question

這個想法是使用N進程編寫N文件。

要寫入的對文件數據從存儲在具有列表作爲值的字典多個文件來了，它看起來像這樣：

dic = {'file1':['data11.txt', 'data12.txt', ..., 'data1M.txt'], 
     'file2':['data21.txt', 'data22.txt', ..., 'data2M.txt'], 
     ... 
     'fileN':['dataN1.txt', 'dataN2.txt', ..., 'dataNM.txt']} 

所以file1是data11 + data12 + ... + data1M等。 。

所以我的代碼如下所示：

jobs = [] 
for d in dic: 
    outfile = str(d)+"_merged.txt" 
    with open(outfile, 'w') as out: 
     p = multiprocessing.Process(target = merger.merger, args=(dic[d], name, out)) 
     jobs.append(p) 
     p.start() 
     out.close() 

和merger.py看起來是這樣的：

def merger(files, name, outfile): 
    time.sleep(2) 
    sys.stdout.write("Merging %n...\n" % name) 

    # the reason for this step is that all the different files have a header 
    # but I only need the header from the first file. 
    with open(files[0], 'r') as infile: 
     for line in infile: 
      print "writing to outfile: ", name, line 
      outfile.write(line) 
    for f in files[1:]: 
     with open(f, 'r') as infile: 
      next(infile) # skip first line 
      for line in infile: 
       outfile.write(line) 
    sys.stdout.write("Done with: %s\n" % name) 

我看到文件夾上應該寫的文件，但它是空的。沒有頭，沒有什麼。我把印刷品放在那裏看是否一切正常，但沒有任何效果。

幫助！

Answer 1

由於工作進程與創建它們的主進程並行運行，因此在工作人員寫入文件之前，名爲out的文件會關閉。即使您因with聲明刪除out.close()，也會發生這種情況。而是傳遞每個進程的文件名並讓進程打開並關閉文件。

Answer 2

問題是您不關閉該子文件中的文件，因此內部緩衝的數據會丟失。您可以將文件移動到子節點，或者將所有東西封裝在try/finally塊中，以確保文件關閉。在父母打開的潛在優勢是，您可以在那裏處理文件錯誤。我不是說它是令人信服的，只是一個選擇。

def merger(files, name, outfile): 
    try: 
     time.sleep(2) 
     sys.stdout.write("Merging %n...\n" % name) 

     # the reason for this step is that all the different files have a header 
     # but I only need the header from the first file. 
     with open(files[0], 'r') as infile: 
      for line in infile: 
       print "writing to outfile: ", name, line 
       outfile.write(line) 
     for f in files[1:]: 
      with open(f, 'r') as infile: 
       next(infile) # skip first line 
       for line in infile: 
        outfile.write(line) 
     sys.stdout.write("Done with: %s\n" % name) 
    finally: 
     outfile.close() 

UPDATE

已經有大約父/子文件decriptors會發生什麼，在子文件有些混亂。如果程序退出時文件仍處於打開狀態，底層C庫不會將數據刷新到磁盤。理論是一個正常運行的程序在退出之前關閉事物。下面是一個例子，其中孩子因爲沒有關閉文件而丟失了數據。

import multiprocessing as mp 
import os 
import time 

if os.path.exists('mytestfile.txt'): 
    os.remove('mytestfile.txt') 

def worker(f, do_close=False): 
    time.sleep(2) 
    print('writing') 
    f.write("this is data") 
    if do_close: 
     print("closing") 
     f.close() 


print('without close') 
f = open('mytestfile.txt', 'w') 
p = mp.Process(target=worker, args=(f, False)) 
p.start() 
f.close() 
p.join() 
print('file data:', open('mytestfile.txt').read()) 

print('with close') 
os.remove('mytestfile.txt') 
f = open('mytestfile.txt', 'w') 
p = mp.Process(target=worker, args=(f, True)) 
p.start() 
f.close() 
p.join() 
print('file data:', open('mytestfile.txt').read()) 

我在Linux上運行它，我得到

without close 
writing 
file data: 
with close 
writing 
closing 
file data: this is data