Python代碼並行化使用多處理

Question

我真的很新來python mutli處理並試圖並行我的代碼，因爲它需要太長時間才能運行。我有一段代碼運行大量的數據來查找是否有任何文件被損壞。到目前爲止，我的代碼是：

def check_Corrupt_1(dirPath, logfile): 

    fileCheck = open(logfile, "w").close() 
    fileCheck = open(logfile, "w") 

    emptydir = [] 
    zero_size = {} 
    #entering the year to be checked (day number) 
    for fname in os.listdir(dirPath): 

     if(os.listdir(os.path.join(dirPath, fname)) == []): 
      emptydir.append(fname) 

     else: 

      #this makes sure that we do not enter an empty directory 
      if fname not in emptydir: 
       inPath = os.path.join(dirPath, fname) 

       for filename in os.listdir(inPath): 
        hdfinfo = os.stat(os.path.join(inPath, filename)) 

        if(hdfinfo.st_size == 0): 
         zero_size[filename] = True 

        else: 

         filepath = "/path/to/file" 

         strin = subprocess.Popen(["hdp", "dumpsds", "-h", os.path.join(inPath, filename)], stdout=subprocess.PIPE).communicate()[0] 
         #print(strin) 
         cmd = 'echo $?' 
         callno = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE) 
         #print(int(callno.stdout.read()[0])) 
         if(int(callno.stdout.read()[0]) != 0): 
          fileCheck.write(os.path.join(inPath, filename) + '\n') 

我每年有365個目錄，每個目錄包含很多要檢查的文件。我正在運行bash命令來檢查文件是否損壞，但是因爲我運行的bash命令有很長的輸出，所以這段代碼需要很長時間才能運行。我希望並行化將有助於使其更快，但不知道如何做到這一點。除了多重處理之外，還有其他方法可以讓它更快嗎？我將不勝感激任何幫助。

Answer 1

從你寫的文章和你發佈的代碼段的簡短瀏覽，似乎大多數繁重的工作似乎是通過hdp命令完成的。所以這就是你想要的parallelize。 你似乎在做的是打開一個子進程。 你也可以嘗試使用線程。你的代碼會是這樣的

#!/usr/bin/python 
import thread 
from subprocess import call 

def check_Corrupt_1(dirPath, logfile): 

    fileCheck = open(logfile, "w").close() 
    fileCheck = open(logfile, "w") 

    emptydir = [] 
    zero_size = {} 
    #entering the year to be checked (day number) 
    for fname in os.listdir(dirPath): 

     if(os.listdir(os.path.join(dirPath, fname)) == []): 
      emptydir.append(fname) 

     else: 

      #this makes sure that we do not enter an empty directory 
      if fname not in emptydir: 
       inPath = os.path.join(dirPath, fname) 

       for filename in os.listdir(inPath): 
        hdfinfo = os.stat(os.path.join(inPath, filename)) 

        if(hdfinfo.st_size == 0): 
         zero_size[filename] = True 

        else: 
         try:  
          thread.start_new_thread(call(["hdp", "dumpsds", "-h"])) 
         except: 
          print "Error generating thread" 

         if(int(callno.stdout.read()[0]) != 0): 
          fileCheck.write(os.path.join(inPath, filename) + '\n')