將多進程池內循環（進程間共享內存）

Question

我正在使用Python軟件包「deap」來解決遺傳算法的一些多目標優化問題。這些功能可能非常昂貴，並且由於GA的進化性質，它的複雜性非常快。現在這個包確實有一些支持，以允許演化計算與多進程並行化。

但是，我想更進一步並多次運行優化，並在一些優化參數上使用不同的值。例如，我可能想用權值的不同值來解決優化問題。

這似乎是一個非常自然的循環案例，但問題是這些參數必須在程序的全局範圍內定義（即在「main」函數之上），以便所有子進程都知道參數。下面是一些僞代碼：

# define deap parameters - have to be in the global scope 
toolbox = base.Toolbox() 
history = tools.History() 
weights = [1, 1, -1] # This is primarily what I want to vary 
creator.create("Fitness",base.Fitness, weights=weights) 
creator.create("Individual", np.ndarray, fitness=creator.Fitness) 

def main(): 
    # run GA to solve multiobjective optimization problem 
    return my_optimized_values 

if __name__=='__main__': 
    ## What I'd like to do but can't ## 
    ## all_weights = list(itertools.product([1, -1],repeat=3)) 
    ## for combo in all_weights: 
    ##  weights = combo 
    ## 
    pool = multiprocessing.Pool(processes=6) 
    # This can be down here, and it distributes the GA computations to a pool of workers 
    toolbox.register("map",pool.map) 
    my_values = main() 

我已經研究了各種可能，如multiprocessing.Value，多的悲愴叉，和其他人，但最終總會有與子進程讀取個人類問題。

我在deap用戶組中提出了這個問題，但它不像SO那麼大。另外，在我看來，這更像是一個概念性的Python問題，而不是一個特定的問題。我目前解決這個問題的方法是多次運行代碼，每次更改一些參數定義。至少在這種情況下，GA計算仍然是並行的，但它確實需要比我想要的更多的手動干預。

任何意見或建議，非常感謝！

Answer 1

使用initializer/initargs關鍵字參數爲Pool可爲每次運行需要更改的全局變量傳遞不同的值。 initializer函數將在其啓動後立即調用initargs作爲其Pool中每個工作進程的參數。您可以將您的全局變量設置爲所需的值，並且在池的整個生命週期內，它們將在每個子內部正確設置。

你需要爲每個運行不同Pool，但不應該是一個問題：

toolbox = base.Toolbox() 
history = tools.History() 
weights = None # We'll set this in the children later. 



def init(_weights): 
    # This will run in each child process. 
    global weights 
    weights = _weights 
    creator.create("Fitness",base.Fitness, weights=weights) 
    creator.create("Individual", np.ndarray, fitness=creator.Fitness) 


if __name__=='__main__': 
    all_weights = list(itertools.product([1, -1],repeat=3)) 
    for combo in all_weights: 
     weights = combo 
     pool = multiprocessing.Pool(processes=6, initializer=init, initargs=(weights,)) 
     toolbox.register("map",pool.map) 
     my_values = main() 
     pool.close() 
     pool.join() 

Answer 2

我也一直不舒服DEAP的利用全球範圍內的，而且我覺得我有爲您提供替代解決方案。

可以在每個循環迭代中導入每個模塊的不同版本，從而避免對全局範圍的依賴。

this_random = importlib.import_module("random") 
this_creator = importlib.import_module("deap.creator") 
this_algorithms = importlib.import_module("deap.algorithms") 
this_base = importlib.import_module("deap.base") 
this_tools = importlib.import_module("deap.tools") 

據我所知，這似乎與多處理有關。

作爲一個例子，下面是DEAP的onemax_mp.py版本，它避免了將全部DEAP文件放在全局範圍內。我在__main__中包含了一個循環，它改變了每次迭代的權重。 （第一次使用的次數最大，第二次最少。）一切正常，多處理。

#!/usr/bin/env python2.7 
# This file is part of DEAP. 
# 
# DEAP is free software: you can redistribute it and/or modify 
# it under the terms of the GNU Lesser General Public License as 
# published by the Free Software Foundation, either version 3 of 
# the License, or (at your option) any later version. 
# 
# DEAP is distributed in the hope that it will be useful, 
# but WITHOUT ANY WARRANTY; without even the implied warranty of 
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the 
# GNU Lesser General Public License for more details. 
# 
# You should have received a copy of the GNU Lesser General Public 
# License along with DEAP. If not, see <http://www.gnu.org/licenses/>. 

import array 
import multiprocessing 
import sys 

if sys.version_info < (2, 7): 
    print("mpga_onemax example requires Python >= 2.7.") 
    exit(1) 

import numpy 
import importlib 


def evalOneMax(individual): 
    return sum(individual), 


def do_onemax_mp(weights, random_seed=None): 
    """ Run the onemax problem with the given weights and random seed. """ 

    # create local copies of each module 
    this_random = importlib.import_module("random") 
    this_creator = importlib.import_module("deap.creator") 
    this_algorithms = importlib.import_module("deap.algorithms") 
    this_base = importlib.import_module("deap.base") 
    this_tools = importlib.import_module("deap.tools") 

    # hoisted from global scope 
    this_creator.create("FitnessMax", this_base.Fitness, weights=weights) 
    this_creator.create("Individual", array.array, typecode='b', 
         fitness=this_creator.FitnessMax) 
    this_toolbox = this_base.Toolbox() 
    this_toolbox.register("attr_bool", this_random.randint, 0, 1) 
    this_toolbox.register("individual", this_tools.initRepeat, 
          this_creator.Individual, this_toolbox.attr_bool, 100) 
    this_toolbox.register("population", this_tools.initRepeat, list, 
          this_toolbox.individual) 
    this_toolbox.register("evaluate", evalOneMax) 
    this_toolbox.register("mate", this_tools.cxTwoPoint) 
    this_toolbox.register("mutate", this_tools.mutFlipBit, indpb=0.05) 
    this_toolbox.register("select", this_tools.selTournament, tournsize=3) 

    # hoisted from __main__ 
    this_random.seed(random_seed) 
    pool = multiprocessing.Pool(processes=4) 
    this_toolbox.register("map", pool.map) 
    pop = this_toolbox.population(n=300) 
    hof = this_tools.HallOfFame(1) 
    this_stats = this_tools.Statistics(lambda ind: ind.fitness.values) 
    this_stats.register("avg", numpy.mean) 
    this_stats.register("std", numpy.std) 
    this_stats.register("min", numpy.min) 
    this_stats.register("max", numpy.max) 

    this_algorithms.eaSimple(pop, this_toolbox, cxpb=0.5, mutpb=0.2, ngen=40, 
          stats=this_stats, halloffame=hof) 

    pool.close() 

if __name__ == "__main__": 
    for tgt_weights in ((1.0,), (-1.0,)): 
     do_onemax_mp(tgt_weights)