multiprocessing.Pool.map（）刪除子類化的ndarray的屬性

Question

當從子類numpy.ndarray的子類的實例列表中使用map()從multiprocessing.Pool()時，自己類的新屬性被刪除。

下面的基礎上，numpy docs subclassing example小例子，再現了問題：

from multiprocessing import Pool 
import numpy as np 


class MyArray(np.ndarray): 

    def __new__(cls, input_array, info=None): 
     obj = np.asarray(input_array).view(cls) 
     obj.info = info 
     return obj 

    def __array_finalize__(self, obj): 
     if obj is None: return 
     self.info = getattr(obj, 'info', None) 

def sum_worker(x): 
    return sum(x) , x.info 

if __name__ == '__main__': 
    arr_list = [MyArray(np.random.rand(3), info=f'foo_{i}') for i in range(10)] 
    with Pool() as p: 
     p.map(sum_worker, arr_list) 

屬性info被丟棄

AttributeError: 'MyArray' object has no attribute 'info' 

使用內置map()工作正常

arr_list = [MyArray(np.random.rand(3), info=f'foo_{i}') for i in range(10)] 
list(map(sum_worker, arr_list2)) 

目的該方法的__array_finalize__()是對象保持屬性切片

arr = MyArray([1,2,3], info='foo') 
subarr = arr[:2] 
print(subarr.info) 

但對於Pool.map()此方法以某種方式不工作...後

Answer 1

由於多使用pickle序列化的數據向/從單獨的進程，這基本上是this question的重複。

適應從這個問題接受的解決方案，你的例子就變成：

from multiprocessing import Pool 
import numpy as np 

class MyArray(np.ndarray): 

    def __new__(cls, input_array, info=None): 
     obj = np.asarray(input_array).view(cls) 
     obj.info = info 
     return obj 

    def __array_finalize__(self, obj): 
     if obj is None: return 
     self.info = getattr(obj, 'info', None) 

    def __reduce__(self): 
     pickled_state = super(MyArray, self).__reduce__() 
     new_state = pickled_state[2] + (self.info,) 
     return (pickled_state[0], pickled_state[1], new_state) 

    def __setstate__(self, state): 
     self.info = state[-1] 
     super(MyArray, self).__setstate__(state[0:-1]) 

def sum_worker(x): 
    return sum(x) , x.info 

if __name__ == '__main__': 
    arr_list = [MyArray(np.random.rand(3), info=f'foo_{i}') for i in range(10)] 
    with Pool() as p: 
     p.map(sum_worker, arr_list) 

注意，第二個答案表明自悲愴使用dill而不是pickle您可能能夠使用pathos.multiprocessing與不適應原代碼。但是，當我測試它時，這不起作用。