函數初始化和對象初始化（多處理）

Question

我最近看到一個關於如何functions are objects in python的答案/評論。所以，我想知道爲什麼當我採用這個例子，並且在初始化一個變量時創建一個類時，它不會以相同的方式工作。 （該類示例收到酸洗錯誤）： PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

有沒有人知道這是爲什麼？從鏈路

示例代碼：

import multiprocessing as mp 

def f(x): 
    f.q.put('Doing: ' + str(x)) 
    return x*x 

def f_init(q): 
    f.q = q 

def main(): 
    jobs = range(1,6) 

    q = mp.Queue() 
    p = mp.Pool(None, f_init, [q]) 
    results = p.imap(f, jobs) 
    p.close() 

    for i in range(len(jobs)): 
     print q.get() 
     print results.next() 

if __name__ == '__main__': 
    main() 

相同示例而puttin f成類：

import multiprocessing as mp 

class F: 
    def __init__(self, q): 
     self.q = q 
    def f(x): 
     self.q.put('Doing: ' + str(x)) 
     return x*x 

def main(): 
    jobs = range(1,6) 

    q = mp.Queue() 
    p = mp.Pool(None) 
    f = F(q) 
    results = p.imap(f.f, jobs) 
    p.close() 

    for i in range(len(jobs)): 
     print q.get() 
     print results.next() 

if __name__ == '__main__': 
    main() 

Answer 1

實例方法不會自動picklable。因此

p.imap(f.f, jobs) 

失敗，因爲p.imap試圖醃製參數。 有「教」泡菜how to pickle instance methods（見史蒂芬Bethard的答案）， 的方式，但你的代碼有一個問題：傳遞的隊列實例導致拋出一個RuntimeError：

RuntimeError: Queue objects should only be shared between processes through inheritance 

該錯誤消息是一點點因爲你可以通過隊列作爲參數p.imap，但你不能先將它傳遞給類F，然後通過f.f將它傳遞給工作進程，這讓我感到困惑（至少對我來說）。

無論如何，由於這些問題，我建議堅持使用原始代碼而不是試圖將代碼包裝到類中。

---

下面是如何來醃製實例方法的例子：

import multiprocessing as mp 
import copy_reg 
import types 

def _pickle_method(method): 
    # Author: Steven Bethard (author of argparse) 
    # http://bytes.com/topic/python/answers/552476-why-cant-you-pickle- 
    # instancemethods 
    func_name = method.im_func.__name__ 
    obj = method.im_self 
    cls = method.im_class 
    cls_name = '' 
    if func_name.startswith('__') and not func_name.endswith('__'): 
     cls_name = cls.__name__.lstrip('_') 
    if cls_name: 
     func_name = '_' + cls_name + func_name 
    return _unpickle_method, (func_name, obj, cls) 


def _unpickle_method(func_name, obj, cls): 
    # Author: Steven Bethard 
    # http://bytes.com/topic/python/answers/552476-why-cant-you-pickle- 
    # instancemethods 
    for cls in cls.mro(): 
     try: 
      func = cls.__dict__[func_name] 
     except KeyError: 
      pass 
     else: 
      break 
    return func.__get__(obj, cls) 

# This call to copy_reg.pickle allows you to pass methods as the first arg to 
# mp.Pool methods. If you comment out this line, `pool.map(self.foo, ...)` results in 
# PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup 
# __builtin__.instancemethod failed 

copy_reg.pickle(types.MethodType, _pickle_method, _unpickle_method) 

class F(object): 
    def f(self, x): 
     fq.put('Doing: ' + str(x))   
     return x*x 

def f_init(q): 
    # http://stackoverflow.com/a/3843313/190597 (Olson) 
    global fq 
    fq = q 

def main(): 
    jobs = range(1,6) 
    q = mp.Queue() 
    p = mp.Pool(None, f_init, [q]) 
    f = F() 
    results = p.imap(f.f, jobs) 
    p.close() 

    for r in results: 
     print(r, q.get()) 

if __name__ == '__main__': 
    main() 

產生

(1, 'Doing: 2') 
(4, 'Doing: 3') 
(9, 'Doing: 4') 
(16, 'Doing: 1') 
(25, 'Doing: 5')