從pickle文件加載multiprocessing.manager.dictionary時出錯

Question

從pickle加載multiprocessing.manager.dictionary時出現錯誤。我確定pickle文件存在於我運行python的目錄中。

Python 2.7.6 (default, Jun 22 2015, 17:58:13) 
[GCC 4.8.2] on linux2 
Type "help", "copyright", "credits" or "license" for more information. 
>>> import pickle 
>>> import multiprocessing 
>>> a = open("test.pkl", "rb") 
>>> pickle.load(a) 
Traceback (most recent call last): 
    File "<stdin>", line 1, in <module> 
    File "/usr/lib/python2.7/pickle.py", line 1378, in load 
    return Unpickler(file).load() 
    File "/usr/lib/python2.7/pickle.py", line 858, in load 
    dispatch[key](self) 
    File "/usr/lib/python2.7/pickle.py", line 1133, in load_reduce 
    value = func(*args) 
    File "/usr/lib/python2.7/multiprocessing/managers.py", line 879, in RebuildProxy 
    return func(token, serializer, incref=incref, **kwds) 
    File "/usr/lib/python2.7/multiprocessing/managers.py", line 733, in __init__ 
    self._incref() 
    File "/usr/lib/python2.7/multiprocessing/managers.py", line 783, in _incref 
    conn = self._Client(self._token.address, authkey=self._authkey) 
    File "/usr/lib/python2.7/multiprocessing/connection.py", line 169, in Client 
    c = SocketClient(address) 
    File "/usr/lib/python2.7/multiprocessing/connection.py", line 304, in SocketClient 
    s.connect(address) 
    File "/usr/lib/python2.7/socket.py", line 224, in meth 
    return getattr(self._sock,name)(*args) 
socket.error: [Errno 2] No such file or directory 
>>> pickle.load(a) 
Traceback (most recent call last): 
    File "<stdin>", line 1, in <module> 
    File "/usr/lib/python2.7/pickle.py", line 1378, in load 
    return Unpickler(file).load() 
    File "/usr/lib/python2.7/pickle.py", line 858, in load 
    dispatch[key](self) 
    File "/usr/lib/python2.7/pickle.py", line 1165, in load_put 
    self.memo[self.readline()[:-1]] = self.stack[-1] 
IndexError: list index out of range 

下面是將代碼保存到泡菜中的代碼部分。

from multiprocessing import Manager, Pool 
import pickle 
from functools import partial 

def output(dic, s): 
    a = open(s + ".pkl", "wb") 
    pickle.dump(dic, a) 
    a.close() 

data_list = [1,2,3,4] 
pool = Pool(processes = 4) 
m = Manager() 
lock = m.Lock() 
dic1 = m.dict() 
func = partial(f, dic1) # f is a function that takes 3 arguments with the returned result stored in dic1 
pool.map(func, data_list) 
output(dic1, "test") 
pool.close() 
pool.join() 

如果我在輸出前打印字典，終端上的結果看起來很好。

Answer 1

Manager.dict()實際上並沒有返回一個字典，而是一個代理對象，由一個不同的進程管理的實際字典。當您在該對象上調用方法時，該調用被轉發到實際字典所在的管理器。

由於管理器和客戶端進程之間的通信是使用pickle協議進行的，所以這些對象是pickleable的，當unpickled再次爲生活在manager進程內的實際對象生成一個代理時，這就是爲什麼你會看到嘗試創建一個連接當你試圖取消字典。

所以，如果你要醃經理字典的內容，它之前被轉化爲實際字典：

... 
pickle.dump(dict(dic), a) 
...