cPickle - 忽略它無法序列化的東西，而不是引發異常

Question

我使用cPickle來序列化用於記錄的數據。

我希望能夠將任何我想要的東西扔進一個對象，然後序列化它。通常這與cPickle一致，但遇到了一個問題，我想要序列化的某個對象包含一個函數。這導致cPickle引發異常。

我寧願cPickle只是跳過它無法處理的東西，而不是導致整個過程崩潰。

什麼是使這種情況發生的好方法？

Answer 1

我假設你正在尋找一個盡力而爲的解決方案，你沒關係如果unpickled結果不能正常工作。

對於您的特定用例，您可能需要register a pickle handler作爲函數對象。只要讓它成爲一個虛擬處理程序，足以滿足您的盡力而爲。爲函數創建一個處理程序是可能的，這相當棘手。爲避免影響其他代碼，您可能需要在退出日誌代碼時註銷該處理程序。

下面是一個例子（沒有任何註銷）：

import cPickle 
import copy_reg 
from types import FunctionType 

# data to pickle: note that o['x'] is a lambda and they 
# aren't natively picklable (at this time) 
o = {'x': lambda x: x, 'y': 1} 

# shows that o is not natively picklable (because of 
# o['x']) 
try: 
    cPickle.dumps(o) 
except TypeError: 
    print "not natively picklable" 
else: 
    print "was pickled natively" 

# create a mechanisms to turn unpickable functions int 
# stub objects (the string "STUB" in this case) 
def stub_pickler(obj): 
    return stub_unpickler,() 
def stub_unpickler(): 
    return "STUB" 
copy_reg.pickle(
    FunctionType, 
    stub_pickler, stub_unpickler) 

# shows that o is now picklable but o['x'] is restored 
# to the stub object instead of its original lambda 
print cPickle.loads(cPickle.dumps(o)) 

它打印：

not natively picklable 
{'y': 1, 'x': 'STUB'} 

Answer 2

爲什麼不只是陷入異常？

try: 
    cPickle.dumps(obj) 
except cPickle.PicklingError: 
    pass 

你可以做到這一點，以保持一切，太...

>>> def safe_pickle(L): 
...  result = [] 
...  for target in L: 
...    try: 
...      result.append(cPickle.dumps(target)) 
...    except (cPickle.PicklingError, TypeError): 
...      result.append(None) 
...  return result 
... 
>>> safe_pickle(["A",open('file.txt')]) 
["S'A'\n.", None] 

捕獲的異常不上升。

Answer 3

或者，嘗試cloudpickle：

>>> import cloudpickle 
>>> squared = lambda x: x ** 2 
>>> pickled_lambda = cloudpickle.dumps(squared) 

>>> import pickle 
>>> new_squared = pickle.loads(pickled_lambda) 
>>> new_squared(2) 
4 

pip install cloudpickle並實現你的夢想。與dask，IPython平行以及PySpark一樣的夢想也一樣。