將IPython並行羣集對象傳遞到批處理執行的自定義類中

我是一位嘗試使用python進行科學編程的新手程序員。我認爲這些帖子（How to work with interactively-defined classes in IPython.parallel?和ipython parallel push custom object）涉及到類似的問題，但對我無用。我想運行我的代碼作爲腳本（PBS或SGE排隊調度程序），我不知道如何使用蒔蘿。將IPython並行羣集對象傳遞到批處理執行的自定義類中

本質上，我試圖使用Ipython並行羣集來分割自定義類方法中定義的計算。

我想將一個集羣對象傳遞到我的自定義類實例中，然後使用該集羣拆分對作爲成員定義的數據進行操作的計算。

已經開始使用ipcluster（/path/to/ipcontroller-client.json）
然後，我想運行python test_parallel.py
凡集羣，test_parallel.py是

class Foo(object): 
    def __init__(self): 
     from numpy import arange 
     self.data = arange(10)*10 

    def A(self, y): 
     print "in A:", y 
     self.data[y] 

    def parallelA(self, z, cl): 
     print "in parallelA:", cl[:].map_sync(self.A, z) 

    def serialA(self, z): 
     print "in serialA:", map(self.A, z) 

if __name__ == "__main__": 

    from IPython.parallel import Client 
    f = '/path/to/security/ipcontroller-client.json' 
    c = Client(f) 

    asdf = Foo() 
    asdf.serialA([1, 3, 5])  ## works 
    asdf.parallelA([1, 3, 5], c) ## doesn't work

的outpu t是

$ ~/Projects/parcellation$ python test_parallel.py 
in serialA: in A: 1 
in A: 3 
in A: 5 
[None, None, None] 
in parallelA: 
Traceback (most recent call last): 
    File "test_parallel.py", line 24, in <module> 
    asdf.parallelA([1, 3, 5], c) ## doesn't work 
    File "test_parallel.py", line 11, in parallelA 
    print "in parallelA:", cl[:].map_sync(self.A, z) 
    File "/usr/local/lib/python2.7/dist-packages/IPython/parallel/client/view.py", line 366, in map_sync 
    return self.map(f,*sequences,**kwargs) 
    File "<string>", line 2, in map 
    File "/usr/local/lib/python2.7/dist-packages/IPython/parallel/client/view.py", line 66, in sync_results 
    ret = f(self, *args, **kwargs) 
    File "/usr/local/lib/python2.7/dist-packages/IPython/parallel/client/view.py", line 624, in map 
    return pf.map(*sequences) 
    File "/usr/local/lib/python2.7/dist-packages/IPython/parallel/client/remotefunction.py", line 271, in map 
    ret = self(*sequences) 
    File "<string>", line 2, in __call__ 
    File "/usr/local/lib/python2.7/dist-packages/IPython/parallel/client/remotefunction.py", line 78, in sync_view_results 
    return f(self, *args, **kwargs) 
    File "/usr/local/lib/python2.7/dist-packages/IPython/parallel/client/remotefunction.py", line 243, in __call__ 
    ar = view.apply(f, *args) 
    File "/usr/local/lib/python2.7/dist-packages/IPython/parallel/client/view.py", line 233, in apply 
    return self._really_apply(f, args, kwargs) 
    File "<string>", line 2, in _really_apply 
    File "/usr/local/lib/python2.7/dist-packages/IPython/parallel/client/view.py", line 66, in sync_results 
    ret = f(self, *args, **kwargs) 
    File "<string>", line 2, in _really_apply 
    File "/usr/local/lib/python2.7/dist-packages/IPython/parallel/client/view.py", line 51, in save_ids 
    ret = f(self, *args, **kwargs) 
    File "/usr/local/lib/python2.7/dist-packages/IPython/parallel/client/view.py", line 567, in _really_apply 
    ident=ident) 
    File "/usr/local/lib/python2.7/dist-packages/IPython/parallel/client/client.py", line 1263, in send_apply_request 
    item_threshold=self.session.item_threshold, 
    File "/usr/local/lib/python2.7/dist-packages/IPython/kernel/zmq/serialize.py", line 145, in pack_apply_message 
    arg_bufs = flatten(serialize_object(arg, buffer_threshold, item_threshold) for arg in args) 
    File "/usr/local/lib/python2.7/dist-packages/IPython/utils/data.py", line 30, in flatten 
    return [x for subseq in seq for x in subseq] 
    File "/usr/local/lib/python2.7/dist-packages/IPython/kernel/zmq/serialize.py", line 145, in <genexpr> 
    arg_bufs = flatten(serialize_object(arg, buffer_threshold, item_threshold) for arg in args) 
    File "/usr/local/lib/python2.7/dist-packages/IPython/kernel/zmq/serialize.py", line 89, in serialize_object 
    buffers.insert(0, pickle.dumps(cobj, PICKLE_PROTOCOL)) 
cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

理解爲什麼這不起作用任何幫助，並修復，需要最少的代碼改變將是非常有益的。

謝謝！

來源

2014-10-10 Sang

我想出了一個解決方案：

class Foo(object): 
    def __init__(self): 
     from numpy import arange 
     self.data = arange(10)*10 

    @staticmethod 
    def A(data, y): 
     print "in A:", y ## doesn't produce an output 
     return data[y] 

    def parallelA(self, z, cl): 
     print "in parallelA:", cl[:].map_sync(self.A, [self.data]*len(z), z) 

if __name__ == "__main__": 

    from IPython.parallel import Client 
    f = '/path/to/security/ipcontroller-client.json' 
    c = Client(f) 

    asdf = Foo() 
    asdf.parallelA([1, 3, 5], c)

輸出當上述代碼運行：

$ python test_parallel.py 
in parallelA: [10, 30, 50]

來源

2014-10-10 04:09:49 Sang

將IPython並行羣集對象傳遞到批處理執行的自定義類中

回答

相關問題