IPython 与地图平行参考 VS 适用

Question

当使用 map 和 dict 之类的参数时，我无法获得并行引用。

from IPython import parallel
from IPython.parallel import Client
from IPython.parallel import Reference

rc = Client()
dview = rc.direct_view()
dview.block = True

dview['l'] = [0, 1]
kws = [{'l': Reference('l')}]
def second(kws):
    l = kws['l']
    return l[1]

dview.apply(second, kws[0])

Returns:

[1, 1, 1, 1, 1, 1, 1, 1]

但是：

dview.map(second, kws)

Returns:

TypeError: 'Reference' object does not support indexing

Answer 1

问题是反序列化不能很好地处理嵌套结构。你可以这样亲眼看看：

from IPython.core.display import display
from IPython.kernel.zmq.serialize import serialize_object, unserialize_object
from IPython.parallel import Reference
l = 5
ref = Reference('l')
kws = [dict(l=ref)]
display(unserialize_object(serialize_object(kws), g=globals()))
# outputs ([{'l': <Reference: 'l'>}], [])
display(unserialize_object(serialize_object(kws[0]), g=globals()))
# outputs ({'l': 5}, [])

我不是 IPython 开发人员，但查看代码后我发现了一种非常简单的修复方法。在ipykernel/pickleutil.py（或IPython/utils/pickleutil.py）中，有一个将类型映射到反序列化方法的字典：

uncan_map = {
    CannedObject : lambda obj, g: obj.get_object(g),
}

只需添加 dict 即可解决问题：

uncan_map = {
    CannedObject : lambda obj, g: obj.get_object(g),
    dict : uncan_dict,
}

我将对引用此问题的 ipykernel 项目提出拉取请求。

IPython 与地图平行参考 VS 适用

IPython parallel reference with map VS apply

parallel-processing

reference

ipython