pickle.PicklingError: Can't pickle '_subprocess_handle' object: <_subprocess_han dle object at 0x00AAAAAAA>

Question

我正在尝试编写 test to compare methods for inter process communication in python，但 OSX 和 windows 给我的第一个样本带来了问题。它几乎是使用队列的手动示例。

我用 python 2.7.9 和 3.4.1 在 linux 上编写了所有代码。而且效果很好。

#!/usr/bin/env python
import multiprocessing as MP
...
self.mpq_main = MP.Queue()
self.mpq_net = MP.Queue()
self.mpt_net = MP.Process( target=self.start_t_net, args=(self.mpq_main, self.mpq_net) )
...
def start_t_net(self, qin, qout):
    self.NET = gcbtestnet.GCBTestNET(qin, qout);

然后在 osx 和 windows 上测试时出现错误。这个问题是针对 windows 个问题的：

C:\Documents and Settings\user1\Desktop>C:\Python27\python.exe gcbtest.py
Traceback (most recent call last):
  File "gcbtest.py", line 82, in <module>
    theclass = GCBTest()
  File "gcbtest.py", line 31, in __init__
    self.mpt_fs.start();
  File "C:\Python27\lib\multiprocessing\process.py", line 130, in start
    self._popen = Popen(self)
  File "C:\Python27\lib\multiprocessing\forking.py", line 277, in __init__
    dump(process_obj, to_child, HIGHEST_PROTOCOL)
  File "C:\Python27\lib\multiprocessing\forking.py", line 199, in dump
    ForkingPickler(file, protocol).dump(obj)
  File "C:\Python27\lib\pickle.py", line 224, in dump
    self.save(obj)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\multiprocessing\forking.py", line 67, in dispatcher
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 401, in save_reduce
    save(args)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 548, in save_tuple
    save(element)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 725, in save_inst
    save(stuff)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 331, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Python27\lib\pickle.py", line 419, in save_reduce
    save(state)
  File "C:\Python27\lib\pickle.py", line 286, in save
    f(self, obj) # Call unbound method with explicit self
  File "C:\Python27\lib\pickle.py", line 649, in save_dict
    self._batch_setitems(obj.iteritems())
  File "C:\Python27\lib\pickle.py", line 681, in _batch_setitems
    save(v)
  File "C:\Python27\lib\pickle.py", line 313, in save
    (t.__name__, obj))
pickle.PicklingError: Can't pickle '_subprocess_handle' object: <_subprocess_han
dle object at 0x00ABD190>
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "C:\Python27\lib\multiprocessing\forking.py", line 381, in main
    self = load(from_parent)
  File "C:\Python27\lib\pickle.py", line 1378, in load
    return Unpickler(file).load()
  File "C:\Python27\lib\pickle.py", line 858, in load
    dispatch[key](self)
  File "C:\Python27\lib\pickle.py", line 880, in load_eof
    raise EOFError
EOFError
UI check
UI check
UI check
...

如果您尝试将对象引用作为进程参数传递，通常会出现该错误，但我只传递 multiprocess.Queues 个实例。

Answer 1

我发现了我的（严重）错误。当我将进程从简单方法移动到对象时，我忘记将所述对象的初始化移动到 .Process() 调用并将其保存在 class 方法中。由于某种原因，它适用于 linux，但不适用于 windows。

这是修复的差异 https://github.com/gcb/python_multiprocess_test/commit/8565384e5cc6cadf959c335b5db0693e646f6777

-self.mpt_net = MP.Process( target=self.start_t_net, args=(self.mpq_main, self.mpq_net) )
+self.mpt_net = MP.Process( target=gcbtestnet.GCBTestNET, args=(self.mpq_main, self.mpq_net) )
...
-def start_t_net(self, qin, qout):
-    self.NET = gcbtestnet.GCBTestNET(qin, qout);

pickle.PicklingError: Can't pickle '_subprocess_handle' object: <_subprocess_han dle object at 0x00AAAAAAA>

pickle.PicklingError: Can't pickle '_subprocess_handle' object: <_subprocess_han dle object at 0x00AAAAAAA>

python

message-queue

pickle

multiprocessing