运行具有 Python 2.7 / Windows 的多处理作业

Question

基于，我想运行这个 multiprocessing 工作 Python 2.7 / Windows:

def main():
    import itertools as it
    from multiprocessing import Pool

    def dothejob(i, j, k):
        print i, j, k

    the_args = it.product(range(100), range(100), range(100))
    pool = Pool(4)

    def jobWrapper(args): 
        return dothejob(*args)

    res = pool.map(jobWrapper, the_args)

if __name__ == '__main__':
    main()

main() 和最后两行是必要的，因为没有它们，就会出现众所周知的错误：

This probably means that you are on Windows and you have forgotten to use the proper idiom in the main module:
if __name__ == '__main__':
    ....

但即便如此，我还是收到了这个错误：

File "C:\Users\User\Desktop\test.py", line 14, in main res = pool.map(jobWrapper, the_args) File "C:\Python27\lib\multiprocessing\pool.py", line 251, in map return self.map_async(func, iterable, chunksize).get() File "C:\Python27\lib\multiprocessing\pool.py", line 558, in get raise self._value cPickle.PicklingError: Can't pickle : attribute lookup >builtin.function failed

这里哪里涉及到cPickle，为什么会报错/如何解决？

Answer 1

所有定义必须在模块范围内：

import itertools as it
from multiprocessing import Pool, freeze_support

def dothejob(i, j, k):
    print i, j, k

def jobWrapper(args): 
    return dothejob(*args)

def main():
    the_args = it.product(range(100), range(100), range(100))
    pool = Pool(4)
    res = pool.map(jobWrapper, the_args)

if __name__ == '__main__':
    freeze_support() #you need this in windows
    main()

您还需要 freeze_support 致电 windows

运行具有 Python 2.7 / Windows 的多处理作业

Run a multiprocessing job with Python 2.7 / Windows

python

windows

multiprocessing

python-multiprocessing

运行 具有 Python 2.7 / Windows 的多处理作业

Run a multiprocessing job with Python 2.7 / Windows

python

windows

multiprocessing

python-multiprocessing

运行具有 Python 2.7 / Windows 的多处理作业