从 pool.map 个进程返回多个列表？

Question

Win 7，x64，Python2.7.12

在下面的代码中，我将启动一些池进程以通过 multiprocessing.Pool.map() 方法进行简单的乘法运算。输出数据收集在 List_1.

注意：这是我实际代码的精简版。实际应用中涉及多个列表，都很大

import multiprocessing
import numpy as np

def createLists(branches):

    firstList = branches[:] * node

    return firstList


def init_process(lNodes):

    global node
    node = lNodes
    print 'Starting', multiprocessing.current_process().name


if __name__ == '__main__':

    mgr = multiprocessing.Manager()
    nodes = mgr.list()
    pool_size = multiprocessing.cpu_count()

    branches = [i for i in range(1, 21)]
    lNodes = 10
    splitBranches = np.array_split(branches, int(len(branches)/pool_size))

    pool = multiprocessing.Pool(processes=pool_size, initializer=init_process, initargs=[lNodes])
    myList_1 = pool.map(createLists, splitBranches)

    pool.close() 
    pool.join()

我现在向 createLists() 添加一个额外的计算并尝试传回两个列表。

import multiprocessing
import numpy as np

def createLists(branches):

    firstList = branches[:] * node
    secondList = branches[:] * node * 2

    return firstList, secondList


def init_process(lNodes):
    global node
    node = lNodes
    print 'Starting', multiprocessing.current_process().name


if __name__ == '__main__':

    mgr = multiprocessing.Manager()
    nodes = mgr.list()
    pool_size = multiprocessing.cpu_count()

    branches = [i for i in range(1, 21)]
    lNodes = 10
    splitBranches = np.array_split(branches, int(len(branches)/pool_size))

    pool = multiprocessing.Pool(processes=pool_size, initializer=init_process, initargs=[lNodes])
    myList_1, myList_2 = pool.map(createLists, splitBranches)

    pool.close() 
    pool.join()

这引发了跟随错误和回溯..

Traceback (most recent call last):

  File "<ipython-input-6-ff188034c708>", line 1, in <module>
    runfile('C:/Users/nr16508/Local Documents/Inter Trab Angle/Parallel/scratchpad.py', wdir='C:/Users/nr16508/Local Documents/Inter Trab Angle/Parallel')

  File "C:\Users\nr16508\AppData\Local\Continuum\Anaconda2\lib\site-packages\spyder\utils\site\sitecustomize.py", line 866, in runfile
    execfile(filename, namespace)

  File "C:\Users\nr16508\AppData\Local\Continuum\Anaconda2\lib\site-packages\spyder\utils\site\sitecustomize.py", line 87, in execfile
    exec(compile(scripttext, filename, 'exec'), glob, loc)

  File "C:/Users/nr16508/Local Documents/Inter Trab Angle/Parallel/scratchpad.py", line 36, in <module>
    myList_1, myList_2 = pool.map(createLists, splitBranches)

ValueError: too many values to unpack

当我试图将两个列表合二为一以传回时，即...

return [firstList, secondList]
......
myList = pool.map(createLists, splitBranches)

...输出变得过于混乱，无法进一步处理。

有没有一种方法可以从池化进程中收集多个列表？

Answer 1

这个问题与多处理或线程池无关。它只是关于如何解压缩列表，这可以用标准的 zip(*...) 习惯用法来完成。

myList_1, myList_2 = zip(*pool.map(createLists, splitBranches))

从 pool.map 个进程返回多个列表？

Returning multiple lists from pool.map processes?

python

threadpool

python-multiprocessing