并行运行不同的函数并按特定顺序获取返回值

Question

我想运行并行不同的函数（foo、bar和baz），然后按照调用顺序获取每个函数返回的值。

我尝试了类似下面的代码，但由于某种原因它在 Windows 上永远循环。是否可以使用例如 multiprocessing 来实现这一点？请注意，函数应该将相同的东西（即数据）作为参数。

注意：下面的代码适用于 Python 3，我想要 Python 2.7 的等效代码，因为它是我需要使用的。

from concurrent.futures import ProcessPoolExecutor
from operator import itemgetter

def foo(data):
    return "foo"

def bar(data):
    return "bar"

def baz(data):
    return "baz"

work = [foo, bar, baz]
data = [1,2,3,4,5]

results = []

with ProcessPoolExecutor(max_workers=4) as pool:
    for i, work_item in enumerate(work):
        future = pool.submit(work_item, data)
        def callback(ret):
            results.append((i, ret.result()))
        future.add_done_callback(callback)

results.sort(key=itemgetter(0))
print(results)

Answer 1

我想我稍微改变了你的回调方法并实现了你想要的。检查结果：

...
with ProcessPoolExecutor(max_workers=4) as pool:
    for i, work_item in enumerate(work):
        future = pool.submit(work_item, data)
        def callback(i):
            def wrapper(ret):
                print(ret, i)
                results.append((i, ret.result()))
            return wrapper
        future.add_done_callback(callback(i))
...

结果如下：

<Future at 0x1057e4d30 state=finished returned str> 0
<Future at 0x1057e4a58 state=finished returned str> 2
<Future at 0x105779240 state=finished returned str> 1
[(0, 'foo'), (1, 'bar'), (2, 'baz')]

我运行此代码在 mac os x 和 python3 上。希望对你有帮助。

对于 python 2.7，您可以尝试这样的操作：

from multiprocessing import Process, Manager

def foo(data, i, results):
    results[i] = "foo"

def bar(data, i, results):
    results[i] = "bar"

def baz(data, i, results):
    results[i] = "baz"

work = [foo, bar, baz]
data = [1,2,3,4,5]

processes = []
results = Manager().dict()
for i, w in enumerate(work):
    p = Process(target=w, args=(data, i, results))
    processes.append(p)
    p.start()

for p in processes:
    p.join()

print results

结果是：

{0: 'foo', 1: 'bar', 2: 'baz'}

How can I recover the return value of a function passed to multiprocessing.Process?

并行运行不同的函数并按特定顺序获取返回值

Runing different functions in parallel and get the returned values in specific order

python

python-multiprocessing