如何使用多处理并行 Theano 函数？

Question

我有一个函数 return 由 theano.function() 编辑，我想在多处理中使用它来加速。以下是一个简化的演示脚本，用于显示我运行遇到问题的地方：

import numpy as np
from multiprocessing import Pool
from functools import partial
import theano
from theano import tensor

def get_theano_func():
    x = tensor.dscalar()
    y = x + 0.1
    f = theano.function([x], [y])
    return f

def func1(func, x):
    return func(x)

def MPjob(xlist):
    f = get_theano_func()
    fp = partial(func1, func=f)
    pool = Pool(processes=5)
    Results = pool.imap(fp, xlist)
    Y = []
    for y in Results:
        Y.append(y[0])
    pool.close()
    return Y

if __name__ == '__main__':
    xlist = np.arange(0, 5, 1)
    Y = MPjob(xlist)
    print(Y)

在上面的代码中，theano 函数 'f' 作为输入参数提供给 'func1()'。如果 MPjob() 运行s 正确，它应该 return [0.1, 1.1, 2.1, 3.1, 4.1]。但是，引发了异常 "TypeError: func1() got multiple values for argument 'func'"。

完整引用日志如下：

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "C:\Python35\lib\multiprocessing\pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
TypeError: func1() got multiple values for argument 'func'
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "F:/DaweiLeng/Code/workspace/Python/General/theano_multiprocess_debug.py", line 36, in <module>
    Y = MPjob(xlist)
  File "F:/DaweiLeng/Code/workspace/Python/General/theano_multiprocess_debug.py", line 29, in MPjob
    for y in Results:
  File "C:\Python35\lib\multiprocessing\pool.py", line 695, in next
    raise value
TypeError: func1() got multiple values for argument 'func'

有人得到提示吗？

Answer 1

原来是和partial()函数有关。完整的解释在这里 https://github.com/Theano/Theano/issues/4720#issuecomment-232029702

如何使用多处理并行 Theano 函数？

How to parallel a Theano function using multiprocessing?

python

multiprocessing

theano