Python 2.7：如何弥补缺失pool.starmap？

Question

我定义了这个函数

def writeonfiles(a,seed):
    random.seed(seed)

    f = open(a, "w+")
    for i in range(0,10):
        j = random.randint(0,10)
        #print j
        f.write(j)
    f.close()

其中 a 是包含文件路径的字符串，seed 是整数种子。我想以这样一种方式并行化一个简单的程序，即每个核心采用我提供的可用路径之一，为其随机生成器播种并在该文件上写入一些随机数，例如，如果我通过矢量

vector = [Test/file1.txt, Test/file2.txt]

和种子

seeds = (123412, 989898),

它为第一个可用核心提供了函数

writeonfiles(Test/file1.txt, 123412)

第二个是相同的函数，但参数不同：

writeonfiles(Test/file2.txt, 989898)

我在 Whosebug 上查看了很多类似的问题，但我无法找到任何解决方案。我试过的是：

def writeonfiles_unpack(args):
    return writeonfiles(*args)
if __name__ == "__main__":
     folder = ["Test/%d.csv" %i for i in range(0,4)]
     seed = [234124, 663123, 12345 ,123833]
     p = multiprocessing.Pool()
     p.map(writeonfiles, (folder,seed))

并给我 TypeError: writeonfiles() takes exactly 2 arguments (1 given).

我也试过了

if __name__ == "__main__":
    folder = ["Test/%d.csv" %i for i in range(0,4)]
    seed = [234124, 663123, 12345 ,123833]
    p = multiprocessing.Process(target=writeonfiles, args= [folder,seed])
    p.start()

但它给了我
文件“/usr/lib/python2.7/random.py”，第 120 行，在种子中超级（随机，自我）.seed（a） TypeError：无法散列的类型：'list'

最后，我尝试了上下文管理器

 @contextmanager
 def poolcontext(*args, **kwargs):
     pool = multiprocessing.Pool(*args, **kwargs)
     yield pool
     pool.terminate()

if __name__ == "__main__":
    folder = ["Test/%d" %i for i in range(0,4)]
    seed = [234124, 663123, 12345 ,123833]
    a = zip(folder, seed)
    with poolcontext(processes = 3) as pool:
    results = pool.map(writeonfiles_unpack,a )

结果是文件“/usr/lib/python2.7/multiprocessing/pool.py”，第 572 行，在 get 提高 self._value

TypeError: 'module' 对象不可调用

Answer 1

Python 2.7 缺少来自 Python 3.3+ 的 starmap 池方法。您可以通过使用包装器装饰目标函数来克服这个问题，包装器解压参数元组并调用目标函数：

import os
from multiprocessing import Pool
import random
from functools import wraps


def unpack(func):
    @wraps(func)
    def wrapper(arg_tuple):
        return func(*arg_tuple)
    return wrapper

@unpack
def write_on_files(a, seed):
    random.seed(seed)
    print("%d opening file %s" % (os.getpid(), a))  # simulate
    for _ in range(10):
        j = random.randint(0, 10)
       print("%d writing %d to file %s" % (os.getpid(), j, a))  # simulate


if __name__ == '__main__':

    folder = ["Test/%d.csv" % i for i in range(0, 4)]
    seed = [234124, 663123, 12345, 123833]

    arguments = zip(folder, seed)

    pool = Pool(4)
    pool.map(write_on_files, iterable=arguments)
    pool.close()
    pool.join()

Python 2.7：如何弥补缺失pool.starmap？

Python 2.7: How to compensate for missing pool.starmap?

python

arguments

multiprocessing

python-2.7

python-multiprocessing