多进程挂起

Question

所以我开始尝试使用 Python 的多处理库。我的目标是加快一个慢速函数的速度，该函数将字符串与其他字符串的大型数据库进行比较并返回最相似的匹配项。为此，我尝试编写一个函数，将任务拆分到不同的 Process 对象中，并将它们设置为运行ning，使用共享变量来捕获结果：

cores = cpu_count() # Number of cores in this computer, i.e. 4
sublistList = chunks(tasks,cores) # Split tasks into subprocessing arrays of evenly-sized chunks, the number of chunks equal to how many cores we have to process them

# Create a multiprocessing function, since this is a large function that will take time and splitting it across cores will ease the load
if __name__ == '__main__':

    freeze_support() # Make sure multiple applications don't spawn, this is necessary for Windows

    jobs = [] # Array of processes
    manager = Manager() # Create a manager
    returns = manager.list() # Shared list variable we use to get return results

    for i in range(0,cores): # For number of cores...

        p = Process(target=workerFunction,args=(w,sublistList[i],returns))
        jobs.append(p) # Add to array of processes to run
        p.start()

    for p in jobs:

        p.join()

然而，当我运行这段代码时，它会创建一个新的应用程序 window 然后无限期地挂起，这是完全奇怪的行为，根本不是我想要的。是什么导致我的代码出现这种情况？我的 worker 函数是否悄悄崩溃并且没有提醒我？我查看了各种其他答案，但 none 的建议答案似乎解决了这个问题。

（如果与问题相关，我是一名入门级软件工程师，具有几年其他语言的编程经验，但对 Python 比较陌生。这是针对小型独立游戏的我的副项目。）

Answer 1

这还不是答案，但我发布它是为了向您展示一个 runnable Minimal, Complete, and Verifiable Example.

的示例

该代码基于您当前问题中的内容，以及使其可运行所缺少的所有其他内容。毫不奇怪，由于所有这些都只是猜测，它不会重现您所说的问题 — 但这可能是由于我的一个或多个猜测在一些重要的方面......这就是为什么你真的应该是提供所有代码的人。

一个观察：最后的 p.join() 调用将使主进程等待每个子进程完成。这将导致主进程在等待每个进程时出现 "hang"。

from multiprocessing import *
from time import sleep

tasks = None

def chunks(tasks, cores):
    return [[i for _ in range(8)] for i in range(cores)]

def workerFunction(w, sublist, returns):
    print('starting workerFunction:', w)
    result = [value+100 for value in sublist]
    returns.append(result)
    sleep(3)
    print('exiting workerFunction:', w)

if __name__ == '__main__':

    # Only do in main process.
    freeze_support()
    cores = cpu_count()
    sublistList = chunks(tasks, cores)
    manager = Manager()
    returns = manager.list()
    jobs = []

    for i in range(cores):
        w = i
        p = Process(target=workerFunction, args=(w, sublistList[i], returns))
        jobs.append(p)
        p.start()

    for i, p in enumerate(jobs, 1):
        print('joining job[{}]'.format(i))
        p.join()

    # Display results.
    for sublist in returns:
        print(sublist)

    print('done')

输出：

joining job[1]
starting workerFunction: 2
starting workerFunction: 1
starting workerFunction: 0
starting workerFunction: 5
starting workerFunction: 7
starting workerFunction: 3
starting workerFunction: 4
starting workerFunction: 6
exiting workerFunction: 2
exiting workerFunction: 0
exiting workerFunction: 1
joining job[2]
exiting workerFunction: 5
joining job[3]
joining job[4]
exiting workerFunction: 7
exiting workerFunction: 3
exiting workerFunction: 4
joining job[5]
exiting workerFunction: 6
joining job[6]
joining job[7]
joining job[8]
[102, 102, 102, 102, 102, 102, 102, 102]
[101, 101, 101, 101, 101, 101, 101, 101]
[100, 100, 100, 100, 100, 100, 100, 100]
[105, 105, 105, 105, 105, 105, 105, 105]
[107, 107, 107, 107, 107, 107, 107, 107]
[103, 103, 103, 103, 103, 103, 103, 103]
[104, 104, 104, 104, 104, 104, 104, 104]
[106, 106, 106, 106, 106, 106, 106, 106]
done
Press any key to continue . . .

多进程挂起

Multiprocessing process(es) hanging

python

python-multiprocessing