使用 asyncio.create_subprocess_exec 设置最大并发数

Question

我需要运行一个程序大约 500 次不同的输入。我想使用 asyncio.create_subprocess_exec 并希望同时限制进程数运行ning 以免阻塞机器。有没有办法设置并发级别？例如，我期望 AbstractEventLoop.set_max_tasks.

Answer 1

作为 by @AndrewSvetlov, you can use an asyncio.Semaphore强制执行限制：

async def run_program(input):
    p = await asyncio.create_subprocess_exec(...)
    # ... communicate with the process ...
    p.terminate()
    return something_useful

async def run_throttled(input, sem):
    async with sem:
        result = await run_program(input)
    return result

LIMIT = 10

async def many_programs(inputs):
    sem = asyncio.Semaphore(LIMIT)
    results = await asyncio.gather(
        *[run_throttled(input, sem) for input in inputs])
    # ...

使用 asyncio.create_subprocess_exec 设置最大并发数

Set max concurrency with asyncio.create_subprocess_exec

python

concurrency

subprocess

python-3.x

python-asyncio