为进程指定特定 CPU - python 多处理

Question

我在多处理设置中使用 Redis 作为 producer/consumer 关系的队列。

我的问题是我的生产者让我的消费者超载然后窃取它 CPU。

我的问题是，我能否在此设置中将整个处理器分配给特定的 function/process（即：消费者）。

Answer 1

这不是 Python 开箱即用的东西。也有点 OS-specific。在 Linux 下查看此答案：

Answer 2

我刚刚在项目中遇到了类似的问题，每个项目都有 3 个 servers/dispatchers 运行个 CPU 核心，产生高度 CPU 密集型工人 - 在同时，它们被设计为使用特定的内核，它们运行发挥了最大的潜力。

我需要确保这样的工作进程永远不会在 CPU 核心运行 server/dispatcher 上生成（或使用 - 因为一些工作人员也使用多处理）这导致后来的操作出现问题。

我使用了 Ioannis Filippidis's 答案中的以下代码，它非常适合将任何进程限制到任何内核。我在这个例子中使用了多处理池，但是 child 中的代码在任何 multiprocessing.Process 中都有效。注意：它不适用于 macOS。

import multiprocessing as mp


def child(worker: int) -> None:
    import psutil
    import time

    p = psutil.Process()
    print(f"Child #{worker}: {p}, affinity {p.cpu_affinity()}", flush=True)
    time.sleep(1)
    p.cpu_affinity([worker])
    print(f"Child #{worker}: Set my affinity to {worker}, affinity now {p.cpu_affinity()}", flush=True)

    time.sleep(1 + 3 * worker)
    print(f"Child #{worker}: Starting CPU intensive task now for 4 seconds on {p.cpu_affinity()}...", flush=True)
    t_end = time.perf_counter() + 4
    while time.perf_counter() < t_end:
        pass
    print(f"Child #{worker}: Finished CPU intensive task on {p.cpu_affinity()}", flush=True)


def main() -> None:
    with mp.Pool() as pool:
        # noinspection PyProtectedMember
        workers: int = pool._processes
        print(f"Running pool with {workers} workers")

        for i in range(workers):
            pool.apply_async(child, (i,))

        # Wait for children to finnish
        pool.close()
        pool.join()

    pass


if __name__ == '__main__':
    main()

控制台输出：

Running pool with 16 workers
Child #0: psutil.Process(pid=16168, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #1: psutil.Process(pid=20864, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #2: psutil.Process(pid=15748, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #4: psutil.Process(pid=20600, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #3: psutil.Process(pid=17900, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #5: psutil.Process(pid=3288, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #7: psutil.Process(pid=19308, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #6: psutil.Process(pid=9768, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #8: psutil.Process(pid=1988, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #9: psutil.Process(pid=13960, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #11: psutil.Process(pid=3068, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #10: psutil.Process(pid=9636, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #12: psutil.Process(pid=18608, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #13: psutil.Process(pid=14356, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #14: psutil.Process(pid=14636, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #15: psutil.Process(pid=17372, name='python.exe', started='23:03:09'), affinity [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
Child #0: Set my affinity to 0, affinity now [0]
Child #1: Set my affinity to 1, affinity now [1]
Child #2: Set my affinity to 2, affinity now [2]
Child #4: Set my affinity to 4, affinity now [4]
Child #3: Set my affinity to 3, affinity now [3]
Child #5: Set my affinity to 5, affinity now [5]
Child #7: Set my affinity to 7, affinity now [7]
Child #6: Set my affinity to 6, affinity now [6]
Child #8: Set my affinity to 8, affinity now [8]
Child #9: Set my affinity to 9, affinity now [9]
Child #10: Set my affinity to 10, affinity now [10]
Child #11: Set my affinity to 11, affinity now [11]
Child #12: Set my affinity to 12, affinity now [12]
Child #13: Set my affinity to 13, affinity now [13]
Child #14: Set my affinity to 14, affinity now [14]
Child #15: Set my affinity to 15, affinity now [15]
Child #0: Starting CPU intensive task now for 4 seconds on [0]...
Child #1: Starting CPU intensive task now for 4 seconds on [1]...
Child #0: Finished CPU intensive task on [0]
Child #2: Starting CPU intensive task now for 4 seconds on [2]...
Child #1: Finished CPU intensive task on [1]
Child #3: Starting CPU intensive task now for 4 seconds on [3]...
Child #2: Finished CPU intensive task on [2]
Child #4: Starting CPU intensive task now for 4 seconds on [4]...
Child #3: Finished CPU intensive task on [3]
Child #5: Starting CPU intensive task now for 4 seconds on [5]...
Child #4: Finished CPU intensive task on [4]
Child #6: Starting CPU intensive task now for 4 seconds on [6]...
Child #5: Finished CPU intensive task on [5]
Child #7: Starting CPU intensive task now for 4 seconds on [7]...
Child #6: Finished CPU intensive task on [6]
Child #8: Starting CPU intensive task now for 4 seconds on [8]...
Child #7: Finished CPU intensive task on [7]
Child #9: Starting CPU intensive task now for 4 seconds on [9]...
Child #8: Finished CPU intensive task on [8]
Child #10: Starting CPU intensive task now for 4 seconds on [10]...
Child #9: Finished CPU intensive task on [9]
Child #11: Starting CPU intensive task now for 4 seconds on [11]...
Child #10: Finished CPU intensive task on [10]
Child #12: Starting CPU intensive task now for 4 seconds on [12]...
Child #11: Finished CPU intensive task on [11]
Child #13: Starting CPU intensive task now for 4 seconds on [13]...
Child #12: Finished CPU intensive task on [12]
Child #14: Starting CPU intensive task now for 4 seconds on [14]...
Child #13: Finished CPU intensive task on [13]
Child #15: Starting CPU intensive task now for 4 seconds on [15]...
Child #14: Finished CPU intensive task on [14]
Child #15: Finished CPU intensive task on [15]

Process finished with exit code 0

在任务管理器中查看：

为进程指定特定 CPU - python 多处理

Designate specific CPU for a process - python multiprocessing

python

cpu

multiprocessing