Asyncio 和 multiprocessing.Process- 如何传递协程？

Question

我有一个协程正在抓取一些数据。我正在尝试添加多处理以加快速度。我想到了使用 multiprocesesing.Process。我的协程叫做 fetch_students_pages

我想做的是将这个协程作为目标传递给进程 class:

    def fetch_students_concurrently(self):
        for _ in range(10):
            Process(target=self.fetch_students_pages).start()

但如果失败：

/usr/lib/python3.7/multiprocessing/process.py:99: RuntimeWarning: coroutine 'StudentPageParser.fetch_students_pages' was never awaited self._target(*self._args, **self._kwargs) RuntimeWarning: Enable tracemalloc to get the object allocation traceback

有没有办法等待它或使用其他解决方案？

Answer 1

考虑改用 map()：

cores = 8

with Pool(cores) as p:
    p.map(self.fetch_students_pages, range(cores))

Asyncio 和 multiprocessing.Process- 如何传递协程？

Asyncio and multiprocessing.Process- how to pass a coroutine?

python

python-3.x

python-asyncio

python-multiprocessing