如何使用 asyncio 和 aiohttp 进行循环而不是循环？

Question

我的代码以这种方式工作，但由于循环，它的速度非常慢，你能帮我，让它与 aiohttp、asyncio 一起工作吗？

def field_info(field_link):
    response = requests.get(field_link)
    soup = BeautifulSoup(response.text, 'html.parser')
    races = soup.findAll('header', {'class': 'dc-field-header'})
    tables = soup.findAll('table', {'class': 'dc-field-comp'})

    for i in range(len(races)):
        race_name = races[i].find('h3').text
        race_time = races[i].find('time').text

        names = tables[i].findAll('span', {'class': 'title'})
        trainers = tables[i].findAll('span', {'class': 'trainer'})
        table = []

        for j in range(len(names)):
            table.append({
                'Name': names[j].text,
                'Trainer': trainers[j].text,
            })

        return {
                'RaceName': race_name,
                'RaceTime': race_time,
                'Table': table
                }


links = [link1, link2, link3]
for link in links:
    scraped_info += field_info(link)

Answer 1

1)创建一个协程异步发起请求：

import asyncio
import aiohttp


async def get_text(url):
    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            return await resp.text()

2) 将所有同步请求替换为等待此协程，使外部函数也成为协程：

async def field_info(field_link):              # async - to make outer function coroutine
    text = await get_text(field_link)          # await - to get result from async funcion
    soup = BeautifulSoup(text, 'html.parser')

3) 使用 asyncio.gather():

使外部代码同时执行作业

async def main():
    links = [link1, link2, link3]

    scraped_info = asyncio.gather(*[
        field_info(link)
        for link
        in links
    ])  # do multiple field_info coroutines concurrently (parallely)

4) 将 top-level 协程传递给 asyncio.run():

asyncio.run(main())

如何使用 asyncio 和 aiohttp 进行循环而不是循环？

How to use asyncio and aiohttp for looping instead of for looping?

web-scraping

python-3.x

python-asyncio

aiohttp