Python 3.x 中异步 HTTP 请求的异常处理

Question

我正在尝试处理异步 HTTP 请求。我从另一个模块调用 async_provider() 函数，并使用生成的 response.text() 执行后续任务。它仅在所有请求都成功时才有效。但是我无法处理失败请求的任何异常（无论异常的原因是什么）。谢谢您的帮助。这是代码的相关部分：

import asyncio
import aiohttp

# i call this function from another module
def async_provider():
    list_a, list_b = asyncio.run(main())
    return list_a, list_b


async def fetch(session, url):
    # session.post request cases
    if url == "http://...1":
        referer = "http://...referer"
        user_agent = (
            "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) "
            "AppleWebKit/605.1.15 (KHTML, like Gecko) "
            "Version/12.1 Safari/605.1.15"
        )
        payload = {'key1': 'value1', 'key2': 'value2'}
        async with session.post(
            url, data=payload, headers={"Referer": referer, "User-Agent": user_agent}
        ) as response:
            if response.status != 200:
                response.raise_for_status()
            return await response.text()

    # session.get request cases
    else:
        async with session.get(url) as response:
            if response.status != 200:
                response.raise_for_status()
            return await response.text()


async def fetch_all(session, urls):
    results = await asyncio.gather(
        *[asyncio.create_task(fetch(session, url)) for url in urls]
    )
    return results


async def main():
    urls = ["http://...1", "http://...2", "http://...3"]
    async with aiohttp.ClientSession() as session:
        response_text_1, response_text_2, response_text_3 = await fetch_all(
            session, urls
        )
    # some task with response text

任何异常都会中断所有请求

Answer 1

检查"return_exceptions" flag on gather。

results = await asyncio.gather(
    *[asyncio.create_task(fetch(session, url)) for url in urls],
    return_exceptions=True
)

它会 return 你完成任务的列表。然后您可以使用他们的 Task.result() 或 Task.exception() 重新提出或检查是否有异常的方法。

Python 3.x 中异步 HTTP 请求的异常处理

Exception handling on asyncronously HTTP-requests in Python 3.x

python-3.x

async-await

python-asyncio

aiohttp