如何在使用 aiohttp 时获取 response_time 和 response_size

Question

是否可以获取使用 aiohttp 发出的每个请求的响应时间和响应大小？

文档似乎在任何地方都没有这些属性。

谢谢

Answer 1

一种可能是：

测量请求前的时间点
请求后测量时间点
区别在于响应时间
用'response.text()'你得到响应并且可以用'len()'

一个小 self-contained 示例可能如下所示：

import time
import asyncio
from aiohttp import ClientSession


async def fetch(session, url):
    start = time.time()
    async with session.get(url) as response:
        result = await response.text()
        end = time.time()
        print(url, ": ", end - start, "response length:", len(result))
        return result


async def crawl(urls: set):
    async with ClientSession() as session:
        tasks = []
        for url in urls:
            tasks.append(
                fetch(session, url)
            )
        await asyncio.gather(*tasks)


if __name__ == "__main__":
    urlSet = {"https://www.software7.biz/tst/number.php",
              "https://www.software7.biz/tst/number1.php",
              "https://www.software7.biz"}
    asyncio.run(crawl(urlSet))

测试

两个端点 number.php 和 number1.php 在服务器端分别有 3 秒的延迟，并且每个端点都返回一个两位数。

调试控制台中的输出如下所示：

https://www.software7.biz :  0.16438698768615723 response length: 4431
https://www.software7.biz/tst/number1.php :  1.249755859375 response length: 2
https://www.software7.biz/tst/number.php :  3.214473009109497 response length: 2

Answer 2

len(response.text()) 将 return 解压缩响应的大小。如果您想要原始压缩响应的大小，您需要在创建 aiohttp.ClientSession 期间设置 auto_decompress=False。之后你可以用len(await response.read())得到它。但它会使 response.text() 不可用，因为它需要未压缩的响应。要使其再次可用，您必须手动解压缩它：

import time
import zlib
import brotli

async with aiohttp.ClientSession(auto_decompress=False) as session:
    start = time.monotonic()
    response = await session.get(url='www.test.com')
    response_time = time.monotonic() - start
    response_size = len(await response.read())

    encoding = response.headers['Content-Encoding']
    if encoding == 'gzip':
        response._body = zlib.decompress(response._body, 16 + zlib.MAX_WBITS)
    elif encoding == 'deflate':
        response._body = zlib.decompress(response._body, -zlib.MAX_WBITS)
    elif encoding == 'br':
        response._body == brotli.decompress(response._body)

    response_text = await response.text()

关于 time.time() 来自 pymotw.com:

Because time.time() looks at the system clock, and the system clock can be changed by the user or system services for synchronizing clocks across multiple computers, calling time.time() repeatedly may produce values that go forwards and backwards. This can result in unexpected behavior when trying to measure durations or otherwise use those times for computation. Avoid those situations by using time.monotonic(), which always returns values that go forward.

aiohttp docs suggest to use loop.time()（也是单调的）：

async def on_request_start(session, trace_config_ctx, params):
    trace_config_ctx.start = asyncio.get_event_loop().time()

async def on_request_end(session, trace_config_ctx, params):
    elapsed = asyncio.get_event_loop().time() - trace_config_ctx.start
    print("Request took {}".format(elapsed))

trace_config = aiohttp.TraceConfig()
trace_config.on_request_start.append(on_request_start)
trace_config.on_request_end.append(on_request_end)
async with aiohttp.ClientSession(trace_configs=[trace_config]) as client:
    client.get('http://example.com/some/redirect/')

如何在使用 aiohttp 时获取 response_time 和 response_size

how to get response_time and response_size while using aiohttp

python

python-3.x

python-asyncio

aiohttp