龙卷风 gen.sleep 添加延迟

Question

我正在尝试以异步方式在请求之间添加延迟。当我使用 Tornado gen.sleep(x) 时，我的函数（启动）没有被执行。如果我从 yield gen.sleep(1.0) 中删除 yield，将调用函数，但不会添加延迟。如何在我的 for 循环中的请求之间添加延迟？我需要控制对外部的每秒请求 API。如果我使用 time.sleep，则在所有请求完成后响应会延迟。尝试添加 @gen.engine 装饰器来启动函数，但没有结果。

代码：

import collections
import tornado.httpclient


class BacklogClient(object):
    MAX_CONCURRENT_REQUESTS = 20

    def __init__(self, ioloop):
        self.ioloop = ioloop
        self.client = tornado.httpclient.AsyncHTTPClient(max_clients=self.MAX_CONCURRENT_REQUESTS)
        self.client.configure(None, defaults=dict(connect_timeout=20, request_timeout=30))
        self.backlog = collections.deque()
        self.concurrent_requests = 0

    def __get_callback(self, function):
        def wrapped(*args, **kwargs):
            self.concurrent_requests -= 1
            self.try_run_request()
            return function(*args, **kwargs)

        return wrapped

    def try_run_request(self):
        while self.backlog and self.concurrent_requests < self.MAX_CONCURRENT_REQUESTS:
            request, callback = self.backlog.popleft()
            self.client.fetch(request, callback=callback)
            self.concurrent_requests += 1

    def fetch(self, request, callback=None):
        wrapped = self.__get_callback(callback)

        self.backlog.append((request, wrapped))
        self.try_run_request()


import time
from tornado import ioloop, httpclient, gen


class TornadoBacklog:
    def __init__(self):

        self.queue = 0
        self.debug = 1
        self.toProcess = [
            'http://google.com',
            'http://yahoo.com',
            'http://nytimes.com',
            'http://msn.com',
            'http://cnn.com',
            'http://twitter.com',
            'http://facebook.com',
        ]


    def handle_request(self, response):

        print response.code
        if not self.backlog.backlog and self.backlog.concurrent_requests == 0:
            ioloop.IOLoop.instance().stop()


    def launch(self):
        self.ioloop = ioloop.IOLoop.current()
        self.backlog = BacklogClient(self.ioloop)

        for item in self.toProcess:
            yield gen.sleep(1.0)
            print item
            self.backlog.fetch(
                httpclient.HTTPRequest(
                    item,
                    method='GET',
                    headers=None,
                ),
                self.handle_request
            )

        self.ioloop.start()



def main():
    start_time = time.time()

    scraper = TornadoBacklog()
    scraper.launch()

    elapsed_time = time.time() - start_time
    print('Process took %f seconds processed %d items.' % (elapsed_time, len(scraper.toProcess)))


if __name__ == "__main__":
    main()

参考：https://github.com/tornadoweb/tornado/issues/1400

Answer 1

Tornado 协程有两个组件：

它们包含 "yield" 个语句
它们装饰有"gen.coroutine"

在 "launch" 函数上使用 "coroutine" 装饰器：

@gen.coroutine
def launch(self):

运行一个 Tornado 协程从头到尾都是这样的：

tornado.ioloop.IOLoop.current().run_sync(launch)

从 "launch" 函数中删除对 "ioloop.start" 的调用：循环运行 "launch" 函数，反之亦然。

龙卷风 gen.sleep 添加延迟

Tornado gen.sleep add delay

python

asynchronous

tornado