Scrapy延迟请求

Question

每次我运行我的代码我的 ip 都被禁止。我需要帮助将每个请求延迟 10 秒。我试图在代码中放置 DOWNLOAD_DELAY 但它没有给出任何结果。感谢任何帮助。

# item class included here
        class DmozItem(scrapy.Item):
            # define the fields for your item here like:
            link = scrapy.Field()
            attr = scrapy.Field()


        class DmozSpider(scrapy.Spider):
            name = "dmoz"
            allowed_domains = ["craigslist.org"]
            start_urls = [
            "https://washingtondc.craigslist.org/search/fua"
            ]

            BASE_URL = 'https://washingtondc.craigslist.org/'

            def parse(self, response):
                links = response.xpath('//a[@class="hdrlnk"]/@href').extract()
                for link in links:
                    absolute_url = self.BASE_URL + link
                    yield scrapy.Request(absolute_url, callback=self.parse_attr)

            def parse_attr(self, response):
                match = re.search(r"(\w+)\.html", response.url)
                if match:
                    item_id = match.group(1)
                    url = self.BASE_URL + "reply/nos/vgm/" + item_id

                    item = DmozItem()
                    item["link"] = response.url

                    return scrapy.Request(url, meta={'item': item}, callback=self.parse_contact)

            def parse_contact(self, response):
                item = response.meta['item']
                item["attr"] = "".join(response.xpath("//div[@class='anonemail']//text()").extract())
                return item

Answer 1

您需要设置项目的 DOWNLOAD_DELAY in settings.py。请注意，您可能还需要限制并发性。默认情况下，并发数为 8，因此您同时访问了 8 个请求的网站。

# settings.py
DOWNLOAD_DELAY = 1
CONCURRENT_REQUESTS_PER_DOMAIN = 2

从 Scrapy 1.0 开始，您还可以在 spider 中进行自定义设置，这样您就可以这样做：

class DmozSpider(Spider):
    name = "dmoz"
    allowed_domains = ["dmoz.org"]
    start_urls = [
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
        "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/",
    ]

    custom_settings = {
        "DOWNLOAD_DELAY": 5,
        "CONCURRENT_REQUESTS_PER_DOMAIN": 2
    }

延迟和并发是按下载器插槽设置的，而不是按请求设置的。要实际检查你有什么下载，你可以尝试这样的事情

def parse(self, response):
    """
    """
    delay = self.crawler.engine.downloader.slots["www.dmoz.org"].delay
    concurrency = self.crawler.engine.downloader.slots["www.dmoz.org"].concurrency
    self.log("Delay {}, concurrency {} for request {}".format(delay, concurrency, response.request))
    return

Scrapy延迟请求

Scrapy delay request

python

web-crawler

scrapy