Scrapy 蜘蛛在尝试遍历已爬网的网址时无法正常工作

Question

我是 Scrapy 的新手。当我试图从论坛中抓取数据时，我的蜘蛛无法正常工作。当我是运行我的蜘蛛时，它只给我打印的 url 并在之后停止。所以我认为问题出在两个函数 parse 和 parse_data 的兼容性上，但我可能错了。这是我的代码：

import scrapy, time


class ForumSpiderSpider(scrapy.Spider):
    name = 'forum_spider'
    allowed_domains = ['visforvoltage.org/latest_tech/']
    start_urls = ['http://visforvoltage.org/latest_tech//']

    def parse(self, response):
        for href in response.css(r"tbody a[href*='/forum/']::attr(href)").extract():
           url = response.urljoin(href)
           print(url)
           req = scrapy.Request(url, callback=self.parse_data)
           time.sleep(10)
           yield req

    def parse_data(self, response):
        for url in response.css('html').extract():
           data = {}
           data['name'] = response.css(r"div[class='author-pane-line author-name'] span[class='username']::text").extract()
           data['date'] = response.css(r"div[class='forum-posted-on']:contains('-') ::text").extract()
           data['title'] = response.css(r"div[class='section'] h1[class='title']::text").extract()
           data['body'] = response.css(r"div[class='field-items'] p::text").extract()
           yield data
       

        next_page = response.css(r"li[class='pager-next'] a[href*='page=']::attr(href)").extract()
        if next_page:
            yield scrapy.Request(
                response.urljoin(next_page),
                callback=self.parse)

这是输出：

2020-07-23 23:09:58 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'visforvoltage.org': <GET https://visforvoltage.org/forum/14521-aquired-a123-m1-cells-need-charger-and-bms>
https://visforvoltage.org/forum/14448-battery-charger-problems
https://visforvoltage.org/forum/14191-vectrix-trickle-charger
https://visforvoltage.org/forum/14460-what-epoxy-would-you-recommend-loose-magnet-repair
https://visforvoltage.org/forum/14429-importance-correct-grounding-and-well-built-plugs
https://visforvoltage.org/forum/14457-147v-charger-24v-lead-acid-charger-and-dying-vectrix-cells
https://visforvoltage.org/forum/6723-lithium-safety-e-bike
https://visforvoltage.org/forum/11488-how-does-24v-4-wire-reversible-motor-work
https://visforvoltage.org/forum/14444-new-sevcon-gen-4-80v-sale
https://visforvoltage.org/forum/14443-new-sevcon-gen-4-80v-sale
https://visforvoltage.org/forum/12495-3500w-hub-motor-question-about-real-power-and-breaker
https://visforvoltage.org/forum/14402-vectrix-vx-1-battery-pack-problem
https://visforvoltage.org/forum/14068-vectrix-trickle-charger
https://visforvoltage.org/forum/2931-drill-motors
https://visforvoltage.org/forum/14384-help-repairing-gio-hub-motor-freewheel-sprocket
https://visforvoltage.org/forum/14381-zev-charger
https://visforvoltage.org/forum/8726-performance-unite-my1020-1000w-motor
https://visforvoltage.org/forum/7012-controler-mod-veloteq
https://visforvoltage.org/forum/14331-scooter-chargers-general-nfpanec
https://visforvoltage.org/forum/14320-charging-nissan-leaf-cells-lifepo4-charger
https://visforvoltage.org/forum/3763-newber-needs-help-new-gift-kollmorgan-hub-motor
https://visforvoltage.org/forum/14096-european-bldc-controller-seller
https://visforvoltage.org/forum/14242-lithium-bms-vs-manual-battery-balancing
https://visforvoltage.org/forum/14236-mosfet-wiring-ignition-key
https://visforvoltage.org/forum/2007-ok-dumb-question-time%3A-about-golf-cart-controllers
https://visforvoltage.org/forum/10524-my-mf70-recommended-powerpoles-arrived-today
https://visforvoltage.org/forum/9460-how-determine-battery-capacity
https://visforvoltage.org/forum/7705-tricking-0-5-v-hall-effect-throttle
https://visforvoltage.org/forum/13446-overcharged-lead-acid-battery-what-do
https://visforvoltage.org/forum/14157-reliable-high-performance-battery-enoeco-bt-p380
https://visforvoltage.org/forum/2702-hands-test-48-volt-20-ah-lifepo4-pack-ping-battery
https://visforvoltage.org/forum/14034-simple-and-cheap-ev-can-bus-adaptor
https://visforvoltage.org/forum/13933-zivan-ng-3-charger-specs-and-use
https://visforvoltage.org/forum/13099-controllers
https://visforvoltage.org/forum/13866-electric-motor-werks-demos-25-kilowatt-diy-chademo-leaf
https://visforvoltage.org/forum/13796-motor-theory-ac-vs-bldc
https://visforvoltage.org/forum/6184-bypass-bms-lifepo4-good-idea-or-not
https://visforvoltage.org/forum/13763-positive-feedback-kelly-controller
https://visforvoltage.org/forum/13764-any-users-smart-battery-drop-replacement-zapino-and-others
https://visforvoltage.org/forum/13760-contactor-or-fuse-position-circuit-rules-why
https://visforvoltage.org/forum/13759-contactor-or-fuse-position-circuit-rules-why
https://visforvoltage.org/forum/12725-repairing-lithium-battery-pack
https://visforvoltage.org/forum/13752-questions-sepex-motor-theory
https://visforvoltage.org/forum/13738-programming-curtis-controller-software
https://visforvoltage.org/forum/13741-making-own-simple-controller
https://visforvoltage.org/forum/12420-idea-charging-electric-car-portably-wo-relying-electricity-infrastructure
2020-07-23 23:17:28 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 2 pages/min), scraped 0 items (at 0 items/min)
2020-07-23 23:17:28 [scrapy.core.engine] INFO: Closing spider (finished)

正如我所见，它没有遍历这些链接并从中收集数据。这可能是什么原因？我将非常感谢任何帮助。谢谢！

Answer 1

问题可能是请求被过滤了，因为它们不属于允许的域。

allowed_domains = ['visforvoltage.org/latest_tech/']

新请求url：

https://visforvoltage.org/forum/14448-battery-charger-problems
https://visforvoltage.org/forum/14191-vectrix-trickle-charger
...

因为请求是针对 url visforvoltage.org/forum/ 而不是 visforvoltage.org/latest_tech/

您可以完全删除允许的域属性，或更改为：

allowed_domains = ['visforvoltage.org']

这将使他们抓取页面，您将在日志的这一行中看到不同的值：

2020-07-23 23:17:28 [scrapy.extensions.logstats] INFO: Crawled 2 pages (at 2 pages/min), scraped 0 items (at 0 items/min)

但是解析中的selector似乎不对。

此 selector 将 select 整个页面，而 extract() 方法将 return 它作为一个列表。所以你将有一个 list，只有一个 string 由页面的所有 HTML 组成。

response.css('html').extract()

您可以阅读更多关于 selectors 和 getall()/extract() 方法 here.

Answer 2

对我有用。

import scrapy, time


class ForumSpiderSpider(scrapy.Spider):
    name = 'forum_spider'
    allowed_domains = ['visforvoltage.org/latest_tech/']
    start_urls = ['http://visforvoltage.org/latest_tech/']

    def parse(self, response):
        for href in response.css(r"tbody a[href*='/forum/']::attr(href)").extract():
            url = response.urljoin(href)
            req = scrapy.Request(url, callback=self.parse_data, dont_filter=True)
            yield req

    def parse_data(self, response):
        for url in response.css('html'):
            data = {}
            data['name'] = url.css(r"div[class='author-pane-line author-name'] span[class='username']::text").extract()
            data['date'] = url.css(r"div[class='forum-posted-on']:contains('-') ::text").extract()
            data['title'] = url.css(r"div[class='section'] h1[class='title']::text").extract()
            data['body'] = url.css(r"div[class='field-items'] p::text").extract()
            yield data

        next_page = response.css(r"li[class='pager-next'] a[href*='page=']::attr(href)").extract()
        if next_page:
            yield scrapy.Request(
                response.urljoin(next_page),
                callback=self.parse)

Scrapy 蜘蛛在尝试遍历已爬网的网址时无法正常工作

Scrapy spider is not working when trying to iterate over crawled urls

css

python

web-crawler

scrapy

web-scraping