"Scrapy" 的图像抓取没有结果

Question

我一直在尝试使用 Scrapy 来抓取 Imgur 的图像，但我运行遇到了问题。

蜘蛛似乎工作正常，但它无法继续访问站点并完成工作。

我找不到我搞砸的地方..

items.py

import scrapy


class ImgurItem(scrapy.Item):
    title = scrapy.Field()
    image_urls = scrapy.Field()
    images = scrapy.Field()

setting.py

BOT_NAME = 'imgur'

SPIDER_MODULES = ['imgur.spiders']
NEWSPIDER_MODULE = 'imgur.spiders'
ITEM_PIPELINES = {'scrapy.pipelines.images.ImagesPipeline': 1}
IMAGES_STORE = '\Users3\Desktop\images'

imgur_spider.py

import scrapy

from scrapy.spiders import Rule, CrawlSpider
from scrapy.linkextractors import LinkExtractor
from imgur.items import ImgurItem

class ImgurSpider(CrawlSpider):
    name = 'imgur'
    allowed_domains = ['imgur.com']
    start_url = ['http://imgur.com']
    rules = [Rule(LinkExtractor(allow=['/gallery/.*']), 'parse_imgur')]

    def parse_imgur(self, response):
        image = ImgurItem()
        image['title'] = response.xpath("//h2[@id='image-title']/text()").extract()
        rel = response.xpath("//img/@src").extract()
        image['image_urls'] = ['http:'+rel[0]]
        return image

日志

2015-10-16 16:36:50 [scrapy] INFO: Scrapy 1.0.3 started (bot: imgur)
2015-10-16 16:36:50 [scrapy] INFO: Optional features available: ssl, http11
2015-10-16 16:36:50 [scrapy] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'imgur.spiders', 'SPIDER_MODULES': ['imgur.spiders'], 'BOT_NAME': 'imgur'}
2015-10-16 16:36:50 [scrapy] INFO: Enabled extensions: CloseSpider, TelnetConsole, LogStats, CoreStats, SpiderState
2015-10-16 16:36:50 [scrapy] INFO: Enabled downloader middlewares: HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, MetaRefreshMiddleware, HttpCompressionMiddleware, RedirectMiddleware, CookiesMiddleware, ChunkedTransferMiddleware, DownloaderStats
2015-10-16 16:36:50 [scrapy] INFO: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlLengthMiddleware, DepthMiddleware
2015-10-16 16:36:50 [scrapy] INFO: Enabled item pipelines: ImagesPipeline
2015-10-16 16:36:50 [scrapy] INFO: Spider opened
2015-10-16 16:36:50 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped     0 items (at 0 items/min)
2015-10-16 16:36:50 [scrapy] DEBUG: Telnet console listening on   127.0.0.1:6023
2015-10-16 16:36:50 [scrapy] INFO: Closing spider (finished)
2015-10-16 16:36:50 [scrapy] INFO: Dumping Scrapy stats:
{'finish_reason': 'finished',
'finish_time': datetime.datetime(2015, 10, 16, 15, 36, 50, 469000),
'log_count/DEBUG': 1,
'log_count/INFO': 7,
'start_time': datetime.datetime(2015, 10, 16, 15, 36, 50, 462000)}
2015-10-16 16:36:50 [scrapy] INFO: Spider closed (finished)

Answer 1

您的抓取工具将 http://imgur.com/gallery/Kgb24，然后查找所有包含“/gallery/”的链接（不需要额外的 .*）。它没有找到任何内容（因为页面上有 none），所以爬虫完成了。

编辑：清理完代码，跳出来start_url = ...应该是start_urls = ...。照原样，Scrapy 什么也抓不到，因为它没有从任何地方开始提供。

"Scrapy" 的图像抓取没有结果

Image scraping with "Scrapy" getting no results

python

image

web-crawler

scrapy