Scrapy

Question

我是 Python 和 Scrapy 的新手，我认为答案应该很简单，但我自己很难弄清楚。该代码获取所有链接，跟随它们并记录文章的标题。我如何将 url 传递给我的物品？我想保存它与文章标题一起使用的短链接。谢谢

def parse(self, response):
    for url in response.xpath("//li[@id]/@data-shortlink").extract():
        yield scrapy.Request(url, callback=self.get_details)

def get_details(self, response):
        article = ArticleItem()
        article['title'] = response.xpath("//h1/text()").extract()
        yield article

Answer 1

因为它包含在 Response() object 中，您可以使用 response.url 得到 URL:

def get_details(self, response):
        article = ArticleItem()
        article['title'] = response.xpath("//h1/text()").extract()
        article['url'] = response.url
        yield article

Scrapy - 在关注链接时保存链接

Scrapy - saving links while following them

python

scrapy-spider