使用 item loader scrapy 获取键中的值

Question

我正在尝试从网页响应页面的键中提取一些值。不幸的是，当我这样做时，它 returns 只有键，我似乎无法获取值。因为每个键都是一个长列表，并且它们已编号，所以我似乎无法弄清楚如何获取所有键的值。

例如，这是我的工作代码：

import scrapy
from scrapy.crawler import CrawlerProcess
from scrapy.loader import ItemLoader
from scrapy.item import Field
from itemloaders.processors import TakeFirst

class DepopItem(scrapy.Item):
    brands = Field(output_processor=TakeFirst())

class DepopSpider(scrapy.Spider):
    name = 'depop'
    allowed_domains = ["depop.com"]
    start_urls = ['https://webapi.depop.com/api/v2/search/filters/aggregates/?brands=1596&itemsPerPage=24&country=gb&currency=GBP&sort=relevance']

    
    custom_settings = {
        'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
    }
    
    def start_requests(self):
        for url in self.start_urls:
            yield scrapy.Request(
                url=url, 
                callback=self.parse,
             )

    def parse(self, response):
        resp= response.json()['brands']
        for item in resp:
            loader = ItemLoader(DepopItem(), selector=item)
            loader.add_value('brands', item)
 
            yield loader.load_item()

这个returns键列表：

{"brands": "1"}
{"brands": "2"}
{"brands": "3"}
{"brands": "4"}
{"brands": "5"}
{"brands": "7"}
{"brands": "9"}

相反，我想要与这些键对应的值：

{"brands": 946}
{"brands": 2376}
{"brands": 1286}
{"brands": 2774}
{"brands": 489}
{"brands": 11572}
{"brands": 1212}

Answer 1

我不确定 scrapy 怎么样，但你可以简单地做：

import requests
import json
from itertools import starmap
from requests.models import Response
from typing import Dict, List


url = "https://webapi.depop.com/api/v2/search/filters/aggregates/?brands=1596&itemsPerPage=24&country=gb&currency=GBP&sort=relevance"
resp: Response = requests.get(url)
data: Dict = json.loads(resp.text).get("brands")
values: List[Dict] = list(starmap(lambda k,v: {"brands": v["count"]}, data.items()))

输出：

[{'brands': 989},
 {'brands': 1838},
 {'brands': 2415},
 {'brands': 1344},
 ...]

Answer 2

使用 values() 或 resp[item]。

示例：

import scrapy
from scrapy.loader import ItemLoader
from scrapy.item import Field
from itemloaders.processors import TakeFirst


class DepopItem(scrapy.Item):
    brands = Field(output_processor=TakeFirst())


class DepopSpider(scrapy.Spider):
    name = 'depop'
    allowed_domains = ["depop.com"]
    start_urls = ['https://webapi.depop.com/api/v2/search/filters/aggregates/?brands=1596&itemsPerPage=24&country=gb&currency=GBP&sort=relevance']

    custom_settings = {
        'USER_AGENT': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36',
    }

    def parse(self, response):
        resp = response.json()['brands']
        for item in resp.values():
            loader = ItemLoader(DepopItem(), selector=item)
            loader.add_value('brands', item['count'])
            yield loader.load_item()

输出：

{'brands': 888}
{'brands': 1}
{'brands': 52}
{'brands': 138}
{'brands': 148}
...
...
...

使用 item loader scrapy 获取键中的值

Get values within keys with item loader scrapy

python

scrapy

web-scraping