在抓取网站时尝试访问变量

Question

我目前正在尝试抓取一个网站（url 在下面的代码中），但是当我拉出 html 我想要使用的部分时，我得到的只是我正在寻找的信息的变量名称。当我手动检查页面的 html 时，变量的实际值存在，但我假设当我抓取页面时，我看到的只是从其他地方引用变量的网站。

我希望有人可以帮助我尝试访问此信息。我试过使用 selenium 抓取网站的 html，但是我似乎只是找回了使用请求时抓取的相同 html（也许我做错了，有人可以告诉我正确的方法).

这是我的代码的精炼版：

import scrapy #For scraping
from scrapy import Selector
import requests

headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0',
           'Accept-Language': 'en-GB,en;q=0.5',
           'Referer': 'https://google.com',
           'DNT': '1'}

url = 'https://groceries.aldi.ie/en-GB/drinks/beer-ciders'
html = requests.get(url, headers=headers).content
sel = Selector(text=html)
x  = sel.xpath('//*[@id="vueSearchResults"]/div//span')#/div[2]/div/div[4]/div/div/span/span

print((x.extract())[8])

然后 returns 以下内容：

<span>{{Product.ListPrice}}</span>

我想从中得到'Product.ListPrice'的实际值。如果有人能指出正确的方向来访问这个变量的信息，或者一种抓取网站 html 的方法——正如用户浏览网页时所看到的那样，我将不胜感激。 ** 建议我通过此 API: 'https://groceries.aldi.ie/api/product/calculatePrices' 发送 POST 请求以及传递请求 headers 和有效负载，但我不完全确定如何把它拉下来（我是这个方面的新手），如果有人能给我一个如何执行这个的例子，我将不胜感激！谢谢！

Answer 1

这是通过 Scrapy 复制 POST 请求的方法。

代码

import scrapy
import json

class Test(scrapy.Spider):
    name = 'test'

    headers = {
        "authority": "groceries.aldi.ie",
        "pragma": "no-cache",
        "cache-control": "no-cache",
        "sec-ch-ua": "\" Not;A Brand\";v=\"99\", \"Google Chrome\";v=\"91\", \"Chromium\";v=\"91\"",
        "accept-language": "en-GB",
        "sec-ch-ua-mobile": "?0",
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.164 Safari/537.36",
        "websiteid": "a763fb4a-0224-4ca8-bdaa-a33a4b47a026",
        "content-type": "application/json",
        "accept": "application/json, text/javascript, */*; q=0.01",
        "x-requested-with": "XMLHttpRequest",
        "origin": "https://groceries.aldi.ie",
        "sec-fetch-site": "same-origin",
        "sec-fetch-mode": "cors",
        "sec-fetch-dest": "empty",
        "referer": "https://groceries.aldi.ie/en-GB/drinks/beer-ciders?sortDirection=asc&page=1"
    }

    body = '{"products":["4088600284026","5391528370382","5391528372836","5391528372850","5391528372874","4088600298696","4088600103709","4088600388700","5035766046028","5000213021934","4088600012551","4088600325934","4088600300153","25389111","4072700001171","4088600012537","4088600012544","4088600013138","4088600013145","4088600103525","4088600103532","4088600103570","4088600103600","4088600135182","4088600141848","4088600142050","4088600158105","4088600217024","4088600217208","4088600241302","4088600249292","4088600249308","4088600280615","4088600281445","4088600283043","4088600284088","4088600295688","4088600295800","4088600295817","4088600303925"]}'

    def start_requests(self):
       url = 'https://groceries.aldi.ie/api/product/calculatePrices'
       yield scrapy.Request(url=url,method='POST', headers=self.headers,body=self.body)

    def parse(self,response):
        data = json.loads(response.body)
        for i in data.get('ProductPrices'):
            print('Listing price is', i.get('ListPrice'),'\t','Unit Price is',i.get('UnitPrice') )

输出（截断）

Listing price is €4.19   Unit Price is €4.23
Listing price is €1.99   Unit Price is €3.98
Listing price is €1.99   Unit Price is €3.98
Listing price is €1.99   Unit Price is €3.98
Listing price is €1.99   Unit Price is €3.98

在抓取网站时尝试访问变量

Trying to access variables while scraping website

html

python

scrapy

web-scraping