如何找到用于抓取数据的标签 'a'？

Question

我需要从这个网站抓取数据https://shop.freedompop.com/products?page=1

我使用BeautifulSoup解析html，发现我需要找到所有class_="product-results-item-link layout-row flex-gt-sm-33 flex-50"

我试过使用 containers = html_soup.find_all('a', class_="product-results-item-link layout-row flex-gt-sm-33 flex-50") 但是找不到

    from requests import get
    from bs4 import BeautifulSoup
    from time import sleep
    from random import randint
    import pandas as pd

    product_names = []
    status = []
    ori_prices = []
    sale_prices = []

    headers = {"Accept-Language": "en-US, en;q=0.5"}

    pages = [str(i) for i in range(1,2)]
    #pages = [str(i) for i in range(1,24)]

    for page in pages:

        response = get('https://shop.freedompop.com/products' + page, headers = headers)
        sleep(2)

        html_soup = BeautifulSoup(response.text, 'html.parser')

        containers = html_soup.find_all('a', class_="product-results-item-link layout-row flex-gt-sm-33 flex-50")

        print(containers)

我希望输出为 18 但实际输出为 []

Answer 1

网站通过 api 动态访问所有产品条目。所以你可以直接使用他们的 API 并获取数据：

https://shop.freedompop.com/api/shop/store/555/item?page=1&pageSize=500&sort=RELEVANCE

import json
from requests import get
from bs4 import BeautifulSoup


response = get('https://shop.freedompop.com/api/shop/store/555/item?pageSize=410&sort=RELEVANCE')
html_soup = BeautifulSoup(response.text, 'html.parser')
parsed_response = json.loads(html_soup.text)


for index,value in enumerate(a.get('results')):
    print(index, value)

Answer 2

正如 Pankaj 所说（所以请接受他的回答，因为我只是在扩展他的初始回复），使用请求 url 以漂亮的 json 格式向您提供数据.您还可以更改参数（即更改 'pageSize': '500' 以获得比第一页上的 18 个更多的产品：

import requests


url = 'https://shop.freedompop.com/api/shop/store/555/item'

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36'}
params = {
'page': '1',
'pageSize': '18',
'sort': 'RELEVANCE'}

jsonData = requests.get(url, headers=headers, params=params).json()

for product in jsonData['results']:
    print (product['title'])

输出：

Netgear Unite Mobile Hotspot (GSM)
LG Tribute 2, 8GB Blue (CDMA)
Samsung Galaxy S5, 16GB Charcoal Black (CDMA)
Samsung Galaxy S5, 16GB Shimmery White (CDMA)
Samsung Galaxy S5, 16GB Shimmery White (CDMA)
Samsung Galaxy S4 Enhanced, 16GB Black Mist (CDMA)
Kyocera Hydro Vibe, 8GB Black (CDMA)
Samsung Galaxy S4, 16GB White Frost (CDMA)
Samsung Galaxy S4, 16GB White Frost (CDMA)
Motorola Moto E (2nd Generation), 8GB Black (CDMA)
Apple iPhone 5s, 16GB Gold (CDMA)
Samsung Galaxy S4, 16GB Black Mist (CDMA)
Franklin Wireless R850 4G LTE Mobile Hotspot (CDMA)
Apple iPhone 6, 16GB Space Gray (CDMA)
Samsung Galaxy S4 Enhanced, 16GB White Frost (CDMA)
Huawei Union, 8GB Black (CDMA)
Samsung Galaxy S5, 16GB Copper Gold (CDMA)
Samsung Galaxy S4 Enhanced, 16GB Black Mist (CDMA)

更改参数：

import requests


url = 'https://shop.freedompop.com/api/shop/store/555/item'

headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36'}
params1 = {
'page': '1',
'pageSize': '18',
'sort': 'RELEVANCE'}

params2 = {
'page': '1',
'pageSize': '500',
'sort': 'RELEVANCE'}

jsonData = requests.get(url, headers=headers, params=params1).json()
print (len(jsonData['results']))

jsonData = requests.get(url, headers=headers, params=params2).json()
print (len(jsonData['results']))

输出：

18
405

如何找到用于抓取数据的标签 'a'？

How to find the tags 'a' for scraping the data?

python

beautifulsoup

scrape