使用 BeautifulSoup 抓取 shopee.co.id 时得到一个空列表

Question

我试图使用 BeautifulSoup 包在 https://shopee.co.id/top_products 上抓取顶级产品的名称和销售额。但无论我尝试什么，它总是 returns 一个空的产品列表。你能解释一下幕后发生的事情吗？这是我的代码：

import requests
from bs4 import BeautifulSoup as bs

headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36',
    'cookie': '_gcl_au=1.1.961206468.1594951946; _med=refer; _fbp=fb.2.1594951949275.1940955365; SPC_IA=-1; SPC_F=y1evilme0ImdfEmNWEc08bul3d8toc33; REC_T_ID=fab983c8-c7d2-11ea-a977-ccbbfe23657a; SPC_SI=uv1y64sfvhx3w6dir503ixw89ve2ixt4; _gid=GA1.3.413262278.1594951963; SPC_U=286107140; SPC_EC=GwoQmu7TiknULYXKODlEi5vEgjawyqNcpIWQjoxjQEW2yJ3H/jsB1Pw9iCgGRGYFfAkT/Ej00ruDcf7DHjg4eNGWbCG+0uXcKb7bqLDcn+A2hEl1XMtj1FCCIES7k17xoVdYW1tGg0qaXnSz0/Uf3iaEIIk7Q9rqsnT+COWVg8Y=; csrftoken=5MdKKnZH5boQXpaAza1kOVLRFBjx1eij; welcomePkgShown=true; _ga=GA1.1.1693450966.1594951955; _dc_gtm_UA-61904553-8=1; REC_MD_30_2002454304=1595153616; _ga_SW6D8G0HXK=GS1.1.1595152099.14.1.1595153019.0; REC_MD_41_1000044=1595153318_0_50_0_49; SPC_R_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="; SPC_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_R_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="'
}

shopee_url = 'https://shopee.co.id/top_products'
response = requests.get(shopee_url, headers=headers)
soup = bs(response.text, "html.parser")

products = soup.find_all("._3S8sOC _2QfAXF")
print(type(products))
print(products)

感谢您阅读到这里。我看过一些类似的答案，但它们似乎并不适用于此。

Answer 1

如果您愿意使用 selenium，您可以使用 selenium 提出请求：

from selenium import webdriver
from bs4 import BeautifulSoup as bs

headers = {
    'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.89 Safari/537.36',
    'cookie': '_gcl_au=1.1.961206468.1594951946; _med=refer; _fbp=fb.2.1594951949275.1940955365; SPC_IA=-1; SPC_F=y1evilme0ImdfEmNWEc08bul3d8toc33; REC_T_ID=fab983c8-c7d2-11ea-a977-ccbbfe23657a; SPC_SI=uv1y64sfvhx3w6dir503ixw89ve2ixt4; _gid=GA1.3.413262278.1594951963; SPC_U=286107140; SPC_EC=GwoQmu7TiknULYXKODlEi5vEgjawyqNcpIWQjoxjQEW2yJ3H/jsB1Pw9iCgGRGYFfAkT/Ej00ruDcf7DHjg4eNGWbCG+0uXcKb7bqLDcn+A2hEl1XMtj1FCCIES7k17xoVdYW1tGg0qaXnSz0/Uf3iaEIIk7Q9rqsnT+COWVg8Y=; csrftoken=5MdKKnZH5boQXpaAza1kOVLRFBjx1eij; welcomePkgShown=true; _ga=GA1.1.1693450966.1594951955; _dc_gtm_UA-61904553-8=1; REC_MD_30_2002454304=1595153616; _ga_SW6D8G0HXK=GS1.1.1595152099.14.1.1595153019.0; REC_MD_41_1000044=1595153318_0_50_0_49; SPC_R_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="; SPC_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_R_T_IV="IETR5YkWloW3OcKf80c6RQ=="; SPC_T_ID="Am9bCo3cc3Jno2mV5RDkLJIVsbIWEDTC6ezJknXdVVRfxlQRoGDcya57fIQsioFKZWhP8/9PAGhldR0L/efzcrKONe62GAzvsztkZHfAl0I="'
}
driver = webdriver.Firefox(executable_path='c:/program/geckodriver.exe')
shopee_url = 'https://shopee.co.id/top_products'
driver.get(shopee_url)
driver.implicitly_wait(15)
response = driver.page_source
driver.close()
soup = bs(response, "html.parser")

products = soup.select(".shopee-item-card__lower-padding")

#print(products)

for product in products:
    name = product.select_one('.shopee-item-card__text-name span')
    print(name.get_text(strip=True))
    price = product.select_one('.shopee-item-card__current-price')
    print(price.text)

请注意，您需要安装 Firefox、geckodriver 和 selenium。

输出：

Kuota Telkomsel Cek Dadakan
Rp300
Aigo 1GB 5Hari
Rp7.375
Paket Data Tri Mix Small 5GB
Rp4.500
Paket Data Tri Mix Small 2.75GB
Rp9.500
*PROMO MURAH* SURPRISE DEAL PAKET DATA INTERNET KUOTA TELKOMSEL SIMPATI AS LOOP OMG 20GB 30GB 50GB
Rp1.000
Kuota Telkomsel Cek Dadakan
Rp500
Paket Data Telkomsel 12GB
Rp98.000
Paket Data Tri AON Unlimited + 6GB
Rp58.000
KUOTA INTERNET PAKET DATA INTERNET TELKOMSEL KUOTA DATA FLASH Kartu Simpati As Loop (KUOTA MURAH)
Rp1.000
[MURAH] PAKET DATA TELKOMSEL ∣ KUOTA INTERNET ∣ PAKET DATA ∣ SIMPATI  ∣ AS  ∣ LOOOP
Rp1.000
Paket Data Tri AddOn 12GB
Rp48.000
Paket Data  Indosat Freedom U 7GB+20GB Apps / 30 Hari
Rp66.500
PAKET DATA TELKOMSEL ISI ULANG SIMPATI 14GB/27GB/37GB - AS 6,5GB/10GB/17GB/32GB - LOOP 3GB/8GB/15GB
Rp38.000
Paket Data Indosat Freedom Internet 4GB / 30 Hari
Rp24.000
Paket Data Indosat Freedom U 1GB+4.5GB Apps / 30 Hari
Rp24.000
KUOTA TELKOMSEL paket kuota data internet telkomsel termurah 52gb injek injeck kuota  promo
Rp1.000
Kuota Telkomsel Simpati 52GB 50GB 24jam Langsung Masuk
Rp1.000
Paket Data Telkomsel 50GB
Rp195.000
Paket Data Indosat Freedom Internet 10GB /30 Hari
Rp48.000
KUOTA TELKOMSEL paket data internet telkomsel kuota internet TELKOMSEL  termurah simpati as loop
Rp1.000
Paket Data Telkomsel Data 25.000
Rp23.500
VOCHER VOUCHER KUOTA AXIS AIGO 1GB 5HARI NASIONAL
Rp7.390
PAKET DATA KUOTA INDOSAT [DISKON 15%] TERMURAH
Rp1.000
VOUCHER Axis 1gb 5 hari 24 Jam KUOTA 1 GB
Rp7.500
Paket Data Indosat Freedom Combo 14GB, 30 Hari
Rp48.500
[PROMO] KUOTA BESAR PAKET OMG TELKOMSEL MURAH
Rp1.000
Paket Data Indosat Freedom Combo 8GB/ 30 Hari
Rp33.500
TELKOMSEL PEMBAYARAN RESELLER/ AGEN atau Penambahan Selisih Harga Produk ThalhahCell
Rp1.000
Aigo 3GB 15Hari
Rp18.500
Paket Data Indosat Freedom U 3GB+15GB Apps / 30 Hari
Rp49.900

使用 BeautifulSoup 抓取 shopee.co.id 时得到一个空列表

Got an empty list when scraping shopee.co.id using BeautifulSoup

python

beautifulsoup

web-crawler