Beautiful Soup findAll 找不到值
Beautiful Soup findAll doesn't find value
我想用下面的代码 beautifulsoup 写 this site 上的产品价格，但是当我写代码时，列表 returns 是空的。
import requests as req from bs4 import BeautifulSoup as bs
url = "https://www.migros.com.tr/temel-gida-c-2?sayfa=1"
headers = {
'User-Agent': (
'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) '
'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36'
) }
urunler = []
rx = req.get(url, headers=headers) sympo = bs(rx.text, 'html.parser')
soup = bs(rx.content, 'lxml') print(sympo.findAll('span', {'class':
'amount'}))
print(sympo.findAll('span', {'class': 'amount'}))
[]
页面由 JS 调用生成，因此您可能需要查看 selenium 包。如果你只是在打印 soup 时检查输出，你只会得到页面的 header 内容：
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<title>Migros Sanal Market: Online Market Alışverişi</title>
<base href="/"/>
<meta content="width=device-width, initial-scale=1, maximum-scale=5, user-scalable=0" name="viewport"/>
<link href="manifest.json" rel="manifest"/>
<link href="/assets/favicon.ico" rel="icon" type="image/x-icon"/>
<meta content="#ff9933" name="theme-color"/>
<meta content="Geniş ürün yelpazesi, en uygun fiyatlı ürünleri ve aynı gün teslimat fırsatıyla online market alışverişinizde Migros Sanal Market size iyi gelecek!" name="description" property="og:description"/>
<meta content="Migros Sanal Market: Online Market Alışverişi" property="og:site_name"/>
<meta content="https://migros-dali-storage-prod.global.ssl.fastly.net/sanalmarket/custom/sanalmarket-seo-34706362.png" property="og:image"/>
<link href="/assets/images/seo/apple-touch-icon-57x57.png" rel="apple-touch-icon" sizes="57x57"/>
<link href="/assets/images/seo/apple-touch-icon-114x114.png" rel="apple-touch-icon" sizes="114x114"/>
<link href="/assets/images/seo/apple-touch-icon-72x72.png" rel="apple-touch-icon" sizes="72x72"/>
<link href="/assets/images/seo/apple-touch-icon-144x144.png" rel="apple-touch-icon" sizes="144x144"/>
<link href="/assets/images/seo/apple-touch-icon-60x60.png" rel="apple-touch-icon" sizes="60x60"/>
<link href="/assets/images/seo/apple-touch-icon-120x120.png" rel="apple-touch-icon" sizes="120x120"/>
<link href="/assets/images/seo/apple-touch-icon-76x76.png" rel="apple-touch-icon" sizes="76x76"/>
<link href="/assets/images/seo/apple-touch-icon-152x152.png" rel="apple-touch-icon" sizes="152x152"/>
<link href="/assets/images/seo/apple-touch-icon.png" rel="apple-touch-icon"/>
<link href="/assets/images/seo/apple-touch-icon-57x57-precomposed.png" rel="apple-touch-icon-precomposed" sizes="57x57"/>
<link href="/assets/images/seo/apple-touch-icon-114x114-precomposed.png" rel="apple-touch-icon-precomposed" sizes="114x114"/>
<link href="/assets/images/seo/apple-touch-icon-72x72-precomposed.png" rel="apple-touch-icon-precomposed" sizes="72x72"/>
<link href="/assets/images/seo/apple-touch-icon-144x144-precomposed.png" rel="apple-touch-icon-precomposed" sizes="144x144"/>
<link href="/assets/images/seo/apple-touch-icon-60x60-precomposed.png" rel="apple-touch-icon-precomposed" sizes="60x60"/>
<link href="/assets/images/seo/apple-touch-icon-120x120-precomposed.png" rel="apple-touch-icon-precomposed" sizes="120x120"/>
<link href="/assets/images/seo/apple-touch-icon-76x76-precomposed.png" rel="apple-touch-icon-precomposed" sizes="76x76"/>
<link href="/assets/images/seo/apple-touch-icon-152x152-precomposed.png" rel="apple-touch-icon-precomposed" sizes="152x152"/>
<link href="/assets/images/seo/apple-touch-icon-precomposed.png" rel="apple-touch-icon-precomposed"/>
<script src="https://www.googleoptimize.com/optimize.js?id=OPT-K4CCSKT"></script>
<!-- To make possible for marketing partners to capture utm tags -->
<!-- For more info, check: https://developers.google.com/web/updates/2020/07/referrer-policy-new-chrome-default -->
<script charset="UTF-8" src="//cdn.segmentify.com/7a036bce-712d-4354-b5aa-2e74724eee62/segmentify.js"></script>
<meta content="no-referrer-when-downgrade" name="referrer"/>
<script src="env.js"></script>
<script async="" src="https://securepubads.g.doubleclick.net/tag/js/gpt.js"></script>
<link href="styles.0fb2916d96e6de1ada0e.css" rel="stylesheet"/></head>
<body class="mat-typography">
<sm-root></sm-root>
<script defer="" src="runtime.8f34fe4ebfd1caac7fe2.js"></script><script defer="" src="polyfills.50ee7712f0e1700f8bcd.js"></script><script defer="" src="scripts.8129d19be5ffb616004f.js"></script><script defer="" src="main.3fec2e7e78a880c219a7.js"></script></body>
</html>
您看到的数据是通过Javascript加载的。您可以使用 requests/json 模块加载它：
import json
import requests
url = "https://www.migros.com.tr/rest/search/screens/temel-gida-c-2?sayfa=1"
data = requests.get(url).json()
# uncomment this to print all data:
# print(json.dumps(data, indent=4))
for p in data["data"]["searchInfo"]["storeProductInfos"]:
print(
"{:<35} {:<10} {:<10}".format(
p["name"], p["regularPrice"], p["salePrice"]
)
)
打印：
Maydanoz Adet 445 445
Soğan Kuru Dökme Kg 195 195
Migros Havuç Beypazarı Paket Kg 875 750
Domates Kg 1495 1495
Kabak Sakız Kg 1990 1990
Dereotu Adet 930 930
Roka Demet 603 603
Salata Kıvırcık Adet 1090 1090
Patlıcan Kemer Kg 1990 1990
Soğan Taze Demet 925 925
Hıyar Kg 1790 1790
Domates Salkım Kg 2290 2290
Biber Kırmızı Kg 2190 2190
Brokoli Kg 3450 3450
Atom Salata Adet 1206 1206
Kereviz Kg 875 875
Karnabahar Kg 1390 1390
Ispanak Kg 1450 1450
Patates Taze Kg 556 556
Biber Köy Usulü Kg 2990 2990
Nane Adet 631 631
Biber Sivri Kg 2690 2690
Pırasa Kg 930 930
Lahana Beyaz Kg 595 595
Biber Dolmalık Kg 2702 2702
Domates Şeker 250 G 837 837
Lahana Kırmızı Kg 1206 1206
Patates Ekonomik Boy File Kg 445 445
Pancar Kg 743 743
Domates Pembe Kg 1850 1850
我想用下面的代码 beautifulsoup 写 this site 上的产品价格，但是当我写代码时，列表 returns 是空的。
import requests as req from bs4 import BeautifulSoup as bs
url = "https://www.migros.com.tr/temel-gida-c-2?sayfa=1"
headers = { 'User-Agent': ( 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) ' 'AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36' ) }
urunler = []
rx = req.get(url, headers=headers) sympo = bs(rx.text, 'html.parser') soup = bs(rx.content, 'lxml') print(sympo.findAll('span', {'class': 'amount'}))
print(sympo.findAll('span', {'class': 'amount'}))
[]
页面由 JS 调用生成，因此您可能需要查看 selenium 包。如果你只是在打印 soup 时检查输出，你只会得到页面的 header 内容：
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<title>Migros Sanal Market: Online Market Alışverişi</title>
<base href="/"/>
<meta content="width=device-width, initial-scale=1, maximum-scale=5, user-scalable=0" name="viewport"/>
<link href="manifest.json" rel="manifest"/>
<link href="/assets/favicon.ico" rel="icon" type="image/x-icon"/>
<meta content="#ff9933" name="theme-color"/>
<meta content="Geniş ürün yelpazesi, en uygun fiyatlı ürünleri ve aynı gün teslimat fırsatıyla online market alışverişinizde Migros Sanal Market size iyi gelecek!" name="description" property="og:description"/>
<meta content="Migros Sanal Market: Online Market Alışverişi" property="og:site_name"/>
<meta content="https://migros-dali-storage-prod.global.ssl.fastly.net/sanalmarket/custom/sanalmarket-seo-34706362.png" property="og:image"/>
<link href="/assets/images/seo/apple-touch-icon-57x57.png" rel="apple-touch-icon" sizes="57x57"/>
<link href="/assets/images/seo/apple-touch-icon-114x114.png" rel="apple-touch-icon" sizes="114x114"/>
<link href="/assets/images/seo/apple-touch-icon-72x72.png" rel="apple-touch-icon" sizes="72x72"/>
<link href="/assets/images/seo/apple-touch-icon-144x144.png" rel="apple-touch-icon" sizes="144x144"/>
<link href="/assets/images/seo/apple-touch-icon-60x60.png" rel="apple-touch-icon" sizes="60x60"/>
<link href="/assets/images/seo/apple-touch-icon-120x120.png" rel="apple-touch-icon" sizes="120x120"/>
<link href="/assets/images/seo/apple-touch-icon-76x76.png" rel="apple-touch-icon" sizes="76x76"/>
<link href="/assets/images/seo/apple-touch-icon-152x152.png" rel="apple-touch-icon" sizes="152x152"/>
<link href="/assets/images/seo/apple-touch-icon.png" rel="apple-touch-icon"/>
<link href="/assets/images/seo/apple-touch-icon-57x57-precomposed.png" rel="apple-touch-icon-precomposed" sizes="57x57"/>
<link href="/assets/images/seo/apple-touch-icon-114x114-precomposed.png" rel="apple-touch-icon-precomposed" sizes="114x114"/>
<link href="/assets/images/seo/apple-touch-icon-72x72-precomposed.png" rel="apple-touch-icon-precomposed" sizes="72x72"/>
<link href="/assets/images/seo/apple-touch-icon-144x144-precomposed.png" rel="apple-touch-icon-precomposed" sizes="144x144"/>
<link href="/assets/images/seo/apple-touch-icon-60x60-precomposed.png" rel="apple-touch-icon-precomposed" sizes="60x60"/>
<link href="/assets/images/seo/apple-touch-icon-120x120-precomposed.png" rel="apple-touch-icon-precomposed" sizes="120x120"/>
<link href="/assets/images/seo/apple-touch-icon-76x76-precomposed.png" rel="apple-touch-icon-precomposed" sizes="76x76"/>
<link href="/assets/images/seo/apple-touch-icon-152x152-precomposed.png" rel="apple-touch-icon-precomposed" sizes="152x152"/>
<link href="/assets/images/seo/apple-touch-icon-precomposed.png" rel="apple-touch-icon-precomposed"/>
<script src="https://www.googleoptimize.com/optimize.js?id=OPT-K4CCSKT"></script>
<!-- To make possible for marketing partners to capture utm tags -->
<!-- For more info, check: https://developers.google.com/web/updates/2020/07/referrer-policy-new-chrome-default -->
<script charset="UTF-8" src="//cdn.segmentify.com/7a036bce-712d-4354-b5aa-2e74724eee62/segmentify.js"></script>
<meta content="no-referrer-when-downgrade" name="referrer"/>
<script src="env.js"></script>
<script async="" src="https://securepubads.g.doubleclick.net/tag/js/gpt.js"></script>
<link href="styles.0fb2916d96e6de1ada0e.css" rel="stylesheet"/></head>
<body class="mat-typography">
<sm-root></sm-root>
<script defer="" src="runtime.8f34fe4ebfd1caac7fe2.js"></script><script defer="" src="polyfills.50ee7712f0e1700f8bcd.js"></script><script defer="" src="scripts.8129d19be5ffb616004f.js"></script><script defer="" src="main.3fec2e7e78a880c219a7.js"></script></body>
</html>
您看到的数据是通过Javascript加载的。您可以使用 requests/json 模块加载它：
import json
import requests
url = "https://www.migros.com.tr/rest/search/screens/temel-gida-c-2?sayfa=1"
data = requests.get(url).json()
# uncomment this to print all data:
# print(json.dumps(data, indent=4))
for p in data["data"]["searchInfo"]["storeProductInfos"]:
print(
"{:<35} {:<10} {:<10}".format(
p["name"], p["regularPrice"], p["salePrice"]
)
)
打印：
Maydanoz Adet 445 445
Soğan Kuru Dökme Kg 195 195
Migros Havuç Beypazarı Paket Kg 875 750
Domates Kg 1495 1495
Kabak Sakız Kg 1990 1990
Dereotu Adet 930 930
Roka Demet 603 603
Salata Kıvırcık Adet 1090 1090
Patlıcan Kemer Kg 1990 1990
Soğan Taze Demet 925 925
Hıyar Kg 1790 1790
Domates Salkım Kg 2290 2290
Biber Kırmızı Kg 2190 2190
Brokoli Kg 3450 3450
Atom Salata Adet 1206 1206
Kereviz Kg 875 875
Karnabahar Kg 1390 1390
Ispanak Kg 1450 1450
Patates Taze Kg 556 556
Biber Köy Usulü Kg 2990 2990
Nane Adet 631 631
Biber Sivri Kg 2690 2690
Pırasa Kg 930 930
Lahana Beyaz Kg 595 595
Biber Dolmalık Kg 2702 2702
Domates Şeker 250 G 837 837
Lahana Kırmızı Kg 1206 1206
Patates Ekonomik Boy File Kg 445 445
Pancar Kg 743 743
Domates Pembe Kg 1850 1850