如何使用美丽的汤从下拉菜单中提取？

Question

我想抓取上面突出显示的体育菜单“文本”。 https://ekusports.com/

from urllib.request import urlopen

from bs4 import BeautifulSoup

import requests

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:88.0) Gecko/20100101 Firefox/88.0"}

url = "https://ekusports.com/"

reqs = requests.get(url, headers=headers)

soup = BeautifulSoup(reqs.text, 'html.parser')

website_text = soup.findAll(text = True)

Answer 1

使用端点获取菜单数据。

方法如下：

import requests


headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:97.0) Gecko/20100101 Firefox/97.0",
    "X-Requested-With": "XMLHttpRequest",
}

r = requests.get("https://ekusports.com/services/sportnames.ashx", headers=headers).json()
print("\n".join([s["sportInfo"]["sport_title"] for s in r["sports"]]))

输出：

Baseball
Beach Volleyball
Bratzke Center
Cheerleading
Colonel Club
Cross Country
Dance Team
Development
EKUSports Builds
Football
General
Marketing/Promotions
Men's Basketball
Men's Cross Country
Men's Golf
Men's Tennis
Men's Track and Field
Name/Image/Likeness (NIL)
Soccer
Softball
Spirit Groups
Tickets
Track & Field
Volleyball
Women's Basketball
Women's Cross Country
Women's Golf
Women's Tennis
Women's Track and Field

Answer 2

如果您使用的是 beautiful soup，它可能无法获得 drop-down 菜单，因为它们是使用 JavaScript 呈现的。你需要一个像

这样的抓取工具

Selenium

自动浏览器和 JavaScript 呈现，浏览器 window 每次点击更新等。Selenium 也非常易于使用。

如何使用美丽的汤从下拉菜单中提取？

how to extract from drop down menu using beautiful soup?

python

beautifulsoup

web-scraping