网页标题打印为None、BeautifulSoup

Question

我正在尝试从 this website 抓取数据，但无法获取网页标题。

我的代码-

import requests
from bs4 import BeautifulSoup

base_url = "https://www.stfrancismedicalcenter.com/find-a-provider/"

content = requests.get(url = base_url).content
soup = BeautifulSoup(content, "html.parser")

profile_link = soup.find("a", {"class": "flex-top-between-block-500"}).get("href")
profile_url = base_url + profile_link[1:]

profile_content = requests.get(url = profile_url).content
profile_soup = BeautifulSoup(profile_content, "html.parser")
print(profile_soup.title.string)

这是我得到的输出。

[Running] python -u "d:\Personal\CS\Web Scrapping\first.py"
None

[Done] exited with code=0 in 3.592 seconds

我想就此提出一些建议！

Answer 1

这里的问题是配置文件的连接路径不正确，find-a-provider 部分重复，因此变成：

https://www.stfrancismedicalcenter.com/find-a-provider//find-a-provider/adegbenga-a-adetola-md/

而是使用您的 url 定义特定的“”baseUrl:

profile_url = 'https://www.stfrancismedicalcenter.com' + profile_link

或

baseUrl = 'https://www.stfrancismedicalcenter.com'
profile_url =  baseUrl + profile_link

例子

import requests
from bs4 import BeautifulSoup

url = "https://www.stfrancismedicalcenter.com/find-a-provider"
baseUrl = 'https://www.stfrancismedicalcenter.com'

content = requests.get(url).content
soup = BeautifulSoup(content, "html.parser")

profile_link = soup.find("a", {"class": "flex-top-between-block-500"}).get("href")
profile_url = baseUrl + profile_link

profile_content = requests.get(url = profile_url).content
profile_soup = BeautifulSoup(profile_content, "html.parser")
profile_soup.title.text

输出

Adegbenga A. Adetola MD

网页标题打印为None、BeautifulSoup

Title of webpage printing as None, BeautifulSoup

python

beautifulsoup

web-scraping

例子

输出