使用 BeautifulSoup 和请求获取 XML 的特定标签

Question

我试图抓取这个 XML 页面，但我无法获取某些标签的内容。掉下来的是可以的，其他的不行

这是我要抓取的页面：https://g1.globo.com/rss/g1/

我正在尝试获取 'pubDate' 标签，当我尝试 find_all 时它返回为空，当我尝试使用 find 时，它返回为 None。这是我的代码。我尝试了很多方法，但都失败了。

rss_globo = requests.get("https://g1.globo.com/rss/g1/").content
bs_globo = BeautifulSoup(rss_globo, 'lxml')
data = bs_globo.find_all('item')
for i in data:
    date = i.find('pubDate').getText()
    print(date)

Answer 1

为了使用 xml，您需要 feature 而不是 parser。

方法如下：

import requests
from bs4 import BeautifulSoup

bs_globo = BeautifulSoup(
    requests.get("https://g1.globo.com/rss/g1/").content,
    features="xml",
)
for i in bs_globo.find_all('item'):
    print(i.find('pubDate').getText())

输出：

Mon, 07 Mar 2022 15:05:42 -0000
Mon, 07 Mar 2022 15:04:53 -0000
Mon, 07 Mar 2022 15:04:41 -0000
Mon, 07 Mar 2022 15:03:38 -0000
Mon, 07 Mar 2022 15:03:15 -0000
Mon, 07 Mar 2022 15:01:14 -0000
Mon, 07 Mar 2022 15:00:37 -0000
Mon, 07 Mar 2022 15:00:26 -0000
Mon, 07 Mar 2022 15:00:09 -0000
Mon, 07 Mar 2022 15:00:04 -0000
Mon, 07 Mar 2022 14:59:32 -0000
Mon, 07 Mar 2022 14:58:46 -0000
Mon, 07 Mar 2022 14:58:04 -0000
Mon, 07 Mar 2022 14:58:02 -0000
Mon, 07 Mar 2022 14:55:24 -0000
Mon, 07 Mar 2022 14:51:20 -0000
Mon, 07 Mar 2022 14:50:45 -0000
Mon, 07 Mar 2022 14:50:22 -0000
Mon, 07 Mar 2022 14:50:07 -0000
Mon, 07 Mar 2022 14:49:01 -0000
Mon, 07 Mar 2022 14:47:23 -0000
Mon, 07 Mar 2022 14:47:21 -0000
Mon, 07 Mar 2022 14:46:34 -0000
Mon, 07 Mar 2022 14:46:31 -0000
Mon, 07 Mar 2022 14:45:45 -0000
Mon, 07 Mar 2022 14:45:02 -0000
Mon, 07 Mar 2022 14:44:37 -0000
Mon, 07 Mar 2022 14:44:16 -0000
Mon, 07 Mar 2022 14:43:37 -0000
Mon, 07 Mar 2022 14:42:56 -0000
Mon, 07 Mar 2022 14:42:39 -0000
Mon, 07 Mar 2022 14:42:16 -0000
Mon, 07 Mar 2022 14:41:51 -0000
Mon, 07 Mar 2022 14:41:41 -0000
Mon, 07 Mar 2022 14:41:35 -0000
Mon, 07 Mar 2022 14:41:09 -0000
Mon, 07 Mar 2022 14:40:38 -0000
Mon, 07 Mar 2022 14:39:27 -0000
Mon, 07 Mar 2022 14:39:15 -0000
Mon, 07 Mar 2022 14:39:13 -0000

Answer 2

我想指出 python 标准库也有用于处理 XML 的工具，即 xml.etree.ElementTree 您可以按如下方式使用它

import xml.etree.ElementTree as ET
import requests
root = ET.fromstring(requests.get("https://g1.globo.com/rss/g1/").text)
for pubDate in root.findall("*/item/pubDate"):
    print(pubDate.text)

输出

Mon, 07 Mar 2022 15:28:38 -0000
Mon, 07 Mar 2022 15:27:46 -0000
Mon, 07 Mar 2022 15:27:17 -0000
Mon, 07 Mar 2022 15:26:41 -0000
Mon, 07 Mar 2022 15:24:59 -0000
Mon, 07 Mar 2022 15:24:36 -0000
Mon, 07 Mar 2022 15:24:22 -0000
Mon, 07 Mar 2022 15:23:53 -0000
Mon, 07 Mar 2022 15:23:28 -0000
Mon, 07 Mar 2022 15:22:35 -0000
Mon, 07 Mar 2022 15:22:34 -0000
Mon, 07 Mar 2022 15:20:14 -0000
Mon, 07 Mar 2022 15:17:29 -0000
Mon, 07 Mar 2022 15:16:49 -0000
Mon, 07 Mar 2022 15:16:30 -0000
Mon, 07 Mar 2022 15:15:49 -0000
Mon, 07 Mar 2022 15:15:16 -0000
Mon, 07 Mar 2022 15:12:35 -0000
Mon, 07 Mar 2022 15:12:27 -0000
Mon, 07 Mar 2022 15:10:54 -0000
Mon, 07 Mar 2022 15:10:41 -0000
Mon, 07 Mar 2022 15:10:37 -0000
Mon, 07 Mar 2022 15:09:12 -0000
Mon, 07 Mar 2022 15:08:45 -0000
Mon, 07 Mar 2022 15:07:46 -0000
Mon, 07 Mar 2022 15:05:42 -0000
Mon, 07 Mar 2022 15:04:53 -0000
Mon, 07 Mar 2022 15:04:41 -0000
Mon, 07 Mar 2022 15:03:38 -0000
Mon, 07 Mar 2022 15:03:15 -0000
Mon, 07 Mar 2022 15:01:14 -0000
Mon, 07 Mar 2022 15:00:37 -0000
Mon, 07 Mar 2022 15:00:26 -0000
Mon, 07 Mar 2022 15:00:09 -0000
Mon, 07 Mar 2022 15:00:04 -0000
Mon, 07 Mar 2022 14:59:32 -0000
Mon, 07 Mar 2022 14:58:46 -0000
Mon, 07 Mar 2022 14:58:04 -0000
Mon, 07 Mar 2022 14:58:02 -0000
Mon, 07 Mar 2022 14:55:24 -0000

*/item/pubDate 是描述您要访问的元素的路径

使用 BeautifulSoup 和请求获取 XML 的特定标签

Get specific tag of a XML with BeautifulSoup and Requests

html

python

xml

beautifulsoup