Beautifulsoup 解析错误

Question

我正尝试在 Google Play 上提取有关应用程序的一些信息，但 BeautifulSoup 似乎不起作用。

link 是这样的（比方说）： https://play.google.com/store/apps/details?id=com.cimaxapp.weirdfacts

我的代码：

url = "https://play.google.com/store/apps/details?id=com.cimaxapp.weirdfacts"
r = requests.get(url)
html = r.content
soup = BeautifulSoup(html)
l = soup.find_all("div", { "class" : "document-subtitles"})
print len(l)
0 #How is this 0?! There is clearly a div with that class

我决定全力以赴，也没用：

i = soup.select('html body.no-focus-outline.sidebar-visible.user-has-no-subscription div#wrapper.wrapper.wrapper-with-footer div#body-content.body-content div.outer-container div.inner-container div.main-content div div.details-wrapper.apps.square-cover.id-track-partial-impression.id-deep-link-item div.details-info div.info-container div.info-box-top')
print i

我做错了什么？

Answer 1

您需要通过 提供 User-Agent header:

来伪装成真正的浏览器

import requests
from bs4 import BeautifulSoup

url = "https://play.google.com/store/apps/details?id=com.cimaxapp.weirdfacts"
r = requests.get(url, headers={
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.94 Safari/537.36"
})
html = r.content
soup = BeautifulSoup(html, "html.parser")

title = soup.find(class_="id-app-title").get_text()
rating = soup.select_one(".document-subtitle .star-rating-non-editable-container")["aria-label"].strip()

print(title)
print(rating)

打印标题和当前评分：

Weird Facts
Rated 4.3 stars out of five stars

要获取附加信息字段值，可以使用以下通用函数：

def get_info(soup, text):
    return soup.find("div", class_="title", text=lambda t: t and t.strip() == text).\
        find_next_sibling("div", class_="content").get_text(strip=True)

那么，如果你这样做：

print(get_info(soup, "Size"))
print(get_info(soup, "Developer"))

您将看到打印：

1.4M
Email email@here.com

Beautifulsoup 解析错误

Beautifulsoup parsing error

python

beautifulsoup

html-parsing

web-scraping