为什么这个数据无法解析到 Beautiful Soup 的 find 方法中？

Question

我一直在关注这个教程：https://kb.objectrocket.com/postgresql/scrape-a-website-to-postgres-with-python-938

我的 app.py 文件如下所示（取自上述教程）：

from flask import Flask  # needed for flask-dependent libraries below
from flask import render_template  # to render the error page
from selenium import webdriver  # to grab source from URL
from bs4 import BeautifulSoup  # for searching through HTML
import psycopg2  # for database access

# set up Postgres database connection and cursor.
t_host = "localhost" # either "localhost", a domain name, or an IP address.
t_port = "5432" # default postgres port
t_dbname = "scrape"
t_user = "postgres"
t_pw = "********"
db_conn = psycopg2.connect(host=t_host, port=t_port, dbname=t_dbname, user=t_user, password=t_pw)
db_cursor = db_conn.cursor()

app = Flask(__name__)


@app.route("/")
@app.route('/import_temp')
def import_temp():
    # set up your webdriver to use Chrome web browser
    my_web_driver = webdriver.Chrome("/usr/local/bin/chromedriver")

    # designate the URL we want to scrape
    #   NOTE: the long string of characters at the end of this URL below is a clue that
    #   maybe this page is so dynamic, like maybe refers to a specific web session and/or day/time,
    #   that we can't necessarily count on it to be the same more than one time.
    #   Which means... we may want to find another source for our data; one that is more
    #   dependable. That said, whatever URL you use, the methodology in this lesson stands.
    t_url = "https://weather.com/weather/today/l/7ebb344012f0c5ff88820d763da89ed94306a86c770fda50c983bf01a0f55c0d"
    # initiate scrape of website page data
    my_web_driver.get(t_url)
    # return entire page into "t_content"
    t_content = my_web_driver.page_source
    # use soup to make page content easily searchable
    soup_in_bowl = BeautifulSoup(t_content, 'html.parser')
    # search for the UNIQUE span and class for the data we are looking for:
    o_temp = soup_in_bowl.find('span', attrs={'class': 'deg-feels'})
    # from the resulting object, "o_temp", get the text parameter and assign it to "n_temp"
    n_temp = o_temp.text

    # Build SQL for purpose of:
    #    saving the temperature data to a new row
    s = ""
    s += "INSERT INTO tbl_temperatures"
    s += "("
    s += "n_temp"
    s += ") VALUES ("
    s += "(%n_temp)"
    s += ")"

    # Trap errors for opening the file
    try:
        db_cursor.execute(s, [n_temp, n_temp])
        db_conn.commit()
    except psycopg2.Error as e:
        t_msg = "Database error: " + e + "/n open() SQL: " + s
        return render_template("error_page.html", t_msg = t_msg)

    # Success!
    # Show a message to user.
    t_msg = "Successful scrape!"
    return render_template("progress.html", t_msg = t_msg)

    # Clean up the cursor and connection objects
    db_cursor.close()
    db_conn.close()

我已经成功导入了所需的库，并创建了一个数据库来存储抓取的数据。但是，当我运行应用程序时，我现在收到一个错误：

FLASK_APP = app.py
FLASK_ENV = development
FLASK_DEBUG = 0
In folder /home/lloyd/PycharmProjects/flaskProject
/home/lloyd/PycharmProjects/flaskProject/venv/bin/python -m flask run
 * Serving Flask app 'app.py' (lazy loading)
 * Environment: development
 * Debug mode: off
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)
[2021-12-29 20:51:32,817] ERROR in app: Exception on / [GET]
Traceback (most recent call last):
  File "/home/lloyd/PycharmProjects/flaskProject/venv/lib/python3.6/site-packages/flask/app.py", line 2073, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/lloyd/PycharmProjects/flaskProject/venv/lib/python3.6/site-packages/flask/app.py", line 1518, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/lloyd/PycharmProjects/flaskProject/venv/lib/python3.6/site-packages/flask/app.py", line 1516, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/lloyd/PycharmProjects/flaskProject/venv/lib/python3.6/site-packages/flask/app.py", line 1502, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  File "/home/lloyd/PycharmProjects/flaskProject/app.py", line 41, in import_temp
    n_temp = o_temp.text
AttributeError: 'NoneType' object has no attribute 'text'
127.0.0.1 - - [29/Dec/2021 20:51:32] "GET / HTTP/1.1" 500 -

我认为该错误表明传递给 BeautifulSoup 方法的数据不正确：Line 41, in import_temp n_temp = o_temp.text

如果有人能告诉我这个异常的原因，以及如何解决它，将不胜感激。

谢谢

Answer 1

您需要在浏览器开发工具中查看网站结构。这是它在这个特定网站上的样子：

<div
  data-testid="FeelsLikeSection"
  class="TodayDetailsCard--feelsLikeTemp--3fwAJ"
>
  <span
    data-testid="TemperatureValue"
    class="TodayDetailsCard--feelsLikeTempValue--Cf9Sl"
    >54°</span
  ><span
    data-testid="FeelsLikeLabel"
    class="TodayDetailsCard--feelsLikeTempLabel--3v2vt"
    >Feels Like</span
  >
</div>

具有真实感觉温度的 <span> 有一个唯一的 class 名称，但是这个 class 名称是自动生成的（可能由 Styled Components 之类的东西生成）并且可能会更改每个网站重建的时间，所以你不应该依赖它。

另一方面，封闭的 <div> 可以通过 data-testid 属性来区分，该属性由网站开发人员放置在那里用于他们的自动化测试。这不太可能改变，因为它会破坏他们的测试。

带温度的<span>也有data-testid属性，但是本站有很多<span>属性值相同，所以按[=18查询=]不会给你一个独特的元素。

所以你可以像这样做一个二级查询：

o_temp_feel = soup_in_bowl.find('div', attrs={'data-testid': 'FeelsLikeSection'})
o_temp = o_temp_feel.find('span', attrs={'data-testid': 'TemperatureValue'})
# from the resulting object, "o_temp", get the text parameter and assign it to "n_temp"
n_temp = o_temp.text

请注意，n_temp 将类似于“54°”，因此如果您需要数值，则需要在转换为整数之前将其截断。

为什么这个数据无法解析到 Beautiful Soup 的 find 方法中？

Why is this data unable to be parsed into the Beautiful Soup find method?

python

postgresql

beautifulsoup

flask