为什么我的爬虫 return none 和 beautifulsoup？

Question

我正在尝试从该网站提取 table 每个州的太阳能价格 https://www.solarreviews.com/solar-panels/solar-panel-cost/ 并将其传递到 CSV 文件中。这是到目前为止的代码。它正在返回 'None'。有人可以向我解释为什么吗？怎么办？我将不胜感激。

from bs4 import BeautifulSoup as soup
import csv

#Open page and grab HTML
my_url = ('https://www.solarreviews.com/solar-panels/solar-panel-cost/')
uClient = uReq(my_url)
page_html = uClient.read()
uClient.close()

#HTML parser
page_soup = soup(page_html, 'html.parser')

#Find table
ele_table = page_soup.find('table',{"class":"table table-hover stateAveragesTable"})

print(ele_table)```

Answer 1

您需要更改 tr 样式属性以便全部显示。请注意，如果您发出太多请求，您也有被 IP 封禁的风险。

import requests,re
from bs4 import BeautifulSoup as bs
import pandas as pd

r = requests.get('https://www.solarreviews.com/solar-panels/solar-panel-cost/')
soup = bs(r.content, 'lxml')
table_html = str(soup.select_one('.stateAveragesTable'))
table_html = re.sub('display: none;','', table_html)
print(pd.read_html(str(table_html)))

为什么我的爬虫 return none 和 beautifulsoup？

Why does my scraper return none with beautifulsoup?

html

python

beautifulsoup

web-scraping

python-3.7