为什么 BeautifulSoup 无法从网站提取数据到 csv？

Question

用户 Chrisvdberge 帮助我创建了以下代码：

import pandas as pd
import requests
from bs4 import BeautifulSoup

url_DAX = 'https://www.eurexchange.com/exchange-en/market-data/statistics/market-statistics-online/100!onlineStats?viewType=4&productGroupId=13394&productId=34642&cp=&month=&year=&busDate=20191114'
req = requests.get(url_DAX, verify=False)
html = req.text
soup = BeautifulSoup(html, 'lxml')
df = pd.read_html(str(html))[0]
df.to_csv('results_DAX.csv')
print(df)

url_DOW = 'https://www.cmegroup.com/trading/equity-index/us-index/e-mini-dow_quotes_settlements_futures.html'
req = requests.get(url_DOW, verify=False)
html = req.text
soup = BeautifulSoup(html, 'lxml')
df = pd.read_html(str(html))[0]
df.to_csv('results_DOW.csv')
print(df)

url_NASDAQ = 'https://www.cmegroup.com/trading/equity-index/us-index/e-mini-nasdaq-100_quotes_settlements_futures.html'
req = requests.get(url_NASDAQ, verify=False)
html = req.text
soup = BeautifulSoup(html, 'lxml')
df = pd.read_html(str(html))[0]
df.to_csv('results_NASDAQ.csv')
print(df)

url_CAC = 'https://live.euronext.com/fr/product/index-futures/FCE-DPAR/settlement-prices'
req = requests.get(url_CAC, verify=False)
html = req.text
soup = BeautifulSoup(html, 'lxml')
df = pd.read_html(str(html))[0]
df.to_csv('results_CAC.csv')
print(df)

我得到以下结果：

3 个 .csv 文件被创建：results_DAX.csv（在这里，一切正常，我有我想要的值。）； results_DOW.csv 和 results_NASDAQ.csv（这里，问题是 .csv 文件没有想要的值。我不明白为什么？）
正如您在代码中看到的那样，应该创建 4 个文件，而不仅仅是 3 个。

所以我的问题是：

如何获取 4 个 csv 文件？
如何获取 results_DOW.csv 和 results_NASDAQ.csv 文件中的值？（也许也在 results_CAC.csv 文件中）

感谢您的回答！ :)

Answer 1

试试这个来获取其他网站。最后一个站点有点棘手，因此您需要尝试使用 Selenium：

import pandas as pd
import requests
from bs4 import BeautifulSoup
from datetime import date, timedelta

url_DAX = 'https://www.eurexchange.com/exchange-en/market-data/statistics/market-statistics-online/100!onlineStats?viewType=4&productGroupId=13394&productId=34642&cp=&month=&year=&busDate=20191114'
df = pd.read_html(url_DAX)[0]
df.to_csv('results_DAX.csv')
print(df)



dt = date.today() - timedelta(days=2)
dateParam =  dt.strftime('%m/%d/%Y')


url_DOW = 'https://www.cmegroup.com/CmeWS/mvc/Settlements/Futures/Settlements/318/FUT'
payload = {
'tradeDate': dateParam,
'strategy': 'DEFAULT',
'pageSize': '500',
'_': '1573920502874'}
response = requests.get(url_DOW, params=payload).json()
df = pd.DataFrame(response['settlements'])
df.to_csv('results_DOW.csv')
print(df)


url_NASDAQ = 'https://www.cmegroup.com/CmeWS/mvc/Settlements/Futures/Settlements/146/FUT'
payload = {
'tradeDate': dateParam,
'strategy': 'DEFAULT',
'pageSize': '500',
'_': '1573920650587'}
response = requests.get(url_NASDAQ, params=payload).json()
df = pd.DataFrame(response['settlements'])
df.to_csv('results_NASDAQ.csv')
print(df)

为什么 BeautifulSoup 无法从网站提取数据到 csv？

Why does BeautifulSoup fail to extract data from websites to csv?

python

beautifulsoup

export-to-csv

pandas