使用 Python 进行网页抓取时如何分隔列和格式化日期？
How to separate columns and format date when web scraping by using Python?
我正在尝试通过使用 Python3 将本网站的图表转换为 .csv 文件来进行网络抓取：2013-14 NBA National TV Schedule
图表开头如下：
Game/Time Network Matchup
Oct. 29, 8 p.m. ET TNT Chicago vs. Miami
Oct. 29, 10:30 p.m. ET TNT LA Clippers vs. LA Lakers
我正在使用这些软件包：
import re
import requests
import pandas as pd
from bs4 import BeautifulSoup
from itertools import groupby
我通过以下方式导入数据：
pd.read_html("https://www.sbnation.com/2013/8/6/4595688/2013-14-nba-national-tv-schedule")[0]
输出样本为：
0 1 2
0 Game/Time Network Matchup
1 Oct. 29, 8 p.m. ET TNT Chicago vs. Miami
2 Oct. 29, 10:30 p.m. ET TNT LA Clippers vs. LA Lakers
我想要的 .csv 文件输出如下所示：
我不确定如何将 game/time 分成单独的列。请注意日期的格式如何设置为 10/29/13。我也不确定如何将比赛分为客场（第一队）和主场（第二队）到不同的列。我知道应该使用 pd.to_datetime 和 str.split()。我如何实现抓取器来获得此输出？
这是我的看法：
df = pd.read_html("https://www.sbnation.com/2013/8/6/4595688/2013-14-nba-national-tv-schedule")[0]
# set the correct column names
df = df.T.set_index([0]).T
# separate date and time
datetime = df['Game/Time'].str.extract('(?P<Date>.*), (?P<Time>.*) ET$')
# extract Home and Away
home_away = df['Matchup'].str.extract('^(?P<Away>.*) vs\. (?P<Home>.*)$')
# join the data
final_df = pd.concat([datetime, home_away, df[['Network']]], axis=1)
输出：
Date Time Away Home Network
1 Oct. 29 8 p.m. Chicago Miami TNT
2 Oct. 29 10:30 p.m. LA Clippers LA Lakers TNT
3 Oct. 31 8 p.m. New York Chicago TNT
4 Oct. 31 10:30 p.m. Golden State LA Clippers TNT
5 Nov. 1 8 p.m. Miami Brooklyn ESPN
.. ... ... ... ... ...
141 Apr. 13 1 p.m. Chicago New York ABC
142 Apr. 15 8 p.m. New York Brooklyn TNT
143 Apr. 15 10:30 p.m. Denver LA Clippers TNT
144 Apr. 16 8 p.m. Atlanta Milwaukee ESPN
145 Apr. 16 10:30 p.m. Golden State Denver ESPN
您可以使用 regex 拆分您的列，您的 time 具有不同的格式，因此我们可以通过使用特定格式并将错误强制转换为 NaT 值来处理这些列。
df = pd.read_html("https://www.sbnation.com/2013/8/6/4595688/2013-14-nba-national-tv-schedule")[0]
# set column
df.columns = df.iloc[0]
df = df.iloc[1:].reset_index(drop=True)
#set date and time column.
df['date'] = pd.to_datetime((df['Game/Time'].str.split(',',expand=True)[0] + ' 2019')
,format='%b. %d %Y')
df['time'] = df['Game/Time'].str.split(',',expand=True)[1]
#time column has different formats, lets handle those.
s = pd.to_datetime(df['time'].str.strip('ET').str.replace('\.','').str.strip(),
format='%H %p',errors='coerce')
s = s.fillna(pd.to_datetime(df['time'].str.strip('ET').str.replace('\.','').str.strip(),
format='%H:%M %p',errors='coerce'))
df['time'] = s.dt.time
#home and away columns.
df['home'] = df['Matchup'].str.extract('(.*)vs(.*)')[0].str.strip()
df['away'] = df['Matchup'].str.extract('(.*)vs(.*)')[1].str.strip('.')
# slice dataframe.
df2 = df[['date','time','home','away','Network']]
print(df2)
0 date time home away Network
0 2019-10-29 08:00:00 Chicago Miami TNT
1 2019-10-29 10:30:00 LA Clippers LA Lakers TNT
2 2019-10-31 08:00:00 New York Chicago TNT
3 2019-10-31 10:30:00 Golden State LA Clippers TNT
4 2019-11-01 08:00:00 Miami Brooklyn ESPN
.. ... ... ... ... ...
140 2019-04-13 01:00:00 Chicago New York ABC
141 2019-04-15 08:00:00 New York Brooklyn TNT
142 2019-04-15 10:30:00 Denver LA Clippers TNT
143 2019-04-16 08:00:00 Atlanta Milwaukee ESPN
144 2019-04-16 10:30:00 Golden State Denver ESPN
df['Date']=df['Date'].dt.strftime('%m/%d/%Y')
这一行应该可以帮助您按照您想要的方式格式化日期
import pandas as pd
import numpy as np
df = pd.read_html("https://www.sbnation.com/2013/8/6/4595688/2013-14-nba-national-tv-schedule",header=0)[0]
df['Date']=df['Game/Time'].str.extract(r'(.*),',expand=True)
df['Time']=df['Game/Time'].str.extract(r',(.*) ET',expand=True)
df['Time']=df['Time'].str.replace('p.m.','PM')
df['Date'] = np.where(df.Date.str.startswith(('10/', 11/', '12/')), df.Date + ' 13', df.Date + ' 14')
df['Date']=pd.to_datetime(df['Date'])
df['Date']=df['Date'].dt.strftime('%m/%d/%Y')
df['Home'] = df['Matchup'].str.extract('(.*)vs')
df['Away'] = df['Matchup'].str.extract('vs.(.*)')
df = df.drop(columns=['Game/Time','Matchup'])
print(df)
Network Date Time Home Away
0 TNT 10/29/2013 8 PM Chicago Miami
1 TNT 10/29/2013 10:30 PM LA Clippers LA Lakers
2 TNT 10/31/2013 8 PM New York Chicago
3 TNT 10/31/2013 10:30 PM Golden State LA Clippers
4 ESPN 11/01/2013 8 PM Miami Brooklyn
希望这就是您要找的。
我正在尝试通过使用 Python3 将本网站的图表转换为 .csv 文件来进行网络抓取：2013-14 NBA National TV Schedule
图表开头如下：
Game/Time Network Matchup
Oct. 29, 8 p.m. ET TNT Chicago vs. Miami
Oct. 29, 10:30 p.m. ET TNT LA Clippers vs. LA Lakers
我正在使用这些软件包：
import re
import requests
import pandas as pd
from bs4 import BeautifulSoup
from itertools import groupby
我通过以下方式导入数据：
pd.read_html("https://www.sbnation.com/2013/8/6/4595688/2013-14-nba-national-tv-schedule")[0]
输出样本为：
0 1 2
0 Game/Time Network Matchup
1 Oct. 29, 8 p.m. ET TNT Chicago vs. Miami
2 Oct. 29, 10:30 p.m. ET TNT LA Clippers vs. LA Lakers
我想要的 .csv 文件输出如下所示：
我不确定如何将 game/time 分成单独的列。请注意日期的格式如何设置为 10/29/13。我也不确定如何将比赛分为客场（第一队）和主场（第二队）到不同的列。我知道应该使用 pd.to_datetime 和 str.split()。我如何实现抓取器来获得此输出？
这是我的看法：
df = pd.read_html("https://www.sbnation.com/2013/8/6/4595688/2013-14-nba-national-tv-schedule")[0]
# set the correct column names
df = df.T.set_index([0]).T
# separate date and time
datetime = df['Game/Time'].str.extract('(?P<Date>.*), (?P<Time>.*) ET$')
# extract Home and Away
home_away = df['Matchup'].str.extract('^(?P<Away>.*) vs\. (?P<Home>.*)$')
# join the data
final_df = pd.concat([datetime, home_away, df[['Network']]], axis=1)
输出：
Date Time Away Home Network
1 Oct. 29 8 p.m. Chicago Miami TNT
2 Oct. 29 10:30 p.m. LA Clippers LA Lakers TNT
3 Oct. 31 8 p.m. New York Chicago TNT
4 Oct. 31 10:30 p.m. Golden State LA Clippers TNT
5 Nov. 1 8 p.m. Miami Brooklyn ESPN
.. ... ... ... ... ...
141 Apr. 13 1 p.m. Chicago New York ABC
142 Apr. 15 8 p.m. New York Brooklyn TNT
143 Apr. 15 10:30 p.m. Denver LA Clippers TNT
144 Apr. 16 8 p.m. Atlanta Milwaukee ESPN
145 Apr. 16 10:30 p.m. Golden State Denver ESPN
您可以使用 regex 拆分您的列，您的 time 具有不同的格式，因此我们可以通过使用特定格式并将错误强制转换为 NaT 值来处理这些列。
df = pd.read_html("https://www.sbnation.com/2013/8/6/4595688/2013-14-nba-national-tv-schedule")[0]
# set column
df.columns = df.iloc[0]
df = df.iloc[1:].reset_index(drop=True)
#set date and time column.
df['date'] = pd.to_datetime((df['Game/Time'].str.split(',',expand=True)[0] + ' 2019')
,format='%b. %d %Y')
df['time'] = df['Game/Time'].str.split(',',expand=True)[1]
#time column has different formats, lets handle those.
s = pd.to_datetime(df['time'].str.strip('ET').str.replace('\.','').str.strip(),
format='%H %p',errors='coerce')
s = s.fillna(pd.to_datetime(df['time'].str.strip('ET').str.replace('\.','').str.strip(),
format='%H:%M %p',errors='coerce'))
df['time'] = s.dt.time
#home and away columns.
df['home'] = df['Matchup'].str.extract('(.*)vs(.*)')[0].str.strip()
df['away'] = df['Matchup'].str.extract('(.*)vs(.*)')[1].str.strip('.')
# slice dataframe.
df2 = df[['date','time','home','away','Network']]
print(df2)
0 date time home away Network
0 2019-10-29 08:00:00 Chicago Miami TNT
1 2019-10-29 10:30:00 LA Clippers LA Lakers TNT
2 2019-10-31 08:00:00 New York Chicago TNT
3 2019-10-31 10:30:00 Golden State LA Clippers TNT
4 2019-11-01 08:00:00 Miami Brooklyn ESPN
.. ... ... ... ... ...
140 2019-04-13 01:00:00 Chicago New York ABC
141 2019-04-15 08:00:00 New York Brooklyn TNT
142 2019-04-15 10:30:00 Denver LA Clippers TNT
143 2019-04-16 08:00:00 Atlanta Milwaukee ESPN
144 2019-04-16 10:30:00 Golden State Denver ESPN
df['Date']=df['Date'].dt.strftime('%m/%d/%Y')
这一行应该可以帮助您按照您想要的方式格式化日期
import pandas as pd
import numpy as np
df = pd.read_html("https://www.sbnation.com/2013/8/6/4595688/2013-14-nba-national-tv-schedule",header=0)[0]
df['Date']=df['Game/Time'].str.extract(r'(.*),',expand=True)
df['Time']=df['Game/Time'].str.extract(r',(.*) ET',expand=True)
df['Time']=df['Time'].str.replace('p.m.','PM')
df['Date'] = np.where(df.Date.str.startswith(('10/', 11/', '12/')), df.Date + ' 13', df.Date + ' 14')
df['Date']=pd.to_datetime(df['Date'])
df['Date']=df['Date'].dt.strftime('%m/%d/%Y')
df['Home'] = df['Matchup'].str.extract('(.*)vs')
df['Away'] = df['Matchup'].str.extract('vs.(.*)')
df = df.drop(columns=['Game/Time','Matchup'])
print(df)
Network Date Time Home Away
0 TNT 10/29/2013 8 PM Chicago Miami
1 TNT 10/29/2013 10:30 PM LA Clippers LA Lakers
2 TNT 10/31/2013 8 PM New York Chicago
3 TNT 10/31/2013 10:30 PM Golden State LA Clippers
4 ESPN 11/01/2013 8 PM Miami Brooklyn
希望这就是您要找的。