Web 抓取循环遍历 R 中的 ID 和年份列表

Question

我正在尝试使用 R 抓取从 2000 年开始的每个 MLB 球员的比赛日志-reference.com。我读了很多有用的东西，但还不够广泛我的目的。例如，Curtis Granderson 的 URL 2016 年游戏日志是 https://www.baseball-reference.com/players/gl.fcgi?id=grandcu01&t=b&year=2016。

如果我有一个球员 ID 和年份列表，我知道我应该能够使用类似于这个按年份获取出席人数的函数以某种方式循环遍历它们：

fetch_attendance <- function(year) {
url <- paste0("http://www.baseball-reference.com/leagues/MLB/", year, 
"-misc.shtml")
data <- readHTMLTable(url, stringsAsFactors = FALSE)
data <- data[[1]]
data$year <- year
data
}

但是，我再次努力创建一个更广泛的函数来完成这项工作。任何帮助深表感谢。谢谢！

Answer 1

要生成 player_id 的列表，您可以执行如下操作：

library(rvest);
scraping_MLB <- read_html("https://www.baseball-reference.com/players/");

player_name1 <- scraping_MLB %>% html_nodes(xpath = '//*[@id="content"]/ul') %>% html_nodes("div")%>% html_nodes("a") %>% html_text()
player_name2 <- lapply(player_name1,function(x)strsplit(x,split = ","))
player_name<- setNames(do.call(rbind.data.frame, player_name2), "Players_Name")

player_id1 <- scraping_MLB %>% html_nodes(xpath = '//*[@id="content"]/ul')%>% html_nodes("div") %>% html_nodes("a") %>% html_attr("href")
player_id <- setNames(as.data.frame(player_id1),"Players_ID")
player_id$Players_ID <- sub("(\/.*\/.*\/)(\w+)(..*)","\2",player_id$Players_ID)

player_df <- cbind(player_name,player_id)
head(player_df)

一旦你有了所有玩家 ID 的列表，你就可以通过概括这个 url https://www.baseball-reference.com/players/gl.fcgi?id=grandcu01&t=b&year=2016.

轻松循环

（编辑说明： 在 OP 澄清后添加了此代码段）
您可以从下面的示例代码开始，并使用 mapply 或其他工具对其进行优化：

#it fetches the data of first four players from player_df for the duration 2000-16
library(rvest);
players_stat = list()
j=1

for (i in 1:nrow(player_df[c(1:4),])){
  for (year in 2000:2016){
    scrapped_page <- read_html(paste0("https://www.baseball-reference.com/players/gl.fcgi?id=",
                                      as.character(player_df$Players_ID[i]),"&t=b&year=",year))
    if (length(html_nodes(scrapped_page, "table")) >=1){
      #scrapped_data <- html_table(html_nodes(scrapped_page, "table")[[1]])
      tab <-html_attrs(html_nodes(scrapped_page, "table"))
      batting_gamelogs<-which(sapply(tab, function(x){x[2]})=="batting_gamelogs")
      scrapped_data <- html_table(html_nodes(scrapped_page, "table")[[batting_gamelogs]], fill=TRUE)
      scrapped_data$Year <- year
      scrapped_data$Players_Name <- player_df$Players_Name[i]

      players_stat[[j]] <- scrapped_data
      names(players_stat)[j] <- as.character(paste0(player_df$Players_ID[i],"_",year))
      j <- j+1
    }
  }
}
players_stat

希望对您有所帮助！

Web 抓取循环遍历 R 中的 ID 和年份列表

Web scraping looping through list of IDs and years in R

xml

loops

r

plyr

web-scraping