使用 R 从 Finviz 抓取股票关键统计数据

Question

我试图从 Finviz 中删除一些库存关键统计数据。我应用了原始问题中的代码：。为了收集尽可能多的股票的统计数据，我创建了一个股票代码和描述列表，如下所示：

Symbol Description
A      Agilent Technologies
AAA    Alcoa Corp
AAC    Aac Holdings Inc
BABA   Alibaba Group Holding Ltd
CRM    Salesforce.Com Inc
...

我选择了第一列并将其作为一个字符存储在 R 中，并将其命名为 stocks。然后我应用了代码：

for (s in stocks) {
url <- paste0("http://finviz.com/quote.ashx?t=", s)
webpage <- readLines(url)
html <- htmlTreeParse(webpage, useInternalNodes = TRUE, asText = TRUE)
tableNodes <- getNodeSet(html, "//table")

# ASSIGN TO STOCK NAMED DFS
assign(s, readHTMLTable(tableNodes[[9]], 
                      header= c("data1", "data2", "data3", "data4", "data5", "data6",
                                "data7", "data8", "data9", "data10", "data11", "data12")))

# ADD COLUMN TO IDENTIFY STOCK 
df <- get(s)
df['stock'] <- s
assign(s, df)
}

# COMBINE ALL STOCK DATA 
stockdatalist <- cbind(mget(stocks))
stockdata <- do.call(rbind, stockdatalist)
# MOVE STOCK ID TO FIRST COLUMN
stockdata <- stockdata[, c(ncol(stockdata), 1:ncol(stockdata)-1)]

但是，对于某些股票，Finviz 没有它们的页面，我收到这样的错误提示：

Error in file(con, "r") : cannot open the connection
In addition: Warning message:
In file(con, "r") :
cannot open URL 'http://finviz.com/quote.ashx?t=AGM.A': HTTP status was '404 
Not Found'

有很多股票有这种情况，所以我无法手动将它们从列表中删除。有没有办法跳过获取这些股票的页面？提前致谢！

Answer 1

也许在这些行中？在使用 forloop 之前尝试过滤股票。

    library(tidyverse)

#AGM.A should produce error
    stocks <- c("AXP","BA","CAT","AGM.A")
    urls <- paste0("http://finviz.com/quote.ashx?t=", stocks)

#Test urls with possibly first and find out NAs
    temp_ind <- map(urls, possibly(readLines, otherwise = NA_real_))
    ind <- map_lgl(map(temp_ind, c(1)), is.na)
    ind <- which(ind == TRUE)
    filter.stocks <- stocks[-ind]

#AGM.A is removed and you can just insert stocks which work to for loop.
        filter.stocks
    [1] "AXP" "BA"  "CAT"

正如 statxiong 指出的那样url.exist这里是更简单的版本：

library(RCurl)
library(tidyverse)

stocks[map_lgl(urls, url.exists)]

使用 R 从 Finviz 抓取股票关键统计数据

Web scraping of stock key stats from Finviz with R

finance

r

stock

web-scraping