CSS 选择器 HTML 与 Scrapy Python

Question

我正在尝试制作一个网络爬虫，以作为个人项目从 Yahoo Finance 中提取一些信息。但是，在雅虎财经的分析页面上，我无法提取特定的值。 HTML 对我来说似乎很复杂，我可以得到一些指导吗？

class yhcrawler(scrapy.Spider):
    name = 'yahoo'
    
    start_urls = [f'https://ca.finance.yahoo.com/quote/{t}/analysis?p={t}' for t in tkrs]
    
    def parse(self, response):
        filename = 'stock_growths.csv'
        
        l = response.css('div#YDC-Col1>div>div>div>div>div>section>table>tbody>tr>td#431::text').extract()
        print(l)

这就是我正在尝试的

l = response.css('div#YDC-Col1>div>div>div>div>div>section>table>tbody>tr>td#431::text').extract()

我得到的结果是空的

2021-04-18 15:12:54 [scrapy.core.engine] DEBUG: Crawled (200) <GET https://ca.finance.yahoo.com/quote/M/analysis?p=M> (referer: None)
[]

我试图获得的值在突出显示的行上，-11.82%

Answer 1

试试这个：

class YahoofinanceSpider(scrapy.Spider):
    name = 'yahoofinance'
    start_urls = ['https://ca.finance.yahoo.com/quote/aapl/analysis?p=aapl']
    
    headers = {
        'User-Agent': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.150 Safari/537.36'
    } 

    def start_requests(self):
        for start_url in self.start_urls:
            yield scrapy.Request(start_url,headers=self.headers)

    def parse(self, response):
        item = response.xpath("//td[./span][contains(.,'Next 5 Years')]/following-sibling::td/text()").getall()
        yield {"item":item}

CSS 选择器 HTML 与 Scrapy Python

CSS Selector HTML with Scrapy Python

html

python

scrapy

web-scraping