尝试使用 puppeteer 从网页中抓取 link

Question

我不明白为什么我无法使用 puppeteer 从该页面检索 href link： PubChem.

我有运行 Chrome 并检查了页面并找到了所需的 Headline 化学品并复制了 Selector 如下所示： #featured-results > div:nth-child(2) > div.box-shadow > div > div.p-md-rectangle.flex-container.flex-nowrap.width-100 > div.flex-grow-1.p-md-left > div.f-medium.p-sm-top.p-sm-bottom.f-1125 > a

然后我有运行这个带有nodejs的JS代码。

     const puppeteer = require('puppeteer')
     puppeteer.launch({ headless: true }).then(async browser => {
         const page = await browser.newPage()
         await page.goto('https://pubchem.ncbi.nlm.nih.gov/#query=MES')
     //    const cookies = await page.cookies()
     //    console.log(cookies)
         const links = await page.evaluate(() => [document.querySelectorAll('#featured-results > div:nth-child(2) > div.box-shadow > div > div.p-md-rectangle.flex-container.flex-nowrap.width-100 > div.flex-grow-1.p-md-left > div.f-medium.p-sm-top.p-sm-bottom.f-1125 > a')].map(link => link.href))
         links.forEach(link => console.log(link))
        
         await browser.close()
     })

但是我的结果是NULL。这里有人能开开我的眼睛吗？谢谢。

Answer 1

您需要等待元素出现。
您需要展开 (...) 以根据 querySelectorAll() 结果创建一个数组。

const puppeteer = require('puppeteer')
puppeteer.launch({ headless: true }).then(async browser => {
    const page = await browser.newPage()
    await page.goto('https://pubchem.ncbi.nlm.nih.gov/#query=MES')

    await page.waitForSelector('#featured-results > div:nth-child(2) > div.box-shadow > div > div.p-md-rectangle.flex-container.flex-nowrap.width-100 > div.flex-grow-1.p-md-left > div.f-medium.p-sm-top.p-sm-bottom.f-1125 > a')

    const links = await page.evaluate(
        () => [...document.querySelectorAll('#featured-results > div:nth-child(2) > div.box-shadow > div > div.p-md-rectangle.flex-container.flex-nowrap.width-100 > div.flex-grow-1.p-md-left > div.f-medium.p-sm-top.p-sm-bottom.f-1125 > a')]
                .map(link => link.href)
    )
    links.forEach(link => console.log(link))

    await browser.close()
})

尝试使用 puppeteer 从网页中抓取 link

Trying to scrape a link from webpage with puppeteer

javascript

node.js

web-scraping

puppeteer