使用 Puppeteer 进行分页

Question

我看了10+教程，但我仍然无法解决这个基本操作。

如何使用 puppeteer 对我正在抓取的 Glassdoor 页面进行分页？

在这里，我成功登录到 glassdoor（登录详细信息已替换为 ****），并抓取了第一页可见的基本详细信息，例如公司名称、发布日期等：

async function scrapeListings(page) {

//navigating to the list of jobs i.e. logging in to Glassdoor
try {
  await page.goto("https://www.glassdoor.co.uk/index.htm");
    await page.click(
      "#SiteNav > nav > div.d-lg-none.d-flex.align-items-center.justify-content-between.px-std.py-xsm.px-md-lg.py-md-std.LockedHomeHeaderStyles__bottomBorder.LockedHomeHeaderStyles__fullWidth > div.d-flex.justify-content-center.order-1.order-md-2.LockedHomeHeaderStyles__flexibleContainer > button",
      { delay: 200 }
    );
    await page.type("#userEmail", "*******", {
      delay: 200,
    });
    await page.type("#userPassword", "*******", { delay: 200 });
    await page.click(".mt-std.d-flex.flex-column.align-items-center", {
      delay: 200,
    });

    await page.waitForNavigation();
    await page.goto(
      "https://www.glassdoor.co.uk/Job/london-internship-jobs-SRCH_IL.0,6_IC2671300_KO7,17_IP1.htm"
    );
  

  const html = await page.content();
  const $ = cheerio.load(html);
  const listings = $("[data-test='jobListing']")
    .map((index, element) => {
      const titleElement = $(element).find(".css-l2wjgv.e1n63ojh0.jobLink");
      const timeElement = $(element).find("[data-test='job-age']");
      const companyName = $(titleElement).text();
    

      const url = "https://www.glassdoor.co.uk" + $(titleElement).attr("href");
      const datePosted = $(timeElement).text();
  

      return { companyName, url, datePosted };
    })
    .get();
  return listings; 
  await page.waitForNavigation({ waitUntil: 'networkidle0'})
} catch (erro) {
    console.error(erro);
  }
}

在第二部分中，我得到了每个职位描述的 url 并抓取了一个包含更多详细信息的页面：


async function scrapeJobDescriptions(listings, page) {
  for (var i = 0; i < listings.length; i++) {
    await page.goto(listings[i].url);
    const html = await page.content();
    const $ = cheerio.load(html);
    // const jobDescription = $(".desc").text();
    const jobDescription = $("#JobDescriptionContainer").html();
    const location = $(".css-f4rs18.css-1e169oc.efy8art2 > div > div > div:nth-child(3)").text()
    const jobSalary = $(".css-1v5elnn.e11nt52q2 .small.css-10zcshf.e1v3ed7e1").text()
    const jobPosition = $(".css-17x2pwl").text();
    const applyLink =
      "https://www.glassdoor.co.uk" +
      $(".css-0.e1h54cx80 a").attr("data-job-url");
    try {
    const companyImage = await page.$eval(
      // ".oc-photo-gallery .photo__10vsfGte img",
      ".css-13u5hxa.epu0oo22 img",
      (img) => img.src
    );
   

    listings[i].jobDescription = jobDescription;
    listings[i].location = location;
    listings[i].jobSalary = jobSalary;
    listings[i].jobPosition = jobPosition;
    listings[i].applyLink = applyLink;
    listings[i].companyImage = companyImage;
    console.log(listings[i].jobDescription);
    const listingModel = new GlassdoorDB(listings[i]);
    await listingModel.save();
    await sleep(1000); //1 second sleep

  
  } catch(err) {
    console.log(err)
}
}
}

async function sleep(miliseconds) {
  return new Promise(resolve => setTimeout(resolve, miliseconds));
}

async function main() {
  await connectToMongoDb();
  const browser = await puppeteer.launch({ headless: false });
  const page = await browser.newPage();
  const listings = await scrapeListings(page);
  const listingsWithJobDescriptions = await scrapeJobDescriptions(
    listings,
    page
  );
  console.log(listings);
}

main();

第一页 30 条结果被抓取后如何进入下一页？

Answer 1

您可以检查分页的 URL 模式 - 正如 Molda 推荐的 - 而不是等待木偶操作然后单击“下一步”按钮：您可以 page.goto 到下一个。例如：https://example.net?page=1等

在大多数分页中，如果有跳转到最后一页的“最后”按钮，您可以检索最后一页的页码(el.href.match(/\d+/))。

例如：

await page.goto('https://example.net')
const lastPage = await page.evaluate(el => el.href.match(/\d+/), (await page.$$('.pager-last > a'))[0])

for (let i = 0; i < parseInt(lastPage[0]) + 1; i++) {
  try {
    await page.goto('https://example.net?page=' + i)
    // manipulating the DOM
  } catch (e) {
    console.error(e)
  }
}

在 Glassdoor 站点上，您将能够从以下 <div>:

中检索分页的长度

<div class="cell middle d-none d-md-block py-sm" data-test="page-x-of-y">Page 1 of 17</div>

const lastPage = await page.evaluate(el => el.innerText.replace(/page\s\d\sof\s/i, ''), (await page.$$('[data-test="page-x-of-y"]')[0])

使用 Puppeteer 进行分页

Pagination with Puppeteer

javascript

node.js

web-scraping

puppeteer