BeautifulSoup 网络抓取 href 问题

Question

我正在编写网络抓取工具，并且正在努力从网页中获取 href link。 URL 是 https://fortune.com/100-fastest-growing-companies/2021/search/。我试图让 href 拉取所有文章 links。这是我试图拉取的 href 的示例

<a class="searchResults__cellWrapper--39MAj" trackerdata="[object Object] href="https://fortune.com/company/atlanticus-holdings/100-fastest-growing-companies/"><div class="searchResults__cellContent--3WEWj"><span><div>Atlanticus Holdings</div></span></div></a>

这是我的代码：

import pandas as pd
import os
import requests
from bs4 import BeautifulSoup
import re



URL = "https://fortune.com/100-fastest-growing-companies/2021/search/"
headers={'User-Agent':'mozilla/5.0'}
page = requests.get(URL,headers=headers)
soup = BeautifulSoup(page.text, "html.parser")

links = []
for link in soup.findAll(class_="searchResults__cellWrapper--39MAj"):
    links.append((link.get('href')))
print(links)

当我 运行 我的代码时，它什么也没给我。有人能指出我正确的方向吗？

Answer 1

本页使用JavaScript添加元素

但它从

读取数据为JSON

https://fortune.com/wp-json/irving/v1/data/franchise-search-results?list_id=3159847&token=Zm9ydHVuZTpCcHNyZmtNZCN5SndjWkkhNHFqMndEOTM=

并且可以用来获取链接

import requests

url = 'https://fortune.com/wp-json/irving/v1/data/franchise-search-results?list_id=3159847&token=Zm9ydHVuZTpCcHNyZmtNZCN5SndjWkkhNHFqMndEOTM='
headers = {'User-Agent': 'Mozilla/5.0'}

response = requests.get(url, headers=headers)
data = response.json()

for item in data[1]['items']:
    print(item['permalink'])

结果：

https://fortune.com/company/progressive/100-fastest-growing-companies/
https://fortune.com/company/adobe-systems/100-fastest-growing-companies/
https://fortune.com/company/salesforce-com/100-fastest-growing-companies/
https://fortune.com/company/newmont-mining/100-fastest-growing-companies/
https://fortune.com/company/lennar/100-fastest-growing-companies/
https://fortune.com/company/kla-tencor/100-fastest-growing-companies/
https://fortune.com/company/jd-com/100-fastest-growing-companies/
https://fortune.com/company/jacobs-engineering-group/100-fastest-growing-companies/
https://fortune.com/company/cigna/100-fastest-growing-companies/
https://fortune.com/company/charles-river-laboratories-intl/100-fastest-growing-companies/
https://fortune.com/company/advanced-micro-devices/100-fastest-growing-companies/
https://fortune.com/company/paypal-holdings/100-fastest-growing-companies/
https://fortune.com/company/netflix/100-fastest-growing-companies/
https://fortune.com/company/alphabet/100-fastest-growing-companies/
https://fortune.com/company/amazon-com/100-fastest-growing-companies/
https://fortune.com/company/veeva-systems/100-fastest-growing-companies/
https://fortune.com/company/align-technology/100-fastest-growing-companies/
https://fortune.com/company/etsy/100-fastest-growing-companies/
https://fortune.com/company/lululemon-athletica/100-fastest-growing-companies/
https://fortune.com/company/trade-desk/100-fastest-growing-companies/
https://fortune.com/company/emergent-biosolutions/100-fastest-growing-companies/
https://fortune.com/company/hibbett/100-fastest-growing-companies/
https://fortune.com/company/business-first-bancshares/100-fastest-growing-companies/
https://fortune.com/company/marinemax/100-fastest-growing-companies/
https://fortune.com/company/spirit-of-texas-bancshares/100-fastest-growing-companies/
https://fortune.com/company/ies-holdings/100-fastest-growing-companies/
https://fortune.com/company/brooks-automation/100-fastest-growing-companies/
https://fortune.com/company/mid-penn-bancorp/100-fastest-growing-companies/
https://fortune.com/company/strategic-education/100-fastest-growing-companies/
https://fortune.com/company/waterstone-financial/100-fastest-growing-companies/
https://fortune.com/company/g-willi-food-international/100-fastest-growing-companies/
https://fortune.com/company/marketaxess-holdings/100-fastest-growing-companies/
https://fortune.com/company/ameris-bancorp/100-fastest-growing-companies/
https://fortune.com/company/shyft-group/100-fastest-growing-companies/
https://fortune.com/company/smartfinancial/100-fastest-growing-companies/
https://fortune.com/company/superior-group-of-companies/100-fastest-growing-companies/
https://fortune.com/company/meta-financial-group/100-fastest-growing-companies/
https://fortune.com/company/addus-homecare/100-fastest-growing-companies/
https://fortune.com/company/metropolitan-bank-holding/100-fastest-growing-companies/
https://fortune.com/company/1-800-flowers-com/100-fastest-growing-companies/
https://fortune.com/company/fs-bancorp/100-fastest-growing-companies/
https://fortune.com/company/arch-capital-group/100-fastest-growing-companies/
https://fortune.com/company/atlanticus-holdings/100-fastest-growing-companies/
https://fortune.com/company/b-riley-financial/100-fastest-growing-companies/
https://fortune.com/company/joint-corp/100-fastest-growing-companies/
https://fortune.com/company/quidel/100-fastest-growing-companies/
https://fortune.com/company/acm-research/100-fastest-growing-companies/
https://fortune.com/company/victory-capital-holdings/100-fastest-growing-companies/
https://fortune.com/company/freedom-holding/100-fastest-growing-companies/
https://fortune.com/company/staar-surgical/100-fastest-growing-companies/
https://fortune.com/company/neurocrine-biosciences/100-fastest-growing-companies/
https://fortune.com/company/kornit-digital/100-fastest-growing-companies/
https://fortune.com/company/enstar-group/100-fastest-growing-companies/
https://fortune.com/company/logitech-international/100-fastest-growing-companies/
https://fortune.com/company/pjt-partners/100-fastest-growing-companies/
https://fortune.com/company/mvb-financial/100-fastest-growing-companies/
https://fortune.com/company/repligen/100-fastest-growing-companies/
https://fortune.com/company/harborone-bancorp/100-fastest-growing-companies/
https://fortune.com/company/capstar-financial-holdings/100-fastest-growing-companies/
https://fortune.com/company/ssr-mining/100-fastest-growing-companies/
https://fortune.com/company/merchants-bancorp/100-fastest-growing-companies/
https://fortune.com/company/boston-beer/100-fastest-growing-companies/
https://fortune.com/company/first-horizon-national-corp/100-fastest-growing-companies/
https://fortune.com/company/lhc-group/100-fastest-growing-companies/
https://fortune.com/company/mercury-systems/100-fastest-growing-companies/
https://fortune.com/company/national-bank-holdings/100-fastest-growing-companies/
https://fortune.com/company/facebook/100-fastest-growing-companies/
https://fortune.com/company/monolithic-power-systems/100-fastest-growing-companies/
https://fortune.com/company/perkinelmer/100-fastest-growing-companies/
https://fortune.com/company/malibu-boats/100-fastest-growing-companies/
https://fortune.com/company/svb-financial-group/100-fastest-growing-companies/
https://fortune.com/company/first-bancshares/100-fastest-growing-companies/
https://fortune.com/company/veritex-holdings/100-fastest-growing-companies/
https://fortune.com/company/myr-group/100-fastest-growing-companies/
https://fortune.com/company/mi-homes/100-fastest-growing-companies/
https://fortune.com/company/slm/100-fastest-growing-companies/
https://fortune.com/company/dow-chemical/100-fastest-growing-companies/
https://fortune.com/company/fox-factory-holding/100-fastest-growing-companies/
https://fortune.com/company/lgi-homes/100-fastest-growing-companies/
https://fortune.com/company/take-two-interactive-software/100-fastest-growing-companies/
https://fortune.com/company/paylocity-holding/100-fastest-growing-companies/
https://fortune.com/company/alarm-com-holdings/100-fastest-growing-companies/
https://fortune.com/company/ssc-technologies-holdings/100-fastest-growing-companies/
https://fortune.com/company/stonex-group/100-fastest-growing-companies/
https://fortune.com/company/century-communities/100-fastest-growing-companies/
https://fortune.com/company/teradyne/100-fastest-growing-companies/
https://fortune.com/company/nmi-holdings/100-fastest-growing-companies/
https://fortune.com/company/virtu-financial/100-fastest-growing-companies/
https://fortune.com/company/ubiquiti/100-fastest-growing-companies/
https://fortune.com/company/scotts-miracle-gro/100-fastest-growing-companies/
https://fortune.com/company/vertex-pharmaceuticals/100-fastest-growing-companies/
https://fortune.com/company/medifast/100-fastest-growing-companies/
https://fortune.com/company/epam-systems/100-fastest-growing-companies/
https://fortune.com/company/medpace-holdings/100-fastest-growing-companies/
https://fortune.com/company/green-brick-partners/100-fastest-growing-companies/
https://fortune.com/company/leucadia-national/100-fastest-growing-companies/
https://fortune.com/company/fortinet/100-fastest-growing-companies/
https://fortune.com/company/appfolio/100-fastest-growing-companies/
https://fortune.com/company/kinsale-capital-group/100-fastest-growing-companies/
https://fortune.com/company/pennymac-financial-services/100-fastest-growing-companies/

要获取其他信息，您可能需要深入研究 JSON。

for item in data[1]['items']:
    print(item['permalink'])

    for f in item['fields']:
        if f['key'] == 'fgc_industry':
            print(f['value'])
        if f['key'] == 'rank':
            print(f['value'])
        if f['key'] == 'title':
            print(f['value'])

Url 有字段 token 但我不确定它是否会改变值。您将不得不在浏览器中观察它以获得很少的请求。如果它发生变化，那么您可能需要在原始 HTML.

中搜索它

base64解码后

import base64

base64.b64decode('Zm9ydHVuZTpCcHNyZmtNZCN5SndjWkkhNHFqMndEOTM=')

它给出 'fortune:BpsrfkMd#yJwcZI!4qj2wD93'，您可能需要在 HTML 中搜索此值。