抓取的数据数量有限？

Question

我正在抓取一个网站，从今天的新闻到 2015/2016 年发布的新闻，一切似乎都正常。这些年过去了，我已经无法抓取新闻了。你能告诉我是否有任何变化吗？我应该得到 672 页从该页面获取标题和摘要：

https://catania.liveuniversity.it/attualita/

但我有大约。 158.

我使用的代码是：

import bs4, requests
import pandas as pd
import re

headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36'}

page_num=1
website="https://catania.liveuniversity.it/attualita/"

while True:
    r = requests.get(website, headers=headers)
    soup = bs4.BeautifulSoup(r.text, 'html')


    title=soup.find_all('h2')
    date=soup.find_all('span', attrs={'class':'updated'})

    if soup.find_all('a', attrs={'class':'page-numbers'}):
      website = f"https://catania.liveuniversity.it/attualita/page/{page_num}"
      page_num +=1
      print(page_num)
    else:
      break


df = pd.DataFrame(list(zip(dates, titles)), 
               columns =['Date', 'Titles'])

我认为标签有一些变化（例如在下一页按钮中，或者只是在 date/title 标签中）。

Answer 1

import requests
from bs4 import BeautifulSoup
from concurrent.futures import ThreadPoolExecutor
import pandas as pd


def main(req, num):
    r = req.get(
        "https://catania.liveuniversity.it/attualita/page/{}/".format(num))
    soup = BeautifulSoup(r.content, 'html.parser')
    try:
        data = [(x.select_one("span.updated").text, x.findAll("a")[1].text, x.select_one("div.entry-content").get_text(strip=True)) for x in soup.select(
            "div.col-lg-8.col-md-8.col-sm-8")]
        return data
    except AttributeError:
        print(r.url)
        return False


with ThreadPoolExecutor(max_workers=30) as executor:
    with requests.Session() as req:
        fs = [executor.submit(main, req, num) for num in range(1, 673)]
        allin = []
        for f in fs:
            f = f.result()
            if f:
                allin.extend(f)
        df = pd.DataFrame.from_records(
            allin, columns=["Date", "Title", "Content"])
        print(df)
        df.to_csv("result.csv", index=False)

抓取的数据数量有限？

Limited number of scraped data?

python

beautifulsoup

web-crawler

web-scraping