如何迭代页面以抓取网络新闻

Question

我一直在努力弄清楚如何迭代页面以抓取多篇新闻文章。

这是我要抓取的页面：（及其后续页面） https://www.startribune.com/search/?page=1&q=China%20COVID-19&refresh=true

我尝试了下面的代码，但是 return 结果不正确：

def scrape(url):
    user_agent = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko'}
    urls = [f"{url}{x}" for x in range(1,10)]
    params = {
        'q': 'China%20COVID-19'
    }
    for page in urls:
        response = requests.get(url=page,
                                headers=user_agent,
                                params=params) 
    print(page)

print(scrape('https://www.startribune.com/search/'))

请提出改进或解决方案！

我期望的结果是：

https://www.startribune.com/search/?page=1&q=China%20COVID-19&refresh=true 
https://www.startribune.com/search/?page=2&q=China%20COVID-19&refresh=true
...
https://www.startribune.com/search/?page=9&q=China%20COVID-19&refresh=true

Answer 1

如评论中所述，确保 params 完整：

def scrape(url):
    user_agent = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; WOW64; Trident/7.0; Touch; rv:11.0) like Gecko'}
    params = {
        'q': 'China%20COVID-19',
        'refresh': 'true',
    }
    for page_no in range(1, 10):
        params['page'] = page_no
        response = requests.get(url=url,
                                headers=user_agent,
                                params=params) 
        print(response.request.url)
        # https://www.startribune.com/search/?q=China%2520COVID-19&refresh=true&page=1

scrape('https://www.startribune.com/search/')

如何迭代页面以抓取网络新闻

How to iterate pages to scrape web news

python

iterator

web-crawler

web-scraping