如何在使用搜索框抓取网站时发送包含搜索框内容的请求？

Question

我正在尝试用漂亮的汤刮 website。为此，在发出请求时，我需要发送应该在搜索框中输入的 Instagram 用户 ID 并抓取响应 HTML。发送请求时应该如何发送用户 ID？谢谢

import requests
from bs4 import BeautifulSoup

URL = "https://product.influencer.in/price-index"
r = requests.get(URL)

soup = BeautifulSoup(r.content, 'lxml') 
print(soup.prettify())

Answer 1

这是对 api 的简单 post 请求以获取该数据。您需要在下面的值中输入您的电子邮件

import requests

instagramHandles = ['leomessi','cristiano']

URL = "https://team.influencer.in/api/v1/price-index/"
headers = {'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/96.0.4664.110 Safari/537.36'}

for instagramHandle in instagramHandles:
    payload = {
        'email': "name@email.com",
        'insta_handle': instagramHandle}
    
    jsonData = requests.post(URL, headers=headers, data=payload).json()
    cost_min = jsonData['data']['cost_min']
    cost_max = jsonData['data']['cost_max']
    
    print(f'{instagramHandle}: {cost_min} - {cost_max}')

输出：

leomessi: 5.4 Cr. - 6.5 Cr.
cristiano: 8.5 Cr. - 10.1 Cr.

如何在使用搜索框抓取网站时发送包含搜索框内容的请求？

How to send a request which includes search box content while scraping a website with search box?

python

beautifulsoup

web-scraping