403 禁止 BeautifulSoup 网页抓取工具

Question

我正在构建一个网络抓取工具来从 https://www.startengine.com/explore 中提取 href，但我很难获得任何 href。我决定打印网页并找出原因。

这是我的代码：

import pandas as pd
import os
import requests
from bs4 import BeautifulSoup
import re

URL = "https://www.startengine.com/explore"
page = requests.get(URL)
soup = BeautifulSoup(page.text, "html.parser")

links = []
print(soup)

这是输出：

<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
</body>
</html>

谁能帮我解决“403 Forbidden”问题？

Answer 1

您需要将您的 user-agent 注入为 header，如下所示：

import pandas as pd
import os
import requests
from bs4 import BeautifulSoup
import re

URL = "https://www.startengine.com/explore"
headers={'User-Agent':'mozilla/5.0'}
page = requests.get(URL,headers=headers)
print(page)
soup = BeautifulSoup(page.text, "html.parser")

links = []
print(soup)

403 禁止 BeautifulSoup 网页抓取工具

403 Forbidden BeautifulSoup Web Scraper

html

python

beautifulsoup