Python 请求库无助于获取正确的网页

Question

我有这个网站：

https://xueqiu.com/hq#exchange=CN&firstName=1&secondName=1_0

我正在尝试通过 Python 的获取请求获取此网页。我也试过改变“用户代理”。但是我无法获取网页，我是这个解析的新手。

    url = 'https://xueqiu.com/hq#exchange=CN&firstName=1&secondName=1_0'
    with request.session() as session:
           response = session.get(url)

有人可以帮我解压吗？

Answer 1

您的数据通过以下 url json 格式加载。所以我使用 json 模块来提取数据。

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup as bs
import requests
import json
headers={
    'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0',
}
def scrape(url):
    with requests.Session() as req:
        req.headers.update(headers)
        r = req.get(url)
        mydata =r.json()
        for data in mydata['data']['list']:
            print(data, sep='*')

url =  'https://xueqiu.com/service/v5/stock/screener/quote/list?page=1&size=30&order=desc&orderby=percent&order_by=percent&market=CN&type=sh_sz&_=1606221698728'
scrape(url)

希望对你有所帮助。

Python 请求库无助于获取正确的网页

Python Request library not helping in Getting the correct webpage

python

get

request

scrapy

web-scraping