无法使用请求库获得正确的响应页面

Question

我正在尝试解析网页 https://xueqiu.com/S/SZ300816 上的评论。但是我无法通过请求库正确获取它：

>>> url = 'https://xueqiu.com/S/SZ300816'
>>> headers
{'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0'}
>>> response = requests.get(url, headers=headers)
>>> from bs4 import BeautifulSoup as bs4
>>> soup = bs4(response.text)
>>> soup.findAll('article', {'class': "timeline__item"})
[]
>>>

有人可以指出我做错了什么吗？谢谢

Answer 1

我从 chrome 开发工具的网络选项卡中获得了 url。通过此 url 以 json 格式加载的数据。我试着解决你的问题，希望能帮到你。

  from urllib.request import Request, urlopen
    from bs4 import BeautifulSoup as bs
    import requests
    import json
    headers={
        'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0',
    }
    def scrape(url):
        with requests.Session() as req:
            req.headers.update(headers)
            r = req.get(url)
            mydata =r.json()
            print(mydata['list'][0])
            print(mydata['list'][0]['text'])
            print(mydata['list'][0]['description'])
    
           
    url =  'https://xueqiu.com/query/v1/symbol/search/status?u=141606248084627&uuid=1331335789820403712&count=10&comment=0&symbol=SZ300816&hl=0&source=all&sort=&page=1&q=&type=11&session_token=null&access_token=db48cfe87b71562f38e03269b22f459d974aa8ae'
    scrape(url)

无法使用请求库获得正确的响应页面

Unable to get correct response page using requests library

python

get

request

scrapy