无法复制 ajax 抓取请求

Question

我正在尝试从网站抓取数据，但内容是动态添加的。我正在使用 Python 和 requests 库。这是示例 https://www.goat.com/sneakers/off-white-x-air-presto-aa3830-002

的 link

url = 'https://www.goat.com/web-api/graphql'  # the API endpoint

# replicate the header in Chrome debugger tool of the successful request
headers = {  
        ':authority': 'www.goat.com',
        ':method': 'POST',
        ':path': '/web-api/graphql',
        ':scheme': 'https',
        "accept": "application/json",
        "accept-encoding": "gzip, deflate, br",
        "accept-language": "vi-VN,vi;q=0.9,fr-FR;q=0.8,fr;q=0.7,en-US;q=0.6,en;q=0.5",
        "content-length": "1153",
        "content-type": "application/json",
        "origin": "https://www.goat.com",
        "referer": "https://www.goat.com/sneakers/off-white-x-air-presto-aa3830-002",
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36",
    }

# the request payload from Chrome debugger tool
data = {"query":'{\n    viewer {\n      productTemplate(slug: \"off-white-x-air-presto-aa3830-002\") {\n        id\n        internal_shot\n        details\n        name\n        original_picture_url\n        sku\n        slug\n        story\n        is_active\n        release_date\n        color\n        special_type\n        upper_material\n        lowest_price_cents\n        new_lowest_price_cents\n        can_return\n        brand {\n          id\n          name\n        }\n        size_brand\n        midsole\n        designer\n        nickname\n        silhouette\n        gender\n        formatted_available_sizes_new_v2 {\n          size\n          price_cents\n          box_condition\n          shoe_condition\n        }\n        product_template_additional_pictures {\n          attribution_url\n          original_picture_url\n          source_url\n        }\n        selling_count\n        used_for_sale_count\n        used_lowest_price_cents\n        goat_clean_for_sale_count\n        new_lowest_price_cents\n        with_defect_for_sale_count\n        category\n      }\n    }\n  }',
"variables":{"slug":"off-white-x-air-presto-aa3830-002"}}

r = session.post(url, headers=headers, data=data)
print(r.text)

发送到服务器的POST请求返回了Internal Server Error (500)，但是浏览器正常执行并获取了数据.

我觉得我遗漏了一些东西，尤其是在 query 有效载荷部分。请告诉我为什么会这样。非常感谢！

Answer 1

我终于想通了。端点只接受 JSON 作为数据并且不正确地执行常规请求（为什么会发生错误 500）。

下面是有效的代码：

替换

r = session.post(url, headers=headers, data=data)

与

r = session.post(url, headers=headers, json=data)

它就像一个魅力！

无法复制 ajax 抓取请求

Unable to replicate ajax request for scraping

python

post

python-requests

graphql