Python 请求库调用适用于 Windows 但不适用于 Linux... 为什么？

Question

python 的长期用户在这里请求。尝试对此端点进行简单调用： https://www.overstock.com/api/product.json?prod_id=10897789

我当前的代码：

import requests

headers = { 'User-Agent': 'Mozilla/5.0', 'Accept': 'application/json' }
url = 'https://www.overstock.com/api/product.json?prod_id=10897789'
r = requests.get( url, headers=headers )
result = r.json()
print( result )

预期结果（缩短）：

{'categoryId': 244, 'subCategoryId': 31446, 'altSubCategoryId': 0, 'taxonomy': {'store': {'id': 1, 'name': 'Rugs', 'apiUrl': 'https://www.overstock.com/api/search.json?taxonomy=sto1', 'htmlUrl': 'https://www.overstock.com/Home-Garden/1/store.html'}, 'department': {'id': 3, 'name': 'Casual Rugs'...

不幸的是，从 Linux 上的同一个脚本，我没有得到相同的结果。到目前为止，我对为什么会发生这种情况感到困惑...

这是丑陋的 Linux 错误：

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/.local/share/virtualenvs/online-project-7j1lNF7P/lib/python3.6/site-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)

可能是什么问题？这是我尝试过的其他方法...

~~Linux 不是运行 python 3.6，而是运行 2.7x 来执行请求。~~
~~在headers中加上'Accept':'application/json'一定可以解决~~
decode数据变量first data = response.decode() (link to SO post) Fail: "AttributeError : 'Response' object 没有属性 'decode'"
~~使用requests.Response.json~~ (link to SO post) 失败：给出与上面相同的错误。
~~升级到python3.9.9可能会解决。~~不行！这对我来说仍然失败。
~~也许是你的防火墙。~~ 不，检查过 ufw 是 Status: inactive

#5 错误（在新 Linux 机器上，升级 python 到 3.9.9）：

`$ python3 test.py
Traceback (most recent call last):
  File "/home/user/test.py", line 13, in <module>
    print(r.json())
  File "/usr/lib/python3/dist-packages/requests/models.py", line 892, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None

@balmy - 这是我在确认请求版本 2.26.0 和 python 3.9...

后得到的输出

$ python3 test3.py
Traceback (most recent call last):
  File "/home/user/test_scripts/test3.py", line 13, in <module>
    print(r.json())
  File "/home/eric/.local/lib/python3.9/site-packages/requests/models.py", line 910, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 2 column 1 (char 1)

@JCaesar - 这是文本（缩短到我认为相关的部分，似乎机器人检测正在发挥作用）

        <div id="bd">
            <div class="ohNoRedBar">
                There was an error processing your request.
            </div>
            <span class="ohNoText"></span>
        </div>

@Philippe - 这是对您评论的回应 'Can you change the print statement to print(r.text) and run python3 test3.py | jq .'...

$ sudo python3 test3.py | jq .
Traceback (most recent call last):
  File "/usr/lib/command-not-found", line 28, in <module>
    from CommandNotFound import CommandNotFound
  File "/usr/lib/python3/dist-packages/CommandNotFound/CommandNotFound.py", line 19, in <module>
    from CommandNotFound.db.db import SqliteDatabase
  File "/usr/lib/python3/dist-packages/CommandNotFound/db/db.py", line 5, in <module>
    import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'
Traceback (most recent call last):
  File "/home/eric/test_scripts/test3.py", line 13, in <module>
    print(r.text)
BrokenPipeError: [Errno 32] Broken pipe

@Philippe - 回答你的下一条评论 $ sudo python3 test3.py | jq . parse error: Invalid numeric literal at line 2, column 10 Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='utf-8'> BrokenPipeError: [Errno 32] Broken pipe

如果您有解决方案，请告诉我。谢谢！

Answer 1

运行 requests macOS 12.0.1 和 Python 3.9.9 上的 2.26.0 我发现网站 requires Accept-Encoding 中的 headers。这对我来说符合预期：

import requests

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15',
    'Accept': 'application/json',
    'Connection': 'keep-alive',
    'Accept-Encoding': 'gzip, deflate, br'
}


with requests.Session() as session:
    (r := session.get('https://www.overstock.com/api/product.json?prod_id=10897789', headers=headers)).raise_for_status()
    print(r.json())

Answer 2

都是因为IP被封了

这是最终挽救了局面的脚本...

import requests

url = "https://www.overstock.com/api/product.json?prod_id=10897789"

headers = {
    'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 11_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/14.1.2 Safari/605.1.15',
    'Accept': 'application/json',
    'Connection': 'keep-alive',
    'Accept-Encoding': 'gzip, deflate, br'
}

http_proxy  = "http://ip:port"
https_proxy = "http://ip:port"

proxyDict = {
              "http"  : http_proxy,
              "https" : https_proxy
            }

r = requests.get(url, headers=headers, proxies=proxyDict)
result = r.json()
print(result)

感谢大家的努力！

在看到这适用于@JCaesar、@diggusbickus、@balmy 和@Philippe 之后，我意识到唯一没有解决的问题是 IP 地址。通过添加旋转住宅代理 IP，我发出了请求并立即获得了数据。

感谢@JCaesar 透露 'Accept-Encoding' 没有它，它根本无法工作。感谢@diggusbickus 对 walrus notation := 的评论，否则我会假设 Python 3.9.x 是运行并升级它。

Python 请求库调用适用于 Windows 但不适用于 Linux... 为什么？

Python requests library call works on Windows but not Linux... WHY?

python

linux

json

web-scraping

python-requests