如何有效地将 JavaScript Json 解析为 Python 字典类型

Question

我正在寻找读取 javascript json 加载到此 page 脚本标签之一的数据的方法。我尝试了 google 和 stackoveflow 上发布的各种 re 模式，但一无所获。

Json Formatter 显示无效 (RFC 8259)。

这是一个代码

import requests,json
from scrapy.selector import Selector

headers = {'Content-Type': 'application/json', 'Accept-Language': 'en-US,en;q=0.5', 'User-Agent': 'Mozilla/5.0 (iPhone; CPU iPhone OS 5_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9B179 Safari/7534.48.3'}

url = 'https://www.zocdoc.com/doctor/andrew-fagelman-md-7363?insuranceCarrier=-1&insurancePlan=-1'

response = requests.get(url,headers = headers)

sel = Selector(text = response.text)

profile_data = sel.css('script:contains(APOLLO_STATE)::text').get('{}').split('__REDUX_STATE__ = JSON.parse(')[-1].split(');\n          window.ZD = {')[0]
    
profile_json = json.loads(profile_data)
    
print(type(profile_json))

问题似乎是无效的 json 格式。 profile_json 的类型是字符串，而上面代码中的一些修改显示错误堆栈

>>> profile_data = sel.css('script:contains(APOLLO_STATE)::text').get('{}').split('__REDUX_STATE__ = JSON.parse("')[-1].split('");\n          window.ZD = {')[0].replace("\","")
>>> profile_json = json.loads(profile_data)
Traceback (most recent call last):
  File "/usr/lib/python3.6/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/usr/lib/python3.6/json/__init__.py", line 354, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.6/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode
    obj, end = self.scan_once(s, idx)
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 1 column 41316 (char 41315)

此处突出显示输出错误：

原文HTML包含这个（严重修剪）：

<script>
   ...
   window.__REDUX_STATE__ = JSON.parse("{\"routing\": ...
   \"awards\":[\"Journal of Urology - \\"Efficacy, Safety, and Use of Viagra in Clinical Practice.\\"\",\"Critical Care Resident of the Year - 2003\"],
   ...

scrapy提取的同一个字符串是这样的：

"awards":[
               "Journal of Urology - ""Efficacy",
               "Safety",
               "and Use of Viagra in Clinical Practice.""",
               "Critical Care Resident of the Year - 2003"
            ],

似乎反斜杠已从中删除，使 JSON 无效。

Answer 1

我不知道这是否是处理问题的有效方法，但下面的代码解决了我的问题。

>>> import js2xml
>>> profile_data = sel.css('script:contains(APOLLO_STATE)::text').get('{}')
>>> parsed = js2xml.parse(profile_data)
>>> js = json.loads(parsed.xpath("//string[contains(text(),'routing')]/text()")[0])

如何有效地将 JavaScript Json 解析为 Python 字典类型

How to parse JavaScript Json into Python dict type, effeciently

javascript

python

json

scrapy