如何抓取arxiv sanity？

Question

我想抓取 "link"、"title" 和 "abstract"

我怎样才能抓取这个？

我试过了

import requests
import json

url = 'http://www.arxiv-sanity.com/top?timefilter=year&vfilter=all'
res = requests.get(url)
text = res.text
# print(text)

d = json.loads(text)
print(d['title'], d['link'], d['abstract'])

但是SONDecodeError: Expecting value: line 1 column 1 (char 0)发生

Answer 1

那个 URL returns 一个 HTML，而不是一个 json 响应。所以你不能JSON解码它。

Answer 2

使用BeautifulSoup:

import requests
import json
from bs4 import BeautifulSoup as bs

url = 'http://www.arxiv-sanity.com/top?timefilter=year&vfilter=all'
res = requests.get(url)
text = res.text
soup=bs(text, "html.parser")
extract=soup.select('script')[6]

target = extract.decode().split('var papers = ')[1]
target2 = target.replace("}, {","}xxx{").replace('[{','{').replace('}];','}')
final = target2.split('xxx')

for i in range(len(final)):
    if i == len(final)-1:
        last = final[i].split('var pid')[0]
        d = json.loads(last)        
        print(d['title'],d['link'],d['abstract'])
    else:
        d = json.loads(final[i])
        print(d['title'],d['link'],d['abstract'])

示例输出：

BERT: Pre-training of Deep Bidirectional Transformers for Language       Understanding 
http://arxiv.org/abs/1810.04805v2 
We introduce a new language representation model called BERT, which stands
for Bidirectional Encoder Representations from Transformers. Unlike recent
language representation models, BERT is designed to pre-train deep
bidirectional representations from unlabeled text by jointly conditioning on
both left and right context in all layers...

等等

如何抓取arxiv sanity？

how to crawling arxiv sanity?

python

web-crawler