POST 查询以访问网站上的 table 数据 (Python)

Question

我想使用 this website 从中提取数据，但是该网站没有 API 以方便提取数据。所以网站上有一个 "table" 包含数据，但是这个 table 分布在多个页面上，下载选项不支持 "bulk" 下载。所以我想想办法通过Python下载这个。
我发现我可以通过 post 请求来做到这一点。我所做的是从第 1 页开始，然后单击 "next page" 按钮并查看网络选项卡，这给了我：

Request URL:https://www.patricbrc.org/api/genome/
Request Method:POST
Status Code:200 OK
Remote Address:128.173.97.11:443
Referrer Policy:no-referrer-when-downgrade

Response Headers
Access-Control-Allow-Origin:https://www.patricbrc.org
Access-Control-Expose-Headers:facet_counts,x-facet-count,Content-Range,X-Content-Range,ETag
Content-Encoding:gzip
Content-Range:items 200-399/45999
Content-Type:application/json; charset=utf-8
Date:Sat, 26 Aug 2017 09:36:40 GMT
ETag:W/"8ac0b-cFdEBUwfdiyTQm/gpJHYzQ"
Server:nginx/1.9.1
Transfer-Encoding:chunked
Vary:Origin, Accept
Vary:Accept-Encoding
X-Powered-By:Express

Request Headers
Accept:application/javascript, application/json, application/json
Accept-Encoding:gzip, deflate, br
Accept-Language:nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Content-Length:82
Content-Type:application/rqlquery+x-www-form-urlencoded
Cookie:sprod=hZ36nWTBiHHaDwdAhCChFGZo; _ga=GA1.2.582935779.1503602311; _gid=GA1.2.174755110.1503602311; _gat=1
Host:www.patricbrc.org
Origin:https://www.patricbrc.org
Range:items=200-399
Referer:https://www.patricbrc.org/view/Taxonomy/2
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
X-Range:items=200-399

Request Payload
eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)
Name
genome/

为了查看页面切换后发生的变化，我再次执行了此操作（因此再次单击 "next page" 按钮）。

Request URL:https://www.patricbrc.org/api/genome/
Request Method:POST
Status Code:200 OK
Remote Address:128.173.97.11:443
Referrer Policy:no-referrer-when-downgrade

Response Headers
Access-Control-Allow-Origin:https://www.patricbrc.org
Access-Control-Expose-Headers:facet_counts,x-facet-count,Content-Range,X-Content-Range,ETag
Content-Encoding:gzip
Content-Range:items 400-599/45999
Content-Type:application/json; charset=utf-8
Date:Sat, 26 Aug 2017 09:44:24 GMT
ETag:W/"5dc73-ODopjbbyl5M2vUJvUGN0Gw"
Server:nginx/1.9.1
Transfer-Encoding:chunked
Vary:Origin, Accept
Vary:Accept-Encoding
X-Powered-By:Express

Request Headers
Accept:application/javascript, application/json, application/json
Accept-Encoding:gzip, deflate, br
Accept-Language:nl-NL,nl;q=0.8,en-US;q=0.6,en;q=0.4
Connection:keep-alive
Content-Length:82
Content-Type:application/rqlquery+x-www-form-urlencoded
Cookie:sprod=hZ36nWTBiHHaDwdAhCChFGZo; _ga=GA1.2.582935779.1503602311; _gid=GA1.2.174755110.1503602311
Host:www.patricbrc.org
Origin:https://www.patricbrc.org
Range:items=400-599
Referer:https://www.patricbrc.org/view/Taxonomy/2
User-Agent:Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.101 Safari/537.36
X-Range:items=400-599

Request Payload
eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)

因此，在从第 1 页 --> 2 和第 2 页 --> 3 开始的 post 请求期间（在我看来）发生变化的重要事项是：

响应headerContent-Range:项目200-399/45999 --> Content-Range:项目 400-599/45999
请求header范围：items=200-399 --> 范围：items= 400-599
请求headerX-Range:items=200-399 --> X-Range :件数=400-599

网站是这样说的：

第 1 页：45999 个结果中的第 1 - 200 个（但 posts 表示 0-199，上面未显示）
第 2 页：45999 个结果中的第 201 - 400 个（但 post 为 200-399，见上文）
第 3 页：45999 个结果中的 401 - 600 个（但 400-599 的 posts，见上文）

关于阅读这些 POST 请求，我零知识。但我想应该有一种说法：

requests.post(some url) and extract the total page numbers (or number of genomes and devide these by the max number per page, which is 200 see post requests).

然后是这样的：

for page_numb in range(page_numbers):
    r = requests.post(some_url...... + page_numb)
    #extract the information from the table

终于是问题了:) 我不知道如何设置 POST 请求，例如

p1 = requests.post(https://www.patricbrc.org/api/genome/... 0-199)
p2 = requests.post(https://www.patricbrc.org/api/genome/... 200-399)
etc...

（我希望响应将包含 table 数据，这些数据由一些分隔符分隔）

我试过的代码：

import requests
import json

url = 'https://www.patricbrc.org/api/genome/'
payload = {"eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)"}
headers = {
    "X-Range": "items=0-199",
    "Range": "items=0-199",
    "content-type": "application/json"}


r = requests.post(url, data=json.dumps(payload), headers=headers)
print(r)

此代码无效，但我想它应该是这样的。
我知道我应该 post 作为 JSON 的有效载荷，但我不知道如何将 eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score) 翻译成 JSON，它是 key:value 对。

Answer 1

这是一个源代码（它的代码丑陋，但只是示例，我建议您使用request.Session）：

import sys
import json
import requests

URL = 'https://www.patricbrc.org/view/Taxonomy/2'
API = 'https://www.patricbrc.org/api/genome/'

# we need to get a Cookie
response = requests.get(URL)
if response.status_code != 200:
    sys.exit('Cookie request failed')
cookie = response.headers['Set-Cookie'].split(';')[0]
etag   = response.headers['ETag']

# now with cookie request a data
headers = {
    'Host'            : 'www.patricbrc.org',
    'Accept'          : 'application/javascript, application/json, application/json',
    'Accept-Language' : 'en-US,en;q=0.5',
    'Referer'         : 'https://www.patricbrc.org/view/Taxonomy/2',
    'X-Range'         : 'items=200-399',  # your pages
    'Range'           : 'items=200-399',  # your pages
    'Content-Type'    : 'application/rqlquery+x-www-form-urlencoded',
    'Cookie'          : '{};'.format(cookie),
    'ETag'            : etag
}

data = 'eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)'
response = requests.post(API, headers=headers, data=data)
if response.status_code != 200:
    sys.exit('API request failed')
data = json.loads(response.text)

# test data
row200 = data[0]
print 'FAMILY     :', row200['family']
print 'GC_CONTENT :', row200['gc_content'] 
print 'GENUS      :', row200['genus']
print 'ORDER      :', row200['order']

test data 块的输出：

FAMILY     : Eubacteriaceae
GC_CONTENT : 38.78
GENUS      : Eubacterium
ORDER      : Clostridiales

编辑您不需要 cookie，这也可以正常工作：

import json
import requests

API = 'https://www.patricbrc.org/api/genome/'

# now with cookie request a data
headers = {
    'Host'            : 'www.patricbrc.org',
    'Accept'          : 'application/javascript, application/json, application/json',
    'Accept-Language' : 'en-US,en;q=0.5',
    'Referer'         : 'https://www.patricbrc.org/view/Taxonomy/2',
    'X-Range'         : 'items=200-399',  # your pages
    'Range'           : 'items=200-399',  # your pages
    'Content-Type'    : 'application/rqlquery+x-www-form-urlencoded',
}

query = 'eq(taxon_lineage_ids,2)&eq(host_name,%22Human%2C%20Homo%20sapiens%22)&sort(-score)'
response = requests.post(API, headers=headers, data=query)
data = json.loads(response.text)

POST 查询以访问网站上的 table 数据 (Python)

POST query to access table data on a website (Python)

python

post

response

request

web-scraping