使用 elasticsearch 遍历查询的所有结果

Question

我正在 Python 中使用 Elasticsearch 的 DSL。我的目标是使用 elasticsearch-dsl-py 在循环中尽可能轻松地处理 Elasticsearch 响应数据。

import datetime
import json
from elasticsearch import Elasticsearch
from elasticsearch_dsl import Search

e_search = Elasticsearch([{'host': 'my-alias', 'port': 5648}])

s = Search(using=e_search, index='sampleindex-2019.10') \
    .filter('range' ,  **{'@timestamp': {'gte': 1571633450000, 'lt': 1571669450000, 'format' : 'epoch_millis'}})

当我执行此操作时，我得到以下值：

response = s.execute()
print(response.success())
>>> True
print(response.took)
>> 41
print(response.hits.total)
>> 6582

但是，当我尝试遍历所有结果时，它似乎只打印出 10 个匹配项：

for h in response:
    print(hit)

<Hit(sampleindex-2019.10/nQGt7G0BGh3E1MmaFw8e): {'startTime': '2019-10-21T13:57:05.621300916+09:00', 'header...'}>
<Hit(sampleindex-2019.10/egCp7G0BGh3E1Mmaq9bC): {'startTime': '2019-10-21T13:53:15.32923433+09:00', 'headers...'}>
<Hit(sampleindex-2019.10/hACo7G0BGh3E1MmaNsXk): {'headers': {'http_version': 'HTTP/1.1', 'http_user_agent': ...}>
<Hit(sampleindex-2019.10/VgCp7G0BGh3E1Mmae9Tv): {'headers': {'http_version': 'HTTP/1.1', 'http_user_agent': ...}>
<Hit(sampleindex-2019.10/nQGt7G0BGh3E1MmaFw8e): {'startTime': '2019-10-21T13:57:05.621300916+09:00', 'header...'}>
<Hit(sampleindex-2019.10/cwGv7G0BGh3E1Mma1Ddj): {'headers': {'http_version': 'HTTP/1.1', 'http_user_agent': ...}>
<Hit(sampleindex-2019.10/PgGv7G0BGh3E1MmaMzCA): {'startTime': '2019-10-21T13:59:11.83491578+09:00', 'headers...'}>
<Hit(sampleindex-2019.10/4wGw7G0BGh3E1MmaSjzb): {'headers': {'http_version': 'HTTP/1.1', 'http_user_agent': ...}>
<Hit(sampleindex-2019.10/cAGs7G0BGh3E1Mma_Q5Z): {'headers': {'http_version': 'HTTP/1.1', 'http_user_agent': ...}>
<Hit(sampleindex-2019.10/6AGw7G0BGh3E1Mma60OW): {'headers': {'http_version': 'HTTP/1.1', 'http_user_agent': ...}>

如果我想使用这个输出数据并做一些事情，比如遍历结果并将信息存储在字典中，我怎样才能尽可能轻松地使用 elasticsearch-dsl-py？

Answer 1

我在 GitHub docs (also at Read The Docs 中找到了这段摘录):

To specify the from/size parameters, use the Python slicing API:

s = s[10:20]

If you want to access all the documents matched by your query you can use the scan method which uses the scan/scroll elasticsearch API:

for hit in s.scan():
    print(hit.title)

Note that in this case the results won't be sorted.

使用 elasticsearch 遍历查询的所有结果

Loop over all results from query using elasticsearch

python

elasticsearch

elasticsearch-dsl

elasticsearch-dsl-py