显示和更改参数 threadpool.bulk.queue_size

Question

我使用的是 elasticsearch 版本 6.7.2，但出现错误 429

{
    "error": {
        "root_cause": [
            {
                "type": "remote_transport_exception",
                "reason": "[localhost][localhost:9300][indices:data/write/bulk[s]]"
            }
        ],
        "type": "es_rejected_execution_exception",
        "reason": "rejected execution of processing of [682604930][indices:data/write/bulk[s][p]]: request: BulkShardRequest [[hboauth2-2020.04.20-000113][0]] containing [index {[hboauth2][_doc][yRuEnHEBn6z33bhuCGjZ], source[{\"Action\":\"Service UpdateUserStatus Start\",\"ClientID\":\"\",\"Code\":0,\"Component\":\"UpdateUserStatus\",\"CustomMessage\":\"\",\"Data\":\"null\",\"LogDate\":1587468830939,\"Message\":\"{OK false}\",\"Type\":\"Info\",\"UserID\":\"4c220ab2-282d-466f-a9f5-da0e59b7b803\"}]}], target allocation id: wZFZUPi5TnSYFg-BNonHug, primary term: 1 on EsThreadPoolExecutor[name = localhost/write, queue capacity = 200, org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor@744831b0[Running, pool size = 16, active threads = 16, queued tasks = 224, completed tasks = 264136589]]"
    },
    "status": 429
}

我找到了解决问题的方法，但是我的版本中找不到参数threadpool.bulk.queue_size。你能帮我找到和更改这个参数吗

Answer 1

如 official example 所示，您可以在 elasticsearch.config 中使用以下配置更改批量线程池的队列大小或更新集群设置。

注意，写入线程池用于批量请求，如同一 link 中所述：

write For single-document index/delete/update and bulk requests. Thread pool type is fixed with a size of # of available processors, queue_size of 200. The maximum size for this pool is 1 + # of available processors.

thread_pool:
    write:
        size: 30. --> no if threads in your case, its 16 as shown in exception
        queue_size: 400 --> here you can add `queue_size`

还提到了更改这些设置的说明

Changing a specific thread pool can be done by setting its type-specific parameters; for example, changing the number of threads in the write thread pool:

编辑：正如@val 在评论部分指出的那样，添加关于更改这些默认设置的免责声明。

请了解这些设置的技术细节和影响，因为通常不建议更改这些设置，更多信息请参见 this official blog

Answer 2

我正在寻找如何改变 thread_pool.write.queue_size 并找到 2 个解决方案第一

curl -XPUT  _cluster/settings -d '{
    "persistent" : {
        "thread_pool.write.queue_size" : <new_size>
    }
}'

第二次写入 elasticsearch.yaml 参数 thread_pool.write.queue_size

所有解决方案都适用于我的 6.2.7 版本吗？

显示和更改参数 threadpool.bulk.queue_size

show and change parameter threadpool.bulk.queue_size

threadpool

elasticsearch