docker stack/swarm 弹性

Question

我有两个节点

[ra@speechanalytics-test ~]$ docker node ls
ID                            HOSTNAME                  STATUS              AVAILABILITY        MANAGER STATUS      ENGINE VERSION
mlwwmkdlzbv0zlapqe1veq3uq     speechanalytics-preprod   Ready               Active                                  18.09.3
se717p88485s22s715rdir9x2 *   speechanalytics-test      Ready               Active              Leader              18.09.3

我正在尝试运行堆栈中有弹性的容器。这是我的 docker-compose.yml 文件

version: '3.4'
services:
  elastic:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.7.0
    environment:
      - cluster.name=single-node
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - esdata:/usr/share/elasticsearch/data
    deploy:
      placement:
        constraints:
          - node.hostname==speechanalytics-preprod

volumes:
  esdata:
    driver: local

从 docker 堆栈开始后

docker stack deploy preprod -c docker-compose.yml

容器在 20 秒内崩溃

docker service logs preprod_elastic 
...
   | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   | OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0
   | [2019-04-03T16:41:30,044][WARN ][o.e.b.JNANatives         ] [unknown] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
   | [2019-04-03T16:41:30,049][WARN ][o.e.b.JNANatives         ] [unknown] This can result in part of the JVM being swapped out.
   | [2019-04-03T16:41:30,049][WARN ][o.e.b.JNANatives         ] [unknown] Increase RLIMIT_MEMLOCK, soft limit: 16777216, hard limit: 16777216
   | [2019-04-03T16:41:30,050][WARN ][o.e.b.JNANatives         ] [unknown] These can be adjusted by modifying /etc/security/limits.conf, for example:
   | OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
   |     # allow user 'elasticsearch' mlockall
   | OpenJDK 64-Bit Server VM warning: UseAVX=2 is not supported on this CPU, setting it to UseAVX=0
   |     elasticsearch soft memlock unlimited
   | [2019-04-03T16:41:02,949][WARN ][o.e.b.JNANatives         ] [unknown] Unable to lock JVM Memory: error=12, reason=Cannot allocate memory
   |     elasticsearch hard memlock unlimited
   | [2019-04-03T16:41:02,954][WARN ][o.e.b.JNANatives         ] [unknown] This can result in part of the JVM being swapped out.
   | [2019-04-03T16:41:30,050][WARN ][o.e.b.JNANatives         ] [unknown] If you are logged in interactively, you will have to re-login for the new limits to take effect.
   | [2019-04-03T16:41:02,954][WARN ][o.e.b.JNANatives         ] [unknown] Increase RLIMIT_MEMLOCK, soft limit: 16777216, hard limit: 16777216
preprod

在我的两个节点上

ra@speechanalytics-preprod:~$ sysctl vm.max_map_count
vm.max_map_count = 262144

有什么解决办法吗？

Answer 1

您在 Elasticsearch 中看到的内存锁错误是一个常见问题，并非使用 Docker 所独有，而是在 Elasticsearch 被告知锁定其内存但无法执行时发生。您可以通过从 docker-compose.yml 文件中删除以下环境变量来避免该错误：

- bootstrap.memory_lock=true

Memlock 可以与 Docker 群模式一起使用，但有一些注意事项。

并非所有适用于 docker-compose（Docker Compose）的选项都适用于 docker stack deploy（Docker Swarm 模式），反之亦然，尽管两者共享 docker-编写 YAML 语法。一个这样的选项是 ulimits:，当它与 docker 堆栈部署一起使用时，将被忽略并显示一条警告消息，如下所示：

Ignoring unsupported options: ulimits

我的猜测是，对于您的 docker-compose.yml 文件，Elasticsearch 在 docker-compose up 上运行良好，但在 docker stack deploy.

上运行不正常

使用Docker Swarm Mode，默认情况下，您定义的Elasticsearch实例会遇到memlock问题。目前，官方支持 docker swarm 服务的 ulimits 设置 not yet。不过，有一些方法可以解决这个问题。

如果主机是 Ubuntu，则可以在 docker 服务中启用无限内存锁（请参阅 here and here）。这可以通过命令实现：

echo -e "[Service]\nLimitMEMLOCK=infinity" | SYSTEMD_EDITOR=tee systemctl edit docker.service
systemctl daemon-reload
systemctl restart docker

然而，将 memlock 设置为无穷大并非没有缺点，正如 Elastic 自己所阐明的那样 here。

根据我的测试，该解决方案适用于 Docker 18.06，但不适用于 18.09。鉴于不一致和 Elasticsearch 启动失败的可能性，更好的选择是不在 Swarm 上部署时将 memlock 与 Elasticsearch 一起使用。相反，您可以选择 Elasticsearch Docs 中提到的任何其他方法来获得类似的结果。

docker stack/swarm 弹性

Elastic in docker stack/swarm

elasticsearch

docker

docker-swarm