当我配置 elasticsearch 输出时，Logstash 似乎没有启动

Question

我有这个 docker-compose.yml 文件放在下面。实际上我需要的是使用 http 输入插件来提供 csv 文件，这就是为什么它出现在配置中的原因。

我正在使用那条简单的 curl 行进行测试，只是为了测试基本功能是否有效，然后继续使用 csv 文件和其余配置。

version: '3.3'
services:
  logstash:
    image: docker.elastic.co/logstash/logstash:6.7.0
    command: bash -c 'bin/logstash-plugin install logstash-input-http && bin/logstash-plugin install logstash-output-elasticsearch && bin/logstash -e "input { http { port => 5044 } } filter { split {} csv {} } output { elasticsearch { hosts => '127.0.0.1:9200' } }"'
    links:
      - elasticsearch
    ports:
      - 5044:5044
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.7.0
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - "9200:9200"
  kibana:
    image: docker.elastic.co/kibana/kibana:6.7.0
    ports:
      - "5601:5601"

configs:
  logstash_config:
    file: ./configs/logstash.conf

volumes:
  esdata1:
    driver: local

当我尝试下面的命令时，它失败了，如下所示：(Windows 10)

curl 'http://127.0.0.1:5044/twitter/tweet/2' -Method Put -Body 'hello2'
curl : The underlying connection was closed: The connection was closed unexpectedly.
At line:1 char:1
+ curl 'http://127.0.0.1:5044/twitter/tweet/2' -Method Put -Body 'hello ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

但是如果在我使用这个 docker_compose.yml 文件时有效：

version: '3.3'
services:
  logstash:
    image: docker.elastic.co/logstash/logstash:6.7.0
    command: bash -c 'bin/logstash -e "input { http { port => 5044 } } filter { split {} csv {} } output { stdout { codec => rubydebug} }" && bin/logstash-plugin install logstash-input-http && bin/logstash-plugin install logstash-output-elasticsearch'
    links:
      - elasticsearch
    ports:
      - 5044:5044
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.7.0
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - "9200:9200"
  kibana:
    image: docker.elastic.co/kibana/kibana:6.7.0
    ports:
      - "5601:5601"

configs:
  logstash_config:
    file: ./configs/logstash.conf

volumes:
  esdata1:
    driver: local

我启动了相同的命令，现在可以使用了。

 curl 'http://127.0.0.1:5044/twitter/tweet/2' -Method Put -Body 'hello2'


StatusCode        : 200
StatusDescription : OK
Content           : ok
RawContent        : HTTP/1.1 200 OK
                    Content-Length: 2
                    Content-Type: text/plain

                    ok
Forms             : {}
Headers           : {[Content-Length, 2], [Content-Type, text/plain]}
Images            : {}
InputFields       : {}
Links             : {}
ParsedHtml        : mshtml.HTMLDocumentClass
RawContentLength  : 2

我迫切需要将数据传递给elastic 欢迎任何关于问题的想法

提前致谢。

Answer 1

首先，您不必安装logstash 插件。包括电池。可以列出插件：bin/logstash-plugin list

接下来，利用 docker 服务发现。您的服务 elasticsearch 可从您的 logstash 容器中解析。容器中的 Localhost 解析为 - container localhost，而不是 host localhost。

HOST localhost =! CONTAINER localhost (in bridge mode)

工作docker-compose.yml

version: '3.3'
services:
  logstash:
    image: docker.elastic.co/logstash/logstash:6.7.0
    command: bash -c 'bin/logstash -e "input { http { port => 5044 } } filter { split {} csv {} } output { elasticsearch { hosts => [\"elasticsearch:9200\"] } }"'
    networks:
      - elastic-net
    ports:
      - 5044:5044

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:6.7.0
    environment:
      - cluster.name=docker-cluster
      - bootstrap.memory_lock=true
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    networks:
      - elastic-net
    ports:
      - "9200:9200"
  kibana:
    image: docker.elastic.co/kibana/kibana:6.7.0
    networks:
      - elastic-net
    ports:
      - "5601:5601"

networks:
  elastic-net:

configs:
  logstash_config:
    file: ./configs/logstash.conf

volumes:
  esdata1:
    driver: local

links 是一个 docker 遗留结构。而是使用 networks 结构。

祝你好运。

当我配置 elasticsearch 输出时，Logstash 似乎没有启动

Logstash seems not to be started when I configure elasticsearch output

elasticsearch

logstash

kibana

docker-compose