如何使用 deviantony/docker-elk 将 CSV 或 JSON 数据导入 Elasticsearch

Question

几天前我才开始使用 elasticsearch 和 docker，在将数据导入 elasticsearch 时遇到了一些问题。我正在使用的弹性堆栈回购是这样的：https://github.com/deviantony/docker-elk

我尝试按照我在网上找到的教程进行操作：https://www.bmc.com/blogs/elasticsearch-load-csv-logstash/ 但是当我加载 kibana 时找不到任何索引。

这是我所做的。我下载了一个示例数据并将其存储在根目录下名为 data 的文件夹中。在 docker-compose.yml 文件中，我做了一个指向我的外部数据文件夹的绑定挂载。

elasticsearch:
    build:
      context: elasticsearch/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./elasticsearch/config/elasticsearch.yml
        target: /usr/share/elasticsearch/config/elasticsearch.yml
        read_only: true
      - type: bind
        source: ./data
        target: /usr/share/elasticsearch/data
    ports:
      - "9200:9200"
      - "9300:9300"
    environment:
      ES_JAVA_OPTS: "-Xmx256m -Xms256m"
      ELASTIC_PASSWORD: password
      # Use single node discovery in order to disable production mode and avoid bootstrap checks
      # see https://www.elastic.co/guide/en/elasticsearch/reference/current/bootstrap-checks.html
      discovery.type: single-node
    networks:
      - elk

在我的 logstash.conf 文件下。这是我改变的：

input {
  tcp {
    port => 5000
  }
  file {
    path => "/usr/share/elasticsearch/data/conn250K.csv"
    start_position => "beginning"
  }
}

filter {
  csv {
    columns => [ "record_id", "duration", "src_bytes", "dest_bytes" ]
  }
}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    user => "elastic"
    password => "password"
    index => "network"
  }
}

启动后，在终端中执行 "docker-compose up" 命令，我找不到任何要在 Kibana 中创建的索引模式，因为没有生成索引。我不知道哪里出了问题。

Answer 1

尝试将外部数据文件夹绑定挂载到 logstash 容器而不是 elasticsearch。

  logstash:
    build:
      context: logstash/
      args:
        ELK_VERSION: $ELK_VERSION
    volumes:
      - type: bind
        source: ./logstash/config/logstash.yml
        target: /usr/share/logstash/config/logstash.yml
        read_only: true
      - type: bind
        source: ./logstash/pipeline
        target: /usr/share/logstash/pipeline
        read_only: true
      - type: bind
        source: ./data
        target: /usr/share/logstash/data
        read_only: true
    ports:
      - "5000:5000/tcp"
      - "5000:5000/udp"
      - "9600:9600"
    environment:
      LS_JAVA_OPTS: "-Xmx256m -Xms256m"
    networks:
      - elk
    depends_on:
      - elasticsearch

如何使用 deviantony/docker-elk 将 CSV 或 JSON 数据导入 Elasticsearch

How to import CSV or JSON data into Elasticsearch using deviantony/docker-elk

elasticsearch

logstash

docker

kibana

docker-compose