ElasticSearch 索引问题，无法解析时间戳

Question

我是 ELK 的新手。我在 Elasticsearch

中创建了索引

{
  "logstash": {
    "aliases": {},
    "mappings": {
      "log": {
        "dynamic_templates": [
          {
            "message_field": {
              "path_match": "message",
              "match_mapping_type": "string",
              "mapping": {
                "norms": false,
                "type": "text"
              }
            }
          },
          {
            "string_fields": {
              "match": "*",
              "match_mapping_type": "string",
              "mapping": {
                "fields": {
                  "keyword": {
                    "type": "keyword"
                  }
                },
                "norms": false,
                "type": "text"
              }
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date"
          },
          "@version": {
            "type": "keyword",
            "include_in_all": false
          },
          "activity": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "beat": {
            "properties": {
              "hostname": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword"
                  }
                }
              },
              "name": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword"
                  }
                }
              },
              "version": {
                "type": "text",
                "norms": false,
                "fields": {
                  "keyword": {
                    "type": "keyword"
                  }
                }
              }
            }
          },
          "filename": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "host": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "input_type": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "message": {
            "type": "text",
            "norms": false
          },
          "offset": {
            "type": "long"
          },
          "source": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "tags": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "timestamp": {
            "type": "date",
            "include_in_all": false,
            "format": "YYYY-MM-DD HH:mm:ss.SSS"
          },
          "type": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          },
          "user": {
            "type": "text",
            "norms": false,
            "fields": {
              "keyword": {
                "type": "keyword"
              }
            }
          }
        }
      }
    },
    "settings": {
      "index": {
        "creation_date": "1488805244467",
        "number_of_shards": "1",
        "number_of_replicas": "0",
        "uuid": "5ijhh193Tr6y_hxaQrW9kg",
        "version": {
          "created": "5020199"
        },
        "provided_name": "logstash"
      }
    }
  }
}

下面是我的logstash配置

input{
    beats{
        port=>5044
    }
}filter{
    grok{
        match=>{"message" => "\[%{TIMESTAMP_ISO8601:timestamp}\] ALL AUDIT: User \[%{GREEDYDATA:user}\] is %{GREEDYDATA:activity} \[%{GREEDYDATA:filename}\] for transfer."}
    }
}output{
    elasticsearch{
        hosts=>"localhost:9200"
        index=> "logstash"
    }

示例数据

[2017-03-05 12:37:21.465] ALL AUDIT: User [user1] is opening file [filename1] for transfer.

但是当我通过 filebeat > logstash > elasticsearch 加载文件时在 elasticsearch 中，我遇到了以下错误

org.elasticsearch.index.mapper.MapperParsingException: failed to parse [timestamp]
Caused by: java.lang.IllegalArgumentException: Invalid format: "2017-03-05T12:36:33.606" is malformed at "12:36:33.606"
    at org.joda.time.format.DateTimeParserBucket.doParseMillis(DateTimeParserBucket.java:187) ~[joda-time-2.9.5.jar:2.9.5]

请帮忙，我应该配置什么时间戳格式？

Answer 1

在您的时间戳映射中，您将格式指定为 "format": "YYYY-MM-DD HH:mm:ss.SSS" 这里您通过节拍发送的格式不一样，请检查：2017-03-05T12:36:33.606

这就是 Elastic 抱怨格式的原因。您的格式应为："YYYY-MM-DD'T'HH:mm:ss.SSS"（注意大写 T）

有关详细信息，请参阅文档：https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-date-format.html

ElasticSearch 索引问题，无法解析时间戳

ElasticSearch indexing issue ,failed to parse timestamp

datetime

jodatime

elasticsearch

logstash