geoip.location 在 apachelog 的 logstash 解析后丢失，Kibana 地图可视化工具不工作

Question

我使用 logstash 将数据插入到弹性搜索中，conf 文件看起来像

  grok {
             match => [
         "message" , "%{COMBINEDAPACHELOG}+%{GREEDYDATA:extra_fields}",
         "message" , "%{COMMONAPACHELOG}+%{GREEDYDATA:extra_fields}"
         ]
         overwrite => [ "message" ]
      }
  date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
  geoip {
      source => "clientip"
    }

现在在 Elastic Search 中映射是按

完成的

{                                                                                                                                                        
   "response" => "200",                                                                                                                              
      "geoip" => {                                                                                                                                   
          "timezone" => "Asia/Kolkata",                                                                                                              
     "country_code3" => "IN",                                                                                                                        
          "location" => {                                                                                                                            
        "lon" => 80.2833,                                                                                                                            
        "lat" => 13.0833                                                                                                                             
    },                                                                                                                                               
       "region_code" => "TN",                                                                                                                        
      "country_name" => "India",                                                                                                                     
         "longitude" => 80.2833,                                                                                                                     
         "city_name" => "Chennai",                                                                                                                   
       "region_name" => "Tamil Nadu",                                                                                                                
          "latitude" => 13.0833,                                                                                                                     
    "continent_code" => "AS",                                                                                                                        
       "postal_code" => "600073",                                                                                                                    
     "country_code2" => "IN",                                                                                                                        
                "ip" => "122.15.151.189"                                                                                                             
},                                                                                                                                                   
  "timestamp" => "31/May/2019:05:12:22 -0700",                                                                                                       
    "request" => "/favicon.ico",                                                                                                                     
      "ident" => "-",                                                                                                                                
       "auth" => "-",                                                                                                                                
      "agent" => "\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36\"",          
   "referrer" => "\"http://example.com/\"",            
 "@timestamp" => 2019-05-31T12:12:22.000Z,                                                                                                           
"httpversion" => "1.1",                                                                                                                              
       "verb" => "GET"                                                                                                                               

}

现在，当尝试在 KIBANA 中加载 make 可视化工具时，geoHash 不起作用，因为 geoip.location 未定义为 "type"："geo_point"。我已经插入了整个数据。那么我能做些什么来修改映射和更新索引。 我的理解是我需要在 grok 中添加一个额外的字段并制作 "geoip.location" 字段并再次重新解析整个日志。但是有没有办法在现有的弹性索引中创建一个列并从 2 列合并数据？

Like geoip.location2 merged from geoip.location.lon and location.lat  

Answer 1

通常您不能修改索引中已经存在的字段的映射
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html#_updating_existing_field_mappings

但是，您可以通过添加另一个字段来更新索引的映射。您可以查看 exact syntax based on the version of Elastic that you're using，但对我来说，在 Elastic 6.2 上，以下内容有效：

PUT geo_test1/_mapping/geo_test1
{
  "properties": {
    "geoip": {
      "properties": {
        "location_geo": {
          "type": "geo_point"
        }
      }
    }
  }
}

然后，您可以使用 update_by_query 和一个简单的小脚本来根据 geoip.location 中已有的数据在新的 "location_geo" 字段中填充数据：

POST geo_test1/_update_by_query
{
  "script": {
    "source": "String lat = ctx._source.geoip.location.lat.toString(); String lon = ctx._source.geoip.location.lon.toString(); ctx._source.geoip.location_geo = lat + ',' + lon",
    "lang": "painless"
  },
  "query": {
    "match_all": {}
  }
}