Logstash 不在弹性搜索中创建索引
Logstash is not creating index in elastic search
它没有在控制台上显示任何错误。这是我的命令 运行 ( On Windows 10. ) -
logstash --verbose -f logstash-sample.conf
这是我的 logstash-sample.conf 文件 -
input {
file {
path => "C:\Users739\Documents\IIT\CSP586\tutorial\project\ChicagoSocialHub\backend-build-divvy-status\divvy_stations_status.csv"
start_position => "beginning"
}
}
filter {
csv{
separator => ","
columns => ["altitude", "availableBikes", "availableDocks", "city", "id", "is_renting", "kioskType", "landMark", "lastCommunicationTime", "latitude", "location", "longitude", "postalCode", "stAddress1", "stAddress2", "stationName", "status", "statusKey", "statusValue", "testStation", "totalDocks"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "divvy_stations_status"
document_type => "status"
}
stdout {
codec => rubydebug
}
}
这里是 logstash 控制台输出 -
C:\Users739\Documents\IIT\CSP586\logstash-6.6.2\bin>logstash --verbose -f logstash-sample.conf
Sending Logstash logs to C:/Users/17739/Documents/IIT/CSP586/logstash-6.6.2/logs which is now configured via log4j2.properties
[2019-03-17T12:56:36,728][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-03-17T12:56:36,745][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.2"}
[2019-03-17T12:56:41,603][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch index=>"divvy_stations_status", id=>"f84c43181aab6f7bf9e89c0412ada5b5ead116534f6661194800152751a28e87", hosts=>[//localhost:9200], document_type=>"status", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_1264be19-323c-4896-8214-929f15a74251", enable_metric=>true, charset=>"UTF-8">, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ilm_enabled=>false, ilm_rollover_alias=>"logstash", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", action=>"index", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2019-03-17T12:56:43,234][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-03-17T12:56:43,548][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-03-17T12:56:43,695][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-03-17T12:56:43,735][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-03-17T12:56:43,739][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-03-17T12:56:43,768][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-03-17T12:56:43,782][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2019-03-17T12:56:43,801][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-03-17T12:56:44,323][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/Users/17739/Documents/IIT/CSP586/logstash-6.6.2/data/plugins/inputs/file/.sincedb_6f34c293ff88e0ad3c31e4a0f32e43d9", :path=>["C:\Users\17739\Documents\IIT\CSP586\tutorial\project\ChicagoSocialHub\backend-build-divvy-status\divvy_stations_status.csv"]}
[2019-03-17T12:56:44,369][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2e991954 run>"}
[2019-03-17T12:56:44,440][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-03-17T12:56:44,441][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-03-17T12:56:44,786][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
在 运行 logstash --verbose -f logstash-sample.conf 命令之后，我的期望是这个新索引 divvy_stations_status 将在弹性搜索索引中可见（也用 kibana 检查过，也没有出现在那里） .这是一个真正的期望吗？这是我的
输出-
解决方案是在文件路径中给出 forward slash / 而不是 backward slash \。这很奇怪，因为在 windows 中我们总是使用正斜杠作为路径分隔符。更奇怪的是它从来没有抛出像 'File not found or something'
这样的错误
因此在 logstash-sample.conf 文件中使用以下 -
path => "C:/Users/17739/Documents/IIT/CSP586/tutorial/project/ChicagoSocialHub/backend-build-divvy-status/divvy_stations_status.csv"
@sapy ,
当您尝试在 Windows 平台中 运行 logstash 时，
您必须将文件路径更改为
C:/Users/17739/Documents/IIT/CSP586/tutorial/project/ChicagoSocialHub/backend-build-divvy-status/divvy_stations_status.csv
即用正斜杠代替反斜杠。
第二个
您需要指定
sincedb_path => "NUL"
在输入插件中
它跟踪监视的日志文件的当前位置。
这两个是开发 logstash conf 文件时的好习惯。
它没有在控制台上显示任何错误。这是我的命令 运行 ( On Windows 10. ) -
logstash --verbose -f logstash-sample.conf
这是我的 logstash-sample.conf 文件 -
input {
file {
path => "C:\Users739\Documents\IIT\CSP586\tutorial\project\ChicagoSocialHub\backend-build-divvy-status\divvy_stations_status.csv"
start_position => "beginning"
}
}
filter {
csv{
separator => ","
columns => ["altitude", "availableBikes", "availableDocks", "city", "id", "is_renting", "kioskType", "landMark", "lastCommunicationTime", "latitude", "location", "longitude", "postalCode", "stAddress1", "stAddress2", "stationName", "status", "statusKey", "statusValue", "testStation", "totalDocks"]
}
}
output {
elasticsearch {
hosts => ["localhost:9200"]
index => "divvy_stations_status"
document_type => "status"
}
stdout {
codec => rubydebug
}
}
这里是 logstash 控制台输出 -
C:\Users739\Documents\IIT\CSP586\logstash-6.6.2\bin>logstash --verbose -f logstash-sample.conf
Sending Logstash logs to C:/Users/17739/Documents/IIT/CSP586/logstash-6.6.2/logs which is now configured via log4j2.properties
[2019-03-17T12:56:36,728][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2019-03-17T12:56:36,745][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.6.2"}
[2019-03-17T12:56:41,603][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document types are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearch index=>"divvy_stations_status", id=>"f84c43181aab6f7bf9e89c0412ada5b5ead116534f6661194800152751a28e87", hosts=>[//localhost:9200], document_type=>"status", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_1264be19-323c-4896-8214-929f15a74251", enable_metric=>true, charset=>"UTF-8">, workers=>1, manage_template=>true, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>false, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2, retry_max_interval=>64, retry_on_conflict=>1, ilm_enabled=>false, ilm_rollover_alias=>"logstash", ilm_pattern=>"{now/d}-000001", ilm_policy=>"logstash-policy", action=>"index", ssl_certificate_verification=>true, sniffing=>false, sniffing_delay=>5, timeout=>60, pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2019-03-17T12:56:43,234][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-03-17T12:56:43,548][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2019-03-17T12:56:43,695][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2019-03-17T12:56:43,735][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-03-17T12:56:43,739][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-03-17T12:56:43,768][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2019-03-17T12:56:43,782][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2019-03-17T12:56:43,801][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-03-17T12:56:44,323][INFO ][logstash.inputs.file ] No sincedb_path set, generating one based on the "path" setting {:sincedb_path=>"C:/Users/17739/Documents/IIT/CSP586/logstash-6.6.2/data/plugins/inputs/file/.sincedb_6f34c293ff88e0ad3c31e4a0f32e43d9", :path=>["C:\Users\17739\Documents\IIT\CSP586\tutorial\project\ChicagoSocialHub\backend-build-divvy-status\divvy_stations_status.csv"]}
[2019-03-17T12:56:44,369][INFO ][logstash.pipeline ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2e991954 run>"}
[2019-03-17T12:56:44,440][INFO ][filewatch.observingtail ] START, creating Discoverer, Watch with file and sincedb collections
[2019-03-17T12:56:44,441][INFO ][logstash.agent ] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-03-17T12:56:44,786][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
在 运行 logstash --verbose -f logstash-sample.conf 命令之后，我的期望是这个新索引 divvy_stations_status 将在弹性搜索索引中可见（也用 kibana 检查过，也没有出现在那里） .这是一个真正的期望吗？这是我的
输出-
解决方案是在文件路径中给出 forward slash / 而不是 backward slash \。这很奇怪，因为在 windows 中我们总是使用正斜杠作为路径分隔符。更奇怪的是它从来没有抛出像 'File not found or something'
因此在 logstash-sample.conf 文件中使用以下 -
path => "C:/Users/17739/Documents/IIT/CSP586/tutorial/project/ChicagoSocialHub/backend-build-divvy-status/divvy_stations_status.csv"
@sapy ,
当您尝试在 Windows 平台中 运行 logstash 时，
您必须将文件路径更改为
C:/Users/17739/Documents/IIT/CSP586/tutorial/project/ChicagoSocialHub/backend-build-divvy-status/divvy_stations_status.csv
即用正斜杠代替反斜杠。 第二个
您需要指定
sincedb_path => "NUL"
在输入插件中
它跟踪监视的日志文件的当前位置。
这两个是开发 logstash conf 文件时的好习惯。