无法将主题映射到 kafka elasticsearch 连接器中的指定索引

Question

试图将主题 "name: localtopic" 映射到索引 "name:indexoftopic"，它在弹性搜索中创建两个新索引 "localtopic and indexoftopic" 和主题数据仅在主题名称索引 "localtopic" 中可见，连接器中未显示错误（分布式模式）

我的配置是

 "config" : {
  "connector.class" : "io.confluent.connect.elasticsearch.ElasticsearchSinkConnector",
  "tasks.max" : "1",
  "topics" : "localtopic", 
  "topic.index.map" : "localtopic:indexoftopic",
  "connection.url" : "aws elasticsearch url",
  "type.name" : "event",
  "key.ignore" : "false",
  "schema.ignore" : "true",
  "schemas.enable" : "false",
  "transforms" : "InsertKey,extractKey",
  "transforms.InsertKey.type" : "org.apache.kafka.connect.transforms.ValueToKey",
  "transforms.InsertKey.fields" : "event-id",
  "transforms.extractKey.type" : "org.apache.kafka.connect.transforms.ExtractField$Key",
  "transforms.extractKey.field" : "event-id"
 }

索引 name:indexoftopic 是在 elasticsearch 中创建的，但数据被 index_name:localtopic 看到 kafkaversion:2.3 connectorversion:5 elasticsearchversion:3.2.0

即使在日志 INFO -- topics.regex = "" 中，我也不知道他的选项，任何人都可以建议。这个怎么用？？？

Answer 1

如果您查看配置选项，建议您改用 RegexRouter 转换

topic.index.map

This option is now deprecated. A future version may remove it completely. Please use single message transforms, such as RegexRouter, to map topic names to index names.

Answer 2

下面对我有用，但是，我只将 1 个主题映射到不同的索引名称

"transforms": "addSuffix",  
"transforms.addSuffix.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.addSuffix.regex": "topic1.*",
"transforms.addSuffix.replacement": "index1",

所以，以上转换任何主题，例如 topic1, topic1-test, topic1<anystring> 将映射到 index1

或者，您也可以使用主题名称替换，通过如下更改最后两行，这将选择

    "transforms.addSuffix.regex": "topic.*",
    "transforms.addSuffix.replacement": "index",

基本上，您可以使用正则表达式替换部分或完整的主题名称。

无法将主题映射到 kafka elasticsearch 连接器中的指定索引

unable to map topics to specified index in kafka elasticsearch connector

elasticsearch

apache-kafka

apache-kafka-connect