转换来自 http 源连接器的 json 结果

Question

我已经使用 http 源连接器获取数据并将记录发布到 kafka“testdata”主题中。我有以下连接作业配置，

curl -X POST http://localhost:8083/connectors -H "Content-Type: application/json" -H "Accept: application/json" -d ' {
"name": "https-connector",
"config": {
       "connector.class": "com.github.castorm.kafka.connect.http.HttpSourceConnector",
       "tasks.max": 1,
       "http.request.headers": "Accept:application/json",
       "http.request.url": "*********************",
       "kafka.topic": "testdata"
 }

}'

http API 结果是，

{"fields":[
   {"id":"displayName","type":"text","name":"Display name"},
   {"id":"firstName","type":"text","name":"First name"}
  ],
"employees":[
   {"id":"4","displayName":"Charlotte Abbott","firstName":"Charlotte"},
   {"id":"5","displayName":"Ashley Adams","firstName":"Ashley"}
  ]
}

我在 kafka 主题中得到以下输出。它在消息的开头和结尾有一些二进制字符串（开始：r，结束：H5ee9b388-2063-3c78-b30a-05f2292ec7bf��^）。 “5ee9b388-2063-3c78-b30a-05f2292ec7bf”是消息的关键。

r{"fields":[{"id":"displayName","type":"text","name":"Display name"},{"id":"firstName","type":"text","name":"First name"}],"employees":[{"id":"4","displayName":"Charlotte Abbott","firstName":"Charlotte"},{"id":"5","displayName":"Ashley Adams","firstName":"Ashley"}]}H5ee9b388-2063-3c78-b30a-05f2292ec7bf�����^

我想在 kafka 中将其作为单独的消息获取，

{"id":"4","displayName":"Charlotte Abbott","firstName":"Charlotte"}
{"id":"5","displayName":"Ashley Adams","firstName":"Ashley"}

Answer 1

It has some binary string in the starting and ending of the message

如果您使用的是 StringConverter 或 JSONConverter，则不应发生这种情况，除非您使用的 API 实际上正在返回它。

would like to get this as separate message in kafka

Kafka Connect API一般不能拆分消息。 Connector 对一条消息发送一个 API 响应；您需要使用辅助流处理器将数组拆分为单独的记录

转换来自 http 源连接器的 json 结果

Convert the json result from http source connector

http

apache-kafka

apache-kafka-connect