Mongodb 主题未看到 Kafka 消息

Mongodb Kafka messages not seen by topic

我遇到了我的主题,尽管 运行 并且操作没有注册我的 MongoDB 中发生的事件。

每次我 insert/modify 记录时,我不再从 kafka-console-consumer 命令获取日志。

有没有办法清除 Kafka 的 cache/offset? 源和接收器连接已启动并且 运行。整个集群也很健康,事情是一切都像往常一样工作,但每隔几周我就会看到它回来,或者当我从其他位置登录到我的 Mongo 云时。

--partition 0 参数没有帮助,也将 retention_ms 更改为 1

我检查了两个连接器的状态并得到 RUNNING:

curl localhost:8083/connectors | jq

curl localhost:8083/connectors/monit_people/status | jq

运行 docker-compose logs connect 我发现:

    WARN Failed to resume change stream: Resume of change stream was not possible, as the resume point may no longer be in the oplog. 286

If the resume token is no longer available then there is the potential for data loss.
Saved resume tokens are managed by Kafka and stored with the offset data.
 
When running Connect in standalone mode offsets are configured using the:
`offset.storage.file.filename` configuration.
When running Connect in distributed mode the offsets are stored in a topic.

Use the `kafka-consumer-groups.sh` tool with the `--reset-offsets` flag to reset offsets.

Resetting the offset will allow for the connector to be resume from the latest resume token. 
Using `copy.existing=true` ensures that all data will be outputted by the connector but it will duplicate existing data.
Future releases will support a configurable `errors.tolerance` level for the source connector and make use of the `postBatchResumeToken

问题需要对 Confluent Platform 进行更多练习,因此现在我通过删除整个容器重新构建了整个环境:

docker system prune -a -f --volumes

docker container stop $(docker container ls -a -q -f "label=io.confluent.docker").

在 运行 docker-compose up -d 之后一切正常。