Kafka Streams 应用程序消费者偏移量在重启时重置为最早

Question

我们有一个 Kafka Streams 应用程序出现了奇怪的行为。当作业被终止并重新启动时，消费者组会随机将其偏移量重置为最早，并且所有旧记录都会重新处理。

是否有任何具体的事情需要完成而我们遗漏了？

"buffered.records.per.partition": 512000, 
"cache.max.bytes.buffering": 134217728, 
"commit.interval.ms": 30000, 
"retries": 2147483647, 
"acks": "all", 
"processing.guarantee": "exactly_once", 
"compression.type": "snappy", 
"auto.offset.reset": "latest"

Answer 1

嗯，您已经设置了 "auto.offset.reset": "latest"，很明显属性没有被应用。

To guarantee at-least-once processing semantics and turn off auto commits, Kafka Streams overrides enable.auto.commit consumer config value to false. Consumers will only commit explicitly via commitSync calls when the Kafka Streams library or a user decides to commit the current processing state

- docs

因此，您似乎还没有对任何流项目执行终端操作以使其进行同步提交。

Kafka Streams 应用程序消费者偏移量在重启时重置为最早

Kafka Streams Application consumer offset getting reset to Earliest on restart

apache-kafka

kafka-consumer-api

apache-kafka-streams