Kafka - 重置分区错误
Kafka - Reset partition errors
我使用 Kafka Streams 进行数据处理。
卡夫卡 1.1
我在 kafka 中生成的数据速率为 35k rps，并且流的消耗速率也与通过 kafka 消费者指标看到的速率相同。但我经常看到 reset partition errors，它表示提取偏移量超出范围。这基本上意味着我的消费者比正在非常积极地发生的日志删除慢。
我的日志保留时间是 168 小时，日志保留字节是 10G。这是我经常在我的消费者那里得到的日志。
我试图在网上搜索相关问题，但没有找到。
[sample-app-deploy-8c4fd5697-4xxbk sample-app] 09:28:24.291 [sample-app-0cf78aad-5faa-4197-853b-bfc08bb38f66-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-0cf78aad-5faa-4197-853b-bfc08bb38f66-StreamThread-1-consumer, groupId=sample-app] Fetch offset 116411050 is out of range for partition sample-topic-4, resetting offset
[sample-app-deploy-8c4fd5697-4xxbk sample-app] 09:28:24.292 [sample-app-0cf78aad-5faa-4197-853b-bfc08bb38f66-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-0cf78aad-5faa-4197-853b-bfc08bb38f66-StreamThread-1-consumer, groupId=sample-app] Resetting offset for partition sample-topic-4 to offset 116411058.
[sample-app-deploy-8c4fd5697-qmjnd sample-app] 09:28:24.306 [sample-app-e10caa03-b881-47f2-b1ce-e9513c12a98c-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-e10caa03-b881-47f2-b1ce-e9513c12a98c-StreamThread-1-consumer, groupId=sample-app] Fetch offset 237000869 is out of range for partition sample-topic-7, resetting offset
[sample-app-deploy-8c4fd5697-qmjnd sample-app] 09:28:24.307 [sample-app-e10caa03-b881-47f2-b1ce-e9513c12a98c-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-e10caa03-b881-47f2-b1ce-e9513c12a98c-StreamThread-1-consumer, groupId=sample-app] Resetting offset for partition sample-topic-7 to offset 237000871.
[sample-app-deploy-8c4fd5697-n5pw8 sample-app] 09:29:56.808 [sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1-consumer, groupId=sample-app] Fetch offset 471945398 is out of range for partition sample-topic-0, resetting offset
[sample-app-deploy-8c4fd5697-n5pw8 sample-app] 09:29:56.810 [sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1-consumer, groupId=sample-app] Resetting offset for partition sample-topic-0 to offset 471945403.
[sample-app-deploy-8c4fd5697-n5pw8 sample-app] 09:34:56.804 [sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1-consumer, groupId=sample-app] Fetch offset 474036996 is out of range for partition sample-topic-0, resetting offset
[sample-app-deploy-8c4fd5697-n5pw8 sample-app] 09:34:56.805 [sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1-consumer, groupId=sample-app] Resetting offset for partition sample-topic-0 to offset 474036997.
[sample-app-deploy-8c4fd5697-cjccm sample-app] 09:39:10.659 [sample-app-a7e7c388-0dd4-45e8-8d5a-3a84effb7dfd-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-a7e7c388-0dd4-45e8-8d5a-3a84effb7dfd-StreamThread-1-consumer, groupId=sample-app] Fetch offset 236702790 is out of range for partition sample-topic-5, resetting offset
任何人都可以帮助指出发生问题的地方吗？
问题出在时间戳上。我们以秒而不是毫秒为单位向分区发送事件时间，这导致时间保留非常激进。
我使用 Kafka Streams 进行数据处理。
卡夫卡 1.1
我在 kafka 中生成的数据速率为 35k rps，并且流的消耗速率也与通过 kafka 消费者指标看到的速率相同。但我经常看到 reset partition errors，它表示提取偏移量超出范围。这基本上意味着我的消费者比正在非常积极地发生的日志删除慢。
我的日志保留时间是 168 小时，日志保留字节是 10G。这是我经常在我的消费者那里得到的日志。
我试图在网上搜索相关问题，但没有找到。
[sample-app-deploy-8c4fd5697-4xxbk sample-app] 09:28:24.291 [sample-app-0cf78aad-5faa-4197-853b-bfc08bb38f66-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-0cf78aad-5faa-4197-853b-bfc08bb38f66-StreamThread-1-consumer, groupId=sample-app] Fetch offset 116411050 is out of range for partition sample-topic-4, resetting offset
[sample-app-deploy-8c4fd5697-4xxbk sample-app] 09:28:24.292 [sample-app-0cf78aad-5faa-4197-853b-bfc08bb38f66-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-0cf78aad-5faa-4197-853b-bfc08bb38f66-StreamThread-1-consumer, groupId=sample-app] Resetting offset for partition sample-topic-4 to offset 116411058.
[sample-app-deploy-8c4fd5697-qmjnd sample-app] 09:28:24.306 [sample-app-e10caa03-b881-47f2-b1ce-e9513c12a98c-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-e10caa03-b881-47f2-b1ce-e9513c12a98c-StreamThread-1-consumer, groupId=sample-app] Fetch offset 237000869 is out of range for partition sample-topic-7, resetting offset
[sample-app-deploy-8c4fd5697-qmjnd sample-app] 09:28:24.307 [sample-app-e10caa03-b881-47f2-b1ce-e9513c12a98c-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-e10caa03-b881-47f2-b1ce-e9513c12a98c-StreamThread-1-consumer, groupId=sample-app] Resetting offset for partition sample-topic-7 to offset 237000871.
[sample-app-deploy-8c4fd5697-n5pw8 sample-app] 09:29:56.808 [sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1-consumer, groupId=sample-app] Fetch offset 471945398 is out of range for partition sample-topic-0, resetting offset
[sample-app-deploy-8c4fd5697-n5pw8 sample-app] 09:29:56.810 [sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1-consumer, groupId=sample-app] Resetting offset for partition sample-topic-0 to offset 471945403.
[sample-app-deploy-8c4fd5697-n5pw8 sample-app] 09:34:56.804 [sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1-consumer, groupId=sample-app] Fetch offset 474036996 is out of range for partition sample-topic-0, resetting offset
[sample-app-deploy-8c4fd5697-n5pw8 sample-app] 09:34:56.805 [sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-1db56df6-1dab-40d2-94c2-e412eff0ee09-StreamThread-1-consumer, groupId=sample-app] Resetting offset for partition sample-topic-0 to offset 474036997.
[sample-app-deploy-8c4fd5697-cjccm sample-app] 09:39:10.659 [sample-app-a7e7c388-0dd4-45e8-8d5a-3a84effb7dfd-StreamThread-1] INFO org.apache.kafka.clients.consumer.internals.Fetcher - [Consumer clientId=sample-app-a7e7c388-0dd4-45e8-8d5a-3a84effb7dfd-StreamThread-1-consumer, groupId=sample-app] Fetch offset 236702790 is out of range for partition sample-topic-5, resetting offset
任何人都可以帮助指出发生问题的地方吗？
问题出在时间戳上。我们以秒而不是毫秒为单位向分区发送事件时间，这导致时间保留非常激进。