Kafka Stream 中的 exactly once 处理配置是否保证只读取已提交的读取?
Does exactly once processing configuration in a Kafka Stream guarantee only read committed reads?
有一系列关于 Kafka 事务和 exactly once 传递的优秀文章。
In one of them作者对消费者的评价:
So on the Consumer side, you have two options for reading transactional messages, expressed through the “isolation.level” consumer config:
read_committed: In addition to reading messages that are not part of a transaction, also be able to read ones that are, after the
transaction is committed.
read_uncommitted: Read all messages in offset order without waiting for transactions to be committed. This option is similar to
the current semantics of a Kafka consumer.
也就是说,普通的消费者应用程序必须指定read_committed
如果只想从主题中读取提交写入。
然而关于卡夫卡流:
All you need to make your Streams application employ exactly once
semantics, is to set this config “processing.guarantee=exactly_once”.
This causes all of the processing to happen exactly once; this
includes making both the processing and also all of the materialized
state created by the processing job that is written back to Kafka,
exactly once.
关于 KStream 中的读取没有明确说明。配置exactly_once
后,KStream是否只读取提交的消息?
是的,KStream 只会读取已提交的消息,文档中没有明确说明,但在 StreamsConfig JavaDoc 中您会找到信息:
If "processing.guarantee" is set to "exactly_once", Kafka Streams does not allow users to overwrite the following properties (Streams setting shown in parentheses):
"isolation.level" (read_committed) - Consumers will always read committed data only
"enable.idempotence" (true) - Producer will always have idempotency enabled
"max.in.flight.requests.per.connection" (5) - Producer will always have one in-flight request per connection
有一系列关于 Kafka 事务和 exactly once 传递的优秀文章。
In one of them作者对消费者的评价:
So on the Consumer side, you have two options for reading transactional messages, expressed through the “isolation.level” consumer config:
read_committed: In addition to reading messages that are not part of a transaction, also be able to read ones that are, after the transaction is committed.
read_uncommitted: Read all messages in offset order without waiting for transactions to be committed. This option is similar to the current semantics of a Kafka consumer.
也就是说,普通的消费者应用程序必须指定read_committed
如果只想从主题中读取提交写入。
然而关于卡夫卡流:
All you need to make your Streams application employ exactly once semantics, is to set this config “processing.guarantee=exactly_once”. This causes all of the processing to happen exactly once; this includes making both the processing and also all of the materialized state created by the processing job that is written back to Kafka, exactly once.
关于 KStream 中的读取没有明确说明。配置exactly_once
后,KStream是否只读取提交的消息?
是的,KStream 只会读取已提交的消息,文档中没有明确说明,但在 StreamsConfig JavaDoc 中您会找到信息:
If "processing.guarantee" is set to "exactly_once", Kafka Streams does not allow users to overwrite the following properties (Streams setting shown in parentheses): "isolation.level" (read_committed) - Consumers will always read committed data only "enable.idempotence" (true) - Producer will always have idempotency enabled "max.in.flight.requests.per.connection" (5) - Producer will always have one in-flight request per connection