跨消费者的 Kafka 日志分发

Kafka log distribution across consumers

apache kafka documentation 提到了以下内容:

If all the consumer instances have the same consumer group, then the records will effectively be load balanced over the consumer instances.

If all the consumer instances have different consumer groups, then each record will be broadcast to all the consumer processes.

这让我在考虑分区时有点不清楚,第二个陈述是否意味着如果我有多个消费者组,是否意味着每个组中的每个消费者都将读取所有分区中的所有记录?! !

据我的拙见,他们在文档中使用的照片仍然与上述不符。

事实上,我正在阅读一篇很棒的文章,kafka in a nutshell 并且下面引用的陈述与文档中提供的照片更加吻合。

Consumers can also be organized into consumer groups for a given topic — each consumer within the group reads from a unique partition and the group as a whole consumes all messages from the entire topic. If you have more consumers than partitions then some consumers will be idle because they have no partitions to read from. If you have more partitions than consumers then consumers will receive messages from multiple partitions. If you have equal numbers of consumers and partitions, each consumer reads messages in order from exactly one partition.

我希望有人能阐明以上内容,并根据 Apache 的官方文档清楚地解释一个场景。

does that mean that each consumer in each group will read all the records in all partitions ?!!

没有。该语句假定每个组只有一个消费者(如 "If all the consumer instances have different consumer groups" 所示)。

所以你的整体理解是正确的。如果你有多个消费者组,一条消息将被发送到每个组一次。