如何将数据发送到 Node JS 中的多个 Kafka 主题分区

Question

在 Node js 应用程序中，当我尝试将消息发送到 Kafka 主题时，所有消息都将转到分区 0。该主题是用 4 个分区创建的，并且想以循环机制发布，我尝试了多种选择但没有运气.

有什么办法可以解决吗？下面是代码片段。

payloads = [
    { topic: 'test-topic', messages: ['TestMessage1', 'TestMessage2', 'TestMessage3', 'TestMessage4']},
];
producer.on('ready', function(){
    producer.send(payloads, function(err, data){
        console.log("Successfully written onto Kafka");
    });

Answer 1

在Kafka中，相同key的消息放在同一个partition中。您可以手动定义分区：

// Force partitioning - default partition is 0
payloads = [ 
    { topic: 'test-topic', messages: ['TestMessage1'], partition: 0 },
    { topic: 'test-topic', messages: ['TestMessage2'], partition: 1 },
    { topic: 'test-topic', messages: ['TestMessage3'], partition: 2 },
    { topic: 'test-topic', messages: ['TestMessage4'], partition: 3 }
];

或对每条消息使用不同的密钥：

payloads = [ 
    { topic: 'test-topic', messages: ['TestMessage1'], key: '1' },
    { topic: 'test-topic', messages: ['TestMessage2'], key: '2' },
    { topic: 'test-topic', messages: ['TestMessage3'], key: '3' },
    { topic: 'test-topic', messages: ['TestMessage4'], key: '4' }
];

// Alternatively, you can use KeyedMessage
km = new KeyedMessage('1', 'TestMessage1'),
km2 = new KeyedMessage('2', 'TestMessage2'),
payloads = [
    { topic: 'test-topic', messages: [ km , km2 ] },
];

如何将数据发送到 Node JS 中的多个 Kafka 主题分区

How to send data to multiple Kafka topic partitions in Node JS

node.js

apache-kafka

kafka-producer-api