如何从主机向 Docker (bitnami) Apache Kafka 发送消息？

Question

我可以让我的 Apache Kafka 生产者在容器运行中发送消息。但是，当我的生产者在主机容器外运行时，它不起作用。我怀疑这是我的 Docker 撰写文件的 Docker 网络问题，但我无法弄清楚。

我尝试了与我的问题类似的在线解决方案 post，但它们对我不起作用。求助！

Docker-编写文件

version: '3'
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
    environment:_
      - KAFKA_BROKER_ID=1
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://localhost:9092
      - ALLOW_PLAINTEXT_LISTENER=yes

主机制作人


//import util.properties packages
import java.util.Properties;

//import simple producer packages
import org.apache.kafka.clients.producer.Producer;

//import KafkaProducer packages
import org.apache.kafka.clients.producer.KafkaProducer;

//import ProducerRecord packages
import org.apache.kafka.clients.producer.ProducerRecord;

//Create java class named “SimpleProducer”
public class SimpleProducer {

   public static void main(String[] args) throws Exception{

      // Check arguments length value
      if(args.length == 0){
         System.out.println("Enter topic name");
         return;
      }

      //Assign topicName to string variable
      String topicName = args[0].toString();

      // create instance for properties to access producer configs   
      Properties props = new Properties();

      //Assign localhost id
      props.put("bootstrap.servers", "localhost:9092");

      //Set acknowledgements for producer requests.      
      props.put("acks", "all");

      //If the request fails, the producer can automatically retry,
      props.put("retries", 0);

      //Specify buffer size in config
      props.put("batch.size", 16384);

      //Reduce the no of requests less than 0   
      props.put("linger.ms", 1);

      //The buffer.memory controls the total amount of memory available to the producer for buffering.   
      props.put("buffer.memory", 33554432);

      props.put("key.serializer", 
         "org.apache.kafka.common.serialization.StringSerializer");

      props.put("value.serializer", 
         "org.apache.kafka.common.serialization.StringSerializer");

      Producer<String, String> producer = new KafkaProducer
         <String, String>(props);

      for(int i = 0; i < 10; i++)
         producer.send(new ProducerRecord<String, String>(topicName, 
            Integer.toString(i), Integer.toString(i)));
               System.out.println("Message sent successfully");
               producer.close();
   }
}

主机生产者应该 post 向 Docker Apache kafka 发送消息，但事实并非如此。它创建了主题，但从未收到消息。我究竟做错了什么？这是bitnami图像，不是Confluent图像。

Answer 1

来自我的 :

What I needed to do was to declare the LISTENERS as both binding to the docker host, and then advertise them differently - one to the docker network, one to the host.

services:
  zookeeper:
    image: confluentinc/cp-zookeeper
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
      ZOOKEEPER_SYNC_LIMIT: 2
  kafka:
    image: confluentinc/cp-kafka
    ports:
      - 9094:9094
    depends_on:
      - zookeeper
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://kafka:9094
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka:9092,OUTSIDE://localhost:9094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,OUTSIDE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL

现在您的本地主机上的 Kafka 可用：9094（根据外部侦听器和 docker-compose 文件中的 ports 条目），以及 Docker 网络内在：9092.

Answer 2

此解决方案适用于 Apache Kafka 的 bitnami Docker 映像。感谢@cricket_007 和@daniu 的解决方案。我在 Kafka 环境部分的 Docker-compose 文件中更新了几行。

这是完整的、更新的 Docker-compose 文件：

version: '3'
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
      - '29092:29092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - KAFKA_LISTENER_SECURITY_PROTOCOL_MAP=PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT 
      - KAFKA_LISTENERS=PLAINTEXT://:9092,PLAINTEXT_HOST://:29092 
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,PLAINTEXT_HOST://localhost:29092
      - ALLOW_PLAINTEXT_LISTENER=yes

如何从主机向 Docker (bitnami) Apache Kafka 发送消息？

How do I send messages to Docker (bitnami) Apache Kafka from host machine?

apache-kafka

docker

kafka-producer-api