如何在 GCP 上的 Kafka 服务器 运行 中处理 GCP 外部的消息
How to process messages outside GCP in a Kafka server running on GCP
我一直在尝试 运行 我本地机器上的消费者连接到 GCP 中的 Kafka 服务器 运行。
Kafka 和 Zookeeper 运行在同一 GCP VM 实例上
第 1 步：启动 Zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
第 2 步：启动 Kafka
bin/kafka-server-start.sh config/server.properties
如果我 运行 GCP VM 实例中的消费者它工作正常：
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
我验证了防火墙规则，我可以从我的本地机器访问，我可以访问 public IP 和 Kafka 服务器 运行 正在使用的端口。
我测试了很多选项，改变了kafka的server.properties，例如：
advertised.host.name=public-ip
或
advertised.listeners=public-ip
按照 connecting-kafka-running-on-ec2-machine-from-my-local-machine 上的答案没有成功。
来自官方documentation:
advertised.listeners
Listeners to publish to ZooKeeper for clients to use. In IaaS environments, this may
need to be different from the interface to which the broker binds. If
this is not set, the value for listeners will be used. Unlike
listeners it is not valid to advertise the 0.0.0.0 meta-address.
在测试了许多不同的选项后，这个解决方案对我有用：
设置两个侦听器，一个 EXTERNAL 使用 public IP，另一个 INTERNAL 使用私有 IP:
# Configure protocol map
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
# Use plaintext for inter-broker communication
inter.broker.listener.name=INTERNAL
# Specify that Kafka listeners should bind to all local interfaces
listeners=INTERNAL://0.0.0.0:9027,EXTERNAL://0.0.0.0:9037
# Separately, specify externally visible address
advertised.listeners=INTERNAL://localhost:9027,EXTERNAL://kafkabroker-n.mydomain.com:9093
解释：
In many scenarios, such as when deploying on AWS, the externally
advertised addresses of the Kafka brokers in the cluster differ from
the internal network interfaces that Kafka uses.
还请记住设置防火墙规则以在其他 EXTERNAL 侦听器上公开端口，以便从外部机器连接到它。
Note: It's important to restrict access to authorized clients only.
You can use network firewall rules to restrict access. This guidance
applies to scenarios that involve both RFC 1918 and public IP;
however, when using public IP addresses, it's even more important to
secure your Kafka endpoint because anyone can access it.
摘自 google solutions.
我一直在尝试 运行 我本地机器上的消费者连接到 GCP 中的 Kafka 服务器 运行。
Kafka 和 Zookeeper 运行在同一 GCP VM 实例上
第 1 步：启动 Zookeeper
bin/zookeeper-server-start.sh config/zookeeper.properties
第 2 步：启动 Kafka
bin/kafka-server-start.sh config/server.properties
如果我 运行 GCP VM 实例中的消费者它工作正常：
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
我验证了防火墙规则，我可以从我的本地机器访问，我可以访问 public IP 和 Kafka 服务器 运行 正在使用的端口。
我测试了很多选项，改变了kafka的server.properties，例如：
advertised.host.name=public-ip
或
advertised.listeners=public-ip
按照 connecting-kafka-running-on-ec2-machine-from-my-local-machine 上的答案没有成功。
来自官方documentation:
advertised.listeners
Listeners to publish to ZooKeeper for clients to use. In IaaS environments, this may need to be different from the interface to which the broker binds. If this is not set, the value for listeners will be used. Unlike listeners it is not valid to advertise the 0.0.0.0 meta-address.
在测试了许多不同的选项后，这个解决方案对我有用：
设置两个侦听器，一个 EXTERNAL 使用 public IP，另一个 INTERNAL 使用私有 IP:
# Configure protocol map
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
# Use plaintext for inter-broker communication
inter.broker.listener.name=INTERNAL
# Specify that Kafka listeners should bind to all local interfaces
listeners=INTERNAL://0.0.0.0:9027,EXTERNAL://0.0.0.0:9037
# Separately, specify externally visible address
advertised.listeners=INTERNAL://localhost:9027,EXTERNAL://kafkabroker-n.mydomain.com:9093
解释：
In many scenarios, such as when deploying on AWS, the externally advertised addresses of the Kafka brokers in the cluster differ from the internal network interfaces that Kafka uses.
还请记住设置防火墙规则以在其他 EXTERNAL 侦听器上公开端口，以便从外部机器连接到它。
Note: It's important to restrict access to authorized clients only. You can use network firewall rules to restrict access. This guidance applies to scenarios that involve both RFC 1918 and public IP; however, when using public IP addresses, it's even more important to secure your Kafka endpoint because anyone can access it.
摘自 google solutions.