apache 动物园管理员 kafka 路径

Question

我有一个设置到 kafka 主题的风暴连接的应用程序。我从我们组织中另一个类似的回购中模仿的设置有一个属性如下所示：

zookeeper.connect=127.0.0.1:2181/kafka_0.9

我发现使用此设置，在我的本地开发环境中，我的应用程序在尝试创建风暴喷口时会抛出错误。该错误可追溯到对 getNumPartitions 的调用。详细错误日志如下：

org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /brokers/topics/Moment_2018_01_16_07_59_08/partitions
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1590) ~[zookeeper-3.4.6.jar:3.4.6-1569965]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.call(GetChildrenBuilderImpl.java:214) ~[curator-framework-2.5.0.jar:na]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.call(GetChildrenBuilderImpl.java:203) ~[curator-framework-2.5.0.jar:na]
at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:107) ~[curator-client-2.5.0.jar:na]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.pathInForeground(GetChildrenBuilderImpl.java:199) ~[curator-framework-2.5.0.jar:na]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:191) ~[curator-framework-2.5.0.jar:na]
at org.apache.curator.framework.imps.GetChildrenBuilderImpl.forPath(GetChildrenBuilderImpl.java:38) ~[curator-framework-2.5.0.jar:na]
at storm.kafka.DynamicBrokersReader.getNumPartitions(DynamicBrokersReader.java:91) ~[storm-kafka-0.9.6.jar:na]

当我像这样从属性中取出尾随 /kafka_0.9 时：

zookeeper.connect=127.0.0.1:2181

然后这个错误就消失了。我的猜测是我们组织的代码在我们的暂存和生产环境中有一个特定的路径 /kafka_0.9 设置（可能指定使用 kafka 0.9）。但我想知道如何在本地开发环境中创建这条路径，以及如何与我们的暂存和生产环境保持一致？目前我的 docker-compose.yml 有以下 zookeeper 和 kafka:

zookeeper:
  image: myorg/zookeeper:3.4.8
  ports:
    - "2181:2181"

kafka:
  image: myorg/kafka:kafka-0.10
  hostname: myapp.docker
  ports:
    - "9092:9092"
  environment:
    EXPOSED_HOST: myapp.docker
    KAFKA_PORT: 9092
    KAFKA_ADVERTISED_PORT: 9092
    ZOOKEEPER_PORT_2181_TCP_ADDR: zookeeper
    ZOOKEEPER_PORT_2181_TCP_PORT: 2181
    ZOOKEEPER_IP: zookeeper
  links:
    - zookeeper
  extra_hosts:
    - "localhost:0.0.0.0"

Answer 1

尝试添加环境变量：CHROOT=/kafka_0.9

Zookeeper also allows you to add a "chroot" path which will make all kafka data for this cluster appear under a particular path. This is a way to setup multiple Kafka clusters or other applications on the same zookeeper cluster. To do this give a connection string in the form hostname1:port1,hostname2:port2,hostname3:port3/chroot/path which would put all this cluster's data under the path /chroot/path. Note that you must create this path yourself prior to starting the broker and consumers must use the same connection string.

apache 动物园管理员 kafka 路径

apache zookeeper kafka path

java

apache-kafka

apache-storm

apache-zookeeper