Kafka Connect - 无法查找分区元数据

Question

运行 Kafka 集群上的 Kafka 连接已关闭自动创建主题。据我所知，在另一个正在运行的集群上，我们创建了主题连接配置、连接偏移量和连接状态。因此，这些主题是在具有相同复制因子和分区的新集群上重新创建的。

尽管我有从连接实例到代理的 netcat 并且那里没有连接问题，但我们在启动服务时遇到错误

Could not look up partition metadata for offset backing store topic in allotted period. This could indicate a connectivity issue, unavailable topic partitions, or if this is your first use of the topic it may have taken too long to create

我可以从连接工具 class 中看到它被抛出的位置。但是已经检查过，这些主题的创建现在有点迷失

        producer = createProducer();
        consumer = createConsumer();

        List<TopicPartition> partitions = new ArrayList<>();

        // We expect that the topics will have been created either manually by the user or automatically by the herder
        List<PartitionInfo> partitionInfos = null;
        long started = time.milliseconds();
        while (partitionInfos == null && time.milliseconds() - started < CREATE_TOPIC_TIMEOUT_MS) {
            partitionInfos = consumer.partitionsFor(topic);
            Utils.sleep(Math.min(time.milliseconds() - started, 1000));
        }
        if (partitionInfos == null)
            throw new ConnectException("Could not look up partition metadata for offset backing store topic in" +
                    " allotted period. This could indicate a connectivity issue, unavailable topic partitions, or if" +
                    " this is your first use of the topic it may have taken too long to create

Answer 1

the topics were created manually,

Kafka Connect 将使用 AdminClient 来初始化自己的主题。没有理由手动完成。

如果这样做，您需要确保设置正确，这些设置在 Confluent 和 Apache Kafka 文档中都有记录，以便开始使用 Connect

Kafka Connect - 无法查找分区元数据

Kafka Connect - Could not look up partition metadata

java

apache-kafka

apache-kafka-connect