org.apache.kafka.connect.errors.ConnectException 使用 Snowflake 连接器启动 Kafka Connect 时出错

Question

我正在按照本教程尝试为 Kafka 启动 Snowflake 连接器实例：https://docs.snowflake.com/en/user-guide/kafka-connector-install.html

JDK version: 13
Kafka Version: 2.5.1
Snowflake Kafka Connector Version: 1.5.5 (installed in `/home/ubuntu/kafka/libs`

安装 Kafka、JDK 和 Snowflake Kafka Connect jar 文件后，我设置了以下属性：

连接-standalone.properties

bootstrap.servers=[my_brokers_urls]
key.converter=org.apache.kafka.connect.json.JsonConverter
value.converter=org.apache.kafka.connect.json.JsonConverter
key.converter.schemas.enable=true
value.converter.schemas.enable=true
offset.storage.file.filename=/tmp/connect.offsets
offset.flush.interval.ms=10000
plugin.path=/home/ubuntu/kafka/libs

SF_connect.properties

name=snowflakeconnector1
connector.class=com.snowflake.kafka.connector.SnowflakeSinkConnector
topics=topic1,topic2
snowflake.topic2table.map= topic1:existing_table1,topic2:existing_table2
buffer.count.records=10000
buffer.flush.time=60
buffer.size.bytes=5000000
snowflake.url.name=<snowflake_url>
snowflake.user.name=kafka_connector_user_1
snowflake.private.key=<private_key>
snowflake.database.name=kafka_db
snowflake.schema.name=kafka_demo_schema
key.converter=org.apache.kafka.connect.storage.StringConverter
value.converter=com.snowflake.kafka.connector.records.SnowflakeJsonConverter
tasks.max=6

在调试时，我还尝试安装了一些额外的 JAR 文件。这是我下载的 jar 的完整列表：

curl "https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.13/2.5.1/kafka_2.13-2.5.1.jar" -o ~/kafka/libs/kafka_2.13-2.5.1.jar
curl "https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/2.5.1/kafka-clients-2.5.1.jar" -o ~/kafka/libs/kafka-clients-2.5.1.jar
curl "https://repo1.maven.org/maven2/org/apache/kafka/connect-transforms/2.5.1/connect-transforms-2.5.1.jar" -o ~/kafka/libs/connect-transforms-2.5.1.jar
curl "https://repo1.maven.org/maven2/com/snowflake/snowflake-kafka-connector/1.5.5/snowflake-kafka-connector-1.5.5.jar" -o ~/kafka/libs/snowflake-kafka-connector-1.5.5.jar
curl "https://repo1.maven.org/maven2/org/apache/kafka/connect-runtime/2.3.0/connect-runtime-2.3.0.jar" -o ~/kafka/libs/connect-runtime-2.3.0.jar
curl "https://repo1.maven.org/maven2/org/apache/kafka/connect-api/2.3.0/connect-api-2.3.0.jar" -o ~/kafka/libs/connect-api-2.3.0.jar
curl "https://repo1.maven.org/maven2/org/apache/avro/avro/1.10.2/avro-1.10.2.jar" -o ~/kafka/libs/avro-1.10.2.jar
curl "https://repo1.maven.org/maven2/org/apache/kafka/connect-json/2.3.0/connect-json-2.3.0.jar" -o ~/kafka/libs/connect-json-2.3.0.jar
curl "https://repo1.maven.org/maven2/com/datamountaineer/kafka-connect-common/2.0.5/kafka-connect-common-2.0.5.jar" -o ~/kafka/libs/kafka-connect-common-2.0.5.jar
curl "https://repo1.maven.org/maven2/org/reflections/reflections/0.9.10/reflections-0.9.10.jar" -o ~/kafka/libs/reflections-0.9.10.jar
curl "https://repo1.maven.org/maven2/org/javassist/javassist/3.27.0-GA/javassist-3.27.0-GA.jar" -o ~/kafka/libs/javassist-3.27.0-GA.jar
curl "https://repo1.maven.org/maven2/org/apache/maven/maven-artifact/3.8.2/maven-artifact-3.8.2.jar" -o ~/kafka/libs/maven-artifact-3.8.2.jar
curl "https://repo1.maven.org/maven2/org/apache/commons/commons-lang3/3.9/commons-lang3-3.9.jar" -o ~/kafka/libs/commons-lang3-3.9.jar
curl "https://repo1.maven.org/maven2/org/slf4j/slf4j-simple/1.6.1/slf4j-simple-1.6.1.jar" -o ~/kafka/libs/slf4j-simple-1.6.1.jar

现在，一切都应该准备好了。但是，当我 运行 命令时：

~/kafka/bin/connect-standalone.sh ~/kafka/config/connect-standalone.properties ~/kafka/config/SF_connect.properties

我有以下错误：

5124 [main] ERROR org.apache.kafka.connect.cli.ConnectStandalone - Stopping due to error
java.lang.NoClassDefFoundError: org/apache/kafka/common/record/InvalidRecordException
        at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:83)
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.common.record.InvalidRecordException
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:602)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)

我该如何解决这个问题？

Answer 1

我终于可以解决所有依赖性问题。这是我所有版本的摘要：

Kafka: 2.5.1
Java: 13

以下是我在plugin.path目录下下载的所有JAR：

curl "https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/2.3.1/kafka-clients-2.3.1.jar" -o ~/kafka/libs/kafka-clients-2.3.1.jar
curl "https://repo1.maven.org/maven2/org/apache/kafka/connect-runtime/2.3.0/connect-runtime-2.3.0.jar" -o ~/kafka/libs/connect-runtime-2.3.0.jar
curl "https://repo1.maven.org/maven2/org/slf4j/slf4j-simple/1.6.1/slf4j-simple-1.6.1.jar" -o ~/kafka/libs/slf4j-simple-1.6.1.jar
curl "https://repo1.maven.org/maven2/org/apache/kafka/kafka_2.13/2.5.1/kafka_2.13-2.5.1.jar" -o ~/kafka/libs/kafka_2.13-2.5.1.jar
curl "https://repo1.maven.org/maven2/org/apache/kafka/connect-transforms/2.5.1/connect-transforms-2.5.1.jar" -o ~/kafka/libs/connect-transforms-2.5.1.jar
curl "https://repo1.maven.org/maven2/com/snowflake/snowflake-kafka-connector/1.5.5/snowflake-kafka-connector-1.5.5.jar" -o ~/kafka/libs/snowflake-kafka-connector-1.5.5.jar
curl "https://repo1.maven.org/maven2/org/apache/kafka/connect-api/2.3.0/connect-api-2.3.0.jar" -o ~/kafka/libs/connect-api-2.3.0.jar
curl "https://repo1.maven.org/maven2/org/apache/avro/avro/1.10.2/avro-1.10.2.jar" -o ~/kafka/libs/avro-1.10.2.jar
curl "https://repo1.maven.org/maven2/org/apache/kafka/connect-json/2.3.0/connect-json-2.3.0.jar" -o ~/kafka/libs/connect-json-2.3.0.jar
curl "https://repo1.maven.org/maven2/org/reflections/reflections/0.9.10/reflections-0.9.10.jar" -o ~/kafka/libs/reflections-0.9.10.jar
curl "https://repo1.maven.org/maven2/org/javassist/javassist/3.27.0-GA/javassist-3.27.0-GA.jar" -o ~/kafka/libs/javassist-3.27.0-GA.jar
curl "https://repo1.maven.org/maven2/org/apache/maven/maven-artifact/3.8.2/maven-artifact-3.8.2.jar" -o ~/kafka/libs/maven-artifact-3.8.2.jar
curl "https://repo1.maven.org/maven2/org/apache/commons/commons-lang3/3.9/commons-lang3-3.9.jar" -o ~/kafka/libs/commons-lang3-3.9.jar