未提供查询时,Kafka mysql 连接器配置文件不起作用
Kafka msql connector configuration file doesnot work when query is not provided
我有一个 Kafka 配置文件,它从 MYSQL 数据库中读取数据非常好
name=local-jbdc
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
connection.url=jdbc:mysql://localhost:3306/book
connection.user=root
connection.password=newpass
topic.prefix=quickstart-events
mode=incrementing
incrementing.column.name=__id
query=select * from book_table
offset.flush.timeout.ms=5000
buffer.memory=200
poll.interval.ms=10000
tasks.max=1
现在,当我取出 query
并提供 table.whitelist
时,它不会读取任何内容。甚至没有错误。
配置如下图
name=local-jbdc
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
connection.url=jdbc:mysql://localhost:3306/book
connection.user=root
connection.password=newpass
topic.prefix=quickstart-events
mode=incrementing
incrementing.column.name=__id
table.whitelist=book_table
offset.flush.timeout.ms=5000
buffer.memory=200
poll.interval.ms=10000
tasks.max=1
谁能帮我理解这个问题的根本原因。另外,我将如何为多个表执行增量模式。
编辑
当我在键盘上使用 Ctrl+C 停止 kafka 时
有这样的日志出现
[2020-11-30 12:35:38,057] INFO [ReplicaManager broker=0] Shut down completely (kafka.server.ReplicaManager)
[2020-11-30 12:35:38,058] INFO Shutting down. (kafka.log.LogManager)
[2020-11-30 12:35:38,106] INFO [ProducerStateManager partition=connect-status-4] Writing producer snapshot at offset 394 (kafka.log.ProducerStateManager)
[2020-11-30 12:35:38,158] INFO [ProducerStateManager partition=__consumer_offsets-18] Writing producer snapshot at offset 1 (kafka.log.ProducerStateManager)
[2020-11-30 12:35:38,219] INFO [ProducerStateManager partition=quickstart-eventsbook_table-0] Writing producer snapshot at offset 19645 (kafka.log.ProducerStateManager)
[2020-11-30 12:35:38,239] INFO [ProducerStateManager partition=quickstart-book_table-0] Writing producer snapshot at offset 2652 (kafka.log.ProducerStateManager)```
问题很简单。当提供 table.whitelist 时,连接器会创建一个主题,并在 table 名称后附加 topic.prefix
。在我的例子中,它创建了一个名为 quickstart-eventsbook_table
的新主题。提供查询时 topic.prefix
也被视为发送数据的主题。
我有一个 Kafka 配置文件,它从 MYSQL 数据库中读取数据非常好
name=local-jbdc
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
connection.url=jdbc:mysql://localhost:3306/book
connection.user=root
connection.password=newpass
topic.prefix=quickstart-events
mode=incrementing
incrementing.column.name=__id
query=select * from book_table
offset.flush.timeout.ms=5000
buffer.memory=200
poll.interval.ms=10000
tasks.max=1
现在,当我取出 query
并提供 table.whitelist
时,它不会读取任何内容。甚至没有错误。
配置如下图
name=local-jbdc
connector.class=io.confluent.connect.jdbc.JdbcSourceConnector
connection.url=jdbc:mysql://localhost:3306/book
connection.user=root
connection.password=newpass
topic.prefix=quickstart-events
mode=incrementing
incrementing.column.name=__id
table.whitelist=book_table
offset.flush.timeout.ms=5000
buffer.memory=200
poll.interval.ms=10000
tasks.max=1
谁能帮我理解这个问题的根本原因。另外,我将如何为多个表执行增量模式。
编辑
当我在键盘上使用 Ctrl+C 停止 kafka 时
有这样的日志出现
[2020-11-30 12:35:38,057] INFO [ReplicaManager broker=0] Shut down completely (kafka.server.ReplicaManager)
[2020-11-30 12:35:38,058] INFO Shutting down. (kafka.log.LogManager)
[2020-11-30 12:35:38,106] INFO [ProducerStateManager partition=connect-status-4] Writing producer snapshot at offset 394 (kafka.log.ProducerStateManager)
[2020-11-30 12:35:38,158] INFO [ProducerStateManager partition=__consumer_offsets-18] Writing producer snapshot at offset 1 (kafka.log.ProducerStateManager)
[2020-11-30 12:35:38,219] INFO [ProducerStateManager partition=quickstart-eventsbook_table-0] Writing producer snapshot at offset 19645 (kafka.log.ProducerStateManager)
[2020-11-30 12:35:38,239] INFO [ProducerStateManager partition=quickstart-book_table-0] Writing producer snapshot at offset 2652 (kafka.log.ProducerStateManager)```
问题很简单。当提供 table.whitelist 时,连接器会创建一个主题,并在 table 名称后附加 topic.prefix
。在我的例子中,它创建了一个名为 quickstart-eventsbook_table
的新主题。提供查询时 topic.prefix
也被视为发送数据的主题。