You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by KhajaAsmath Mohammed <md...@gmail.com> on 2019/12/26 02:30:55 UTC
Kafka connect issue with mysql - Debezium CDC
Hi,
I am trying to do POC for kafka CDC with database and ingest it into kafka.
I have been trying to make it work from past 5 days and had no luck. can
someone help me what is wrong in my approach.
Error Received:
*{"error_code":400,"message":"Connector configuration is invalid and
contains the following 1 error(s):\nUnable to connect: Communications link
failure\n\nThe last packet sent successfully to the server was 0
milliseconds ago. The driver has not received any packets from the
server.\nYou can also find the above list of errors at the endpoint
`/{connectorType}/config/validate`"*
I am able to connect mysql in docker container and also from my laptop
using sqlconnector with host as localhost and port 3306,I even tried with
IP address of database and no luck.
ALso tried various options of changing binding address for mysql and my
assumption is that connection is being made but somehow it is not
processing.
I can also see the request is sent to kafka connector where I can see the
same error in distributed.log
*Here are my code snippets.*
Jar used is debezium-connector-mysql-1.0.0.Final
Script file to call the rest api request.
#!/bin/bash
curl -X POST http://localhost:8083/connectors -H 'Content-Type:
application/json' -H 'Accept: application/json' -d
@./lms_dev_source_jdbc_config.json
JSON file to send request to connector.
{
"name": "mysql-connector-flattened",
"config": {
"connector.class": "io.debezium.connector.mysql.MySqlConnector",
"database.hostname":"localhost",
"database.port": "3306",
"database.user": "root",
"database.password": "Bismillah1!",
"database.server.id": "4",
"database.server.name": "recipes_database",
"database.history.kafka.bootstrap.servers": "localhost:9092",
"database.history.kafka.topic": "dbhistory.fullfillment" ,
"include.schema.changes": "true" ,
"transforms": "unwrap,changetopic",
"transforms.unwrap.type": "io.debezium.transforms.UnwrapFromEnvelope",
"transforms.changetopic.type":"org.apache.kafka.connect.transforms.RegexRouter",
"transforms.changetopic.regex":"(.*)",
"transforms.changetopic.replacement":"$1-smt"
}
}
DockerCompse File:
version: '2'
services:
db:
image: mysql:8.0
command: --default-authentication-plugin=mysql_native_password
restart: always
environment:
MYSQL_DATABASE: db
MYSQL_USER: debezium
MYSQL_PASSWORD: Bismillah1!
MYSQL_ROOT_PASSWORD: Bismillah1!
ports:
- 3306:3306
expose:
- 3306
kafka-cluster:
image: landoop/fast-data-dev:cp3.3.0
restart: always
environment:
ADV_HOST: 127.0.0.1 # Change to 192.168.99.100 if using
Docker Toolbox
RUNTESTS: 0 # Disable Running tests so the
cluster starts faster
ports:
- 2181:2181 # Zookeeper
- 3030:3030 # Landoop UI
- 8081-8083:8081-8083 # REST Proxy, Schema Registry, Kafka
Connect ports
- 9581-9585:9581-9585 # JMX Ports
- 9092:9092 # Kafka Broker
Re: Kafka connect issue with mysql - Debezium CDC
Posted by KhajaAsmath Mohammed <md...@gmail.com>.
Hi,
I was able to resolve this issue by changing the name of server to docker
container name.
[image: image.png]
Thanks,
Asmath
On Wed, Dec 25, 2019 at 8:30 PM KhajaAsmath Mohammed <
mdkhajaasmath@gmail.com> wrote:
> Hi,
>
> I am trying to do POC for kafka CDC with database and ingest it into
> kafka. I have been trying to make it work from past 5 days and had no luck.
> can someone help me what is wrong in my approach.
>
> Error Received:
>
> *{"error_code":400,"message":"Connector configuration is invalid and
> contains the following 1 error(s):\nUnable to connect: Communications link
> failure\n\nThe last packet sent successfully to the server was 0
> milliseconds ago. The driver has not received any packets from the
> server.\nYou can also find the above list of errors at the endpoint
> `/{connectorType}/config/validate`"*
>
> I am able to connect mysql in docker container and also from my laptop
> using sqlconnector with host as localhost and port 3306,I even tried with
> IP address of database and no luck.
> ALso tried various options of changing binding address for mysql and my
> assumption is that connection is being made but somehow it is not
> processing.
>
> I can also see the request is sent to kafka connector where I can see the
> same error in distributed.log
>
> *Here are my code snippets.*
>
> Jar used is debezium-connector-mysql-1.0.0.Final
>
> Script file to call the rest api request.
>
> #!/bin/bash
> curl -X POST http://localhost:8083/connectors -H 'Content-Type:
> application/json' -H 'Accept: application/json' -d
> @./lms_dev_source_jdbc_config.json
>
>
> JSON file to send request to connector.
>
> {
> "name": "mysql-connector-flattened",
> "config": {
> "connector.class": "io.debezium.connector.mysql.MySqlConnector",
> "database.hostname":"localhost",
> "database.port": "3306",
> "database.user": "root",
> "database.password": "Bismillah1!",
> "database.server.id": "4",
> "database.server.name": "recipes_database",
> "database.history.kafka.bootstrap.servers": "localhost:9092",
> "database.history.kafka.topic": "dbhistory.fullfillment" ,
> "include.schema.changes": "true" ,
> "transforms": "unwrap,changetopic",
> "transforms.unwrap.type": "io.debezium.transforms.UnwrapFromEnvelope",
>
> "transforms.changetopic.type":"org.apache.kafka.connect.transforms.RegexRouter",
> "transforms.changetopic.regex":"(.*)",
> "transforms.changetopic.replacement":"$1-smt"
> }
> }
>
>
> DockerCompse File:
>
> version: '2'
>
> services:
> db:
> image: mysql:8.0
> command: --default-authentication-plugin=mysql_native_password
> restart: always
> environment:
> MYSQL_DATABASE: db
> MYSQL_USER: debezium
> MYSQL_PASSWORD: Bismillah1!
> MYSQL_ROOT_PASSWORD: Bismillah1!
> ports:
> - 3306:3306
> expose:
> - 3306
>
> kafka-cluster:
>
> image: landoop/fast-data-dev:cp3.3.0
> restart: always
> environment:
> ADV_HOST: 127.0.0.1 # Change to 192.168.99.100 if using Docker Toolbox
> RUNTESTS: 0 # Disable Running tests so the cluster starts faster
> ports:
> - 2181:2181 # Zookeeper
> - 3030:3030 # Landoop UI
> - 8081-8083:8081-8083 # REST Proxy, Schema Registry, Kafka Connect ports
> - 9581-9585:9581-9585 # JMX Ports
> - 9092:9092 # Kafka Broker
>
>
>
>
>
>
>
>
>
>
>
>
>