[GitHub] [pulsar] dennisylyung commented on issue #6034: [Pulsar IO][Issue 5633]Support avro schema for debezium connector

[GitBox <gi...@apache.org>]

dennisylyung commented on issue #6034: [Pulsar IO][Issue 5633]Support avro schema for debezium connector
URL: https://github.com/apache/pulsar/pull/6034#issuecomment-613278588
 
 
   > @dennisylyung Hi, could you provide some config info about the error logs, such as the pulsar source config and the reproduce steps, thanks.
   > 
   > This is my test steps to follow the demo http://pulsar.apache.org/docs/en/io-debezium-source/#example-of-mysql, but I didn't reproduce your problem.
   > 
   > ```
   > # start debezium
   > docker run -it --rm \
   > --name mysql \
   > -p 3306:3306 \
   > -e MYSQL_ROOT_PASSWORD=debezium \
   > -e MYSQL_USER=mysqluser \
   > -e MYSQL_PASSWORD=mysqlpw debezium/example-mysql:0.8
   > 
   > # start pulsar
   > bin/pulsar standalone
   > 
   > # start source connector yaml
   > bin/pulsar-admin source localrun \
   > --source-config-file debezium-mysql-source-config.yaml
   > 
   > # subscribe the topic
   > bin/pulsar-client consume -s "sub-products" public/default/dbserver1.inventory.products -n 0
   > 
   > # start mysql
   > docker run -it --rm \
   > --name mysqlterm \
   > --link mysql \
   > --rm mysql:5.7 sh \
   > -c 'exec mysql -h"$MYSQL_PORT_3306_TCP_ADDR" -P"$MYSQL_PORT_3306_TCP_PORT" -uroot -p"$MYSQL_ENV_MYSQL_ROOT_PASSWORD"'
   > ```
   
   This is the config I was using
   ```
   tenant: "my-tenant"
   namespace: "debezium"
   name: "debezium-mysql-source"
   topicName: "my-tenant/debezium/debezium-mysql-topic"
   archive: "connectors/pulsar-io-debezium-mysql-2.6.0-SNAPSHOT.nar"
   
   parallelism: 1
   
   resources:
     cpu: 0.2
     ram: 1073741824
   
   configs:
     ## sourceTask
     task.class: "io.debezium.connector.mysql.MySqlConnectorTask"
   
     ## config for mysql, docker image: debezium/example-mysql:0.8
     database.hostname: "db.hostname"
     database.port: "3306"
     database.user: "kafka_connect"
     database.password: "password"
     database.server.id: "89447"
     database.server.name: "myname"
     database.whitelist: "myschema"
     table.whitelist: "myschema.mytable"
   
     database.history: "org.apache.pulsar.io.debezium.PulsarDatabaseHistory"
     database.history.pulsar.topic: "my-tenant/debezium/history-topic"
     database.history.pulsar.service.url: "pulsar://pulsar-broker.dev-pulsar.svc.cluster.local:6650"
     ## KEY_CONVERTER_CLASS_CONFIG, VALUE_CONVERTER_CLASS_CONFIG
     key.converter: "org.apache.pulsar.kafka.shade.io.confluent.connect.avro.AvroConverter"
     value.converter: "org.apache.pulsar.kafka.shade.io.confluent.connect.avro.AvroConverter"
     schema.registry.url: "http://schema-registry.kafka.internal.data.mydomain.com:8081"
     ## PULSAR_SERVICE_URL_CONFIG
     pulsar.service.url: "pulsar://pulsar-broker.dev-pulsar.svc.cluster.local:6650"
     ## OFFSET_STORAGE_TOPIC_CONFIG
     offset.storage.topic: "my-tenant/debezium/offset-topic"
     ## Extra settings
     # database.history.store.only.monitored.tables.ddl: "true"
     database.serverTimezone: "Asia/Hong_Kong"
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services