You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Avani Panchal <av...@adit.com.INVALID> on 2023/06/26 06:11:33 UTC
How to Integrate MySQL database with Kafka? Need Demo
Hi,
In my application I want to sync my client's data to my SQL server. at
client place the database is MYSQL.
How can I achieve this using Kafka? I read a lot of documents but I don't
understand which setup I need and how I can achieve it.
I was also wondering about "Book a demo with Kafka" but didn't find it.
Please help me.
Thank you,
Avani
Re: How to Integrate MySQL database with Kafka? Need Demo
Posted by Avani Panchal <av...@adit.com.INVALID>.
thanks Roman.
On Tue, Jun 27, 2023 at 3:15 PM Roman Schmitz <ro...@gmail.com>
wrote:
> Hi Avani,
>
> not sure what you mean by creating your own code. Of course you can write
> an own Connector but especially in Source Connectors offset handling can
> challenging depending on the system.
> There are hundreds of different Connectors out there that can connect to
> existing systems. If you find one that suits your needs that's probably a
> better idea than starting from scratch.
> Typically there are lots of configuration options and for quite simple
> transformations you can use SMTs. To run Connectors, simply install some
> Connect instance, have the jars for your connectors installed and
> configure/deploy your Connector(s) using the Connect API.
>
> Kind regards,
> Roman
>
> Am Di., 27. Juni 2023 um 11:26 Uhr schrieb Avani Panchal
> <av...@adit.com.invalid>:
>
> > Hi Roman,
> >
> > Thanks for replying, I want to ask one question: can I create my own code
> > to integrate data using Kafka connectors?
> >
> > Kind regards,
> > Avani Panchal
> >
> > On Tue, Jun 27, 2023 at 12:29 PM Roman Schmitz <ro...@gmail.com>
> > wrote:
> >
> > > Hi Avani,
> > >
> > > totally agree - depending on the way you'd like to integrate your
> source
> > > that would be probably Debezium for CDC or JDBC source that can be used
> > > either in a "full-load" as well as in an incremental way.
> > > As a sink JDBC sink with the corresponding driver for the DB. Kindly
> > have a
> > > look at the Connect Documentation and some examples.
> > > If you'd like to test with runnable Docker-based examples, you can
> have a
> > > look at this Docker Playground Repo:
> > >
> https://github.com/vdesabou/kafka-docker-playground/tree/master/connect
> > >
> > > Kind regards,
> > > Roman
> > >
> > > Am Di., 27. Juni 2023 um 08:48 Uhr schrieb Avani Panchal
> > > <av...@adit.com.invalid>:
> > >
> > > > Hi Sagar,
> > > >
> > > > Thank you for your email.
> > > >
> > > > Thanks & Regards,
> > > > Avani Panchal
> > > >
> > > >
> > > > On Tue, Jun 27, 2023 at 12:15 PM Sagar <sa...@gmail.com>
> > > wrote:
> > > >
> > > > > Hi Avani,
> > > > >
> > > > > I already shared the documentation link for Kafka Connect. Let me
> > share
> > > > it
> > > > > again: https://kafka.apache.org/documentation/#connect
> > > > >
> > > > > Regarding the connector documentation, you should be able to find
> > them
> > > by
> > > > > just searching for JDBC source connector, JDBC sink connector and
> > > > Debezium
> > > > > connector for MySQL.
> > > > >
> > > > > Let me know if that works.
> > > > >
> > > > > Thanks!
> > > > > Sagar.
> > > > >
> > > > >
> > > > >
> > > > > On Mon, Jun 26, 2023 at 2:10 PM Avani Panchal
> > > > > <av...@adit.com.invalid> wrote:
> > > > >
> > > > > > Hi Sagar,
> > > > > >
> > > > > > Thank you for the information, you solved our confusion.
> > > > > > I also saw lots of links for documentation on Kafka, but I am
> > > confused
> > > > > > which document I should use.
> > > > > > So can you share the proper link from where I can read the
> > documents.
> > > > > >
> > > > > > Thanks,
> > > > > > Avani Panchal
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 26, 2023 at 1:48 PM Sagar <sagarmeansocean@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Hey Avani,
> > > > > > >
> > > > > > > Kafka Connect <https://kafka.apache.org/documentation/#connect
> >
> > is
> > > > the
> > > > > > > tool
> > > > > > > to use when you want to stream data to/from Kafka via external
> > > > systems.
> > > > > > One
> > > > > > > would typically configure connectors which allow streaming data
> > > > to/from
> > > > > > > Kafka. There are 2 types of connectors:
> > > > > > > 1) Source Connectors: Which stream data from external systems
> > like
> > > > > > > databases etc to Kafka and
> > > > > > > 2) Sink Connectors: Which stream data from Kafka to external
> > > systems.
> > > > > > >
> > > > > > > Since you want to stream data from MySQL to SQL Server, with
> > Kafka
> > > > > > Connect
> > > > > > > it would be a 2 step process:
> > > > > > >
> > > > > > > 1) Capture changes from MySQL to Kafka using connectors like
> JDBC
> > > > > source
> > > > > > > connector or Debezium MySQL connector.
> > > > > > > 2) Once the data is in Kafka, you can use JDBC sink connectors
> to
> > > > > stream
> > > > > > > data from Kafka topics to the tables in SQL Server.
> > > > > > >
> > > > > > > Note that this is a very simplified view of how you can achieve
> > > your
> > > > > goal
> > > > > > > of streaming changes from MySQL to SQL Server and I would
> > recommend
> > > > > > reading
> > > > > > > the documentation of the individual connectors and the Kafka
> > > Connect
> > > > > > > framework to understand how to make it work for your usecase.
> > > > > > >
> > > > > > > Thanks for your interest on Apache Kafka!
> > > > > > >
> > > > > > > Thanks!
> > > > > > > Sagar.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Jun 26, 2023 at 11:42 AM Avani Panchal
> > > > > > > <av...@adit.com.invalid> wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > > In my application I want to sync my client's data to my SQL
> > > > server.
> > > > > at
> > > > > > > > client place the database is MYSQL.
> > > > > > > >
> > > > > > > > How can I achieve this using Kafka? I read a lot of documents
> > > but I
> > > > > > don't
> > > > > > > > understand which setup I need and how I can achieve it.
> > > > > > > >
> > > > > > > > I was also wondering about "Book a demo with Kafka" but
> didn't
> > > find
> > > > > it.
> > > > > > > >
> > > > > > > > Please help me.
> > > > > > > >
> > > > > > > > Thank you,
> > > > > > > > Avani
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: How to Integrate MySQL database with Kafka? Need Demo
Posted by Roman Schmitz <ro...@gmail.com>.
Hi Avani,
not sure what you mean by creating your own code. Of course you can write
an own Connector but especially in Source Connectors offset handling can
challenging depending on the system.
There are hundreds of different Connectors out there that can connect to
existing systems. If you find one that suits your needs that's probably a
better idea than starting from scratch.
Typically there are lots of configuration options and for quite simple
transformations you can use SMTs. To run Connectors, simply install some
Connect instance, have the jars for your connectors installed and
configure/deploy your Connector(s) using the Connect API.
Kind regards,
Roman
Am Di., 27. Juni 2023 um 11:26 Uhr schrieb Avani Panchal
<av...@adit.com.invalid>:
> Hi Roman,
>
> Thanks for replying, I want to ask one question: can I create my own code
> to integrate data using Kafka connectors?
>
> Kind regards,
> Avani Panchal
>
> On Tue, Jun 27, 2023 at 12:29 PM Roman Schmitz <ro...@gmail.com>
> wrote:
>
> > Hi Avani,
> >
> > totally agree - depending on the way you'd like to integrate your source
> > that would be probably Debezium for CDC or JDBC source that can be used
> > either in a "full-load" as well as in an incremental way.
> > As a sink JDBC sink with the corresponding driver for the DB. Kindly
> have a
> > look at the Connect Documentation and some examples.
> > If you'd like to test with runnable Docker-based examples, you can have a
> > look at this Docker Playground Repo:
> > https://github.com/vdesabou/kafka-docker-playground/tree/master/connect
> >
> > Kind regards,
> > Roman
> >
> > Am Di., 27. Juni 2023 um 08:48 Uhr schrieb Avani Panchal
> > <av...@adit.com.invalid>:
> >
> > > Hi Sagar,
> > >
> > > Thank you for your email.
> > >
> > > Thanks & Regards,
> > > Avani Panchal
> > >
> > >
> > > On Tue, Jun 27, 2023 at 12:15 PM Sagar <sa...@gmail.com>
> > wrote:
> > >
> > > > Hi Avani,
> > > >
> > > > I already shared the documentation link for Kafka Connect. Let me
> share
> > > it
> > > > again: https://kafka.apache.org/documentation/#connect
> > > >
> > > > Regarding the connector documentation, you should be able to find
> them
> > by
> > > > just searching for JDBC source connector, JDBC sink connector and
> > > Debezium
> > > > connector for MySQL.
> > > >
> > > > Let me know if that works.
> > > >
> > > > Thanks!
> > > > Sagar.
> > > >
> > > >
> > > >
> > > > On Mon, Jun 26, 2023 at 2:10 PM Avani Panchal
> > > > <av...@adit.com.invalid> wrote:
> > > >
> > > > > Hi Sagar,
> > > > >
> > > > > Thank you for the information, you solved our confusion.
> > > > > I also saw lots of links for documentation on Kafka, but I am
> > confused
> > > > > which document I should use.
> > > > > So can you share the proper link from where I can read the
> documents.
> > > > >
> > > > > Thanks,
> > > > > Avani Panchal
> > > > >
> > > > >
> > > > > On Mon, Jun 26, 2023 at 1:48 PM Sagar <sa...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Hey Avani,
> > > > > >
> > > > > > Kafka Connect <https://kafka.apache.org/documentation/#connect>
> is
> > > the
> > > > > > tool
> > > > > > to use when you want to stream data to/from Kafka via external
> > > systems.
> > > > > One
> > > > > > would typically configure connectors which allow streaming data
> > > to/from
> > > > > > Kafka. There are 2 types of connectors:
> > > > > > 1) Source Connectors: Which stream data from external systems
> like
> > > > > > databases etc to Kafka and
> > > > > > 2) Sink Connectors: Which stream data from Kafka to external
> > systems.
> > > > > >
> > > > > > Since you want to stream data from MySQL to SQL Server, with
> Kafka
> > > > > Connect
> > > > > > it would be a 2 step process:
> > > > > >
> > > > > > 1) Capture changes from MySQL to Kafka using connectors like JDBC
> > > > source
> > > > > > connector or Debezium MySQL connector.
> > > > > > 2) Once the data is in Kafka, you can use JDBC sink connectors to
> > > > stream
> > > > > > data from Kafka topics to the tables in SQL Server.
> > > > > >
> > > > > > Note that this is a very simplified view of how you can achieve
> > your
> > > > goal
> > > > > > of streaming changes from MySQL to SQL Server and I would
> recommend
> > > > > reading
> > > > > > the documentation of the individual connectors and the Kafka
> > Connect
> > > > > > framework to understand how to make it work for your usecase.
> > > > > >
> > > > > > Thanks for your interest on Apache Kafka!
> > > > > >
> > > > > > Thanks!
> > > > > > Sagar.
> > > > > >
> > > > > >
> > > > > > On Mon, Jun 26, 2023 at 11:42 AM Avani Panchal
> > > > > > <av...@adit.com.invalid> wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > > In my application I want to sync my client's data to my SQL
> > > server.
> > > > at
> > > > > > > client place the database is MYSQL.
> > > > > > >
> > > > > > > How can I achieve this using Kafka? I read a lot of documents
> > but I
> > > > > don't
> > > > > > > understand which setup I need and how I can achieve it.
> > > > > > >
> > > > > > > I was also wondering about "Book a demo with Kafka" but didn't
> > find
> > > > it.
> > > > > > >
> > > > > > > Please help me.
> > > > > > >
> > > > > > > Thank you,
> > > > > > > Avani
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: How to Integrate MySQL database with Kafka? Need Demo
Posted by Avani Panchal <av...@adit.com.INVALID>.
Hi Roman,
Thanks for replying, I want to ask one question: can I create my own code
to integrate data using Kafka connectors?
Kind regards,
Avani Panchal
On Tue, Jun 27, 2023 at 12:29 PM Roman Schmitz <ro...@gmail.com>
wrote:
> Hi Avani,
>
> totally agree - depending on the way you'd like to integrate your source
> that would be probably Debezium for CDC or JDBC source that can be used
> either in a "full-load" as well as in an incremental way.
> As a sink JDBC sink with the corresponding driver for the DB. Kindly have a
> look at the Connect Documentation and some examples.
> If you'd like to test with runnable Docker-based examples, you can have a
> look at this Docker Playground Repo:
> https://github.com/vdesabou/kafka-docker-playground/tree/master/connect
>
> Kind regards,
> Roman
>
> Am Di., 27. Juni 2023 um 08:48 Uhr schrieb Avani Panchal
> <av...@adit.com.invalid>:
>
> > Hi Sagar,
> >
> > Thank you for your email.
> >
> > Thanks & Regards,
> > Avani Panchal
> >
> >
> > On Tue, Jun 27, 2023 at 12:15 PM Sagar <sa...@gmail.com>
> wrote:
> >
> > > Hi Avani,
> > >
> > > I already shared the documentation link for Kafka Connect. Let me share
> > it
> > > again: https://kafka.apache.org/documentation/#connect
> > >
> > > Regarding the connector documentation, you should be able to find them
> by
> > > just searching for JDBC source connector, JDBC sink connector and
> > Debezium
> > > connector for MySQL.
> > >
> > > Let me know if that works.
> > >
> > > Thanks!
> > > Sagar.
> > >
> > >
> > >
> > > On Mon, Jun 26, 2023 at 2:10 PM Avani Panchal
> > > <av...@adit.com.invalid> wrote:
> > >
> > > > Hi Sagar,
> > > >
> > > > Thank you for the information, you solved our confusion.
> > > > I also saw lots of links for documentation on Kafka, but I am
> confused
> > > > which document I should use.
> > > > So can you share the proper link from where I can read the documents.
> > > >
> > > > Thanks,
> > > > Avani Panchal
> > > >
> > > >
> > > > On Mon, Jun 26, 2023 at 1:48 PM Sagar <sa...@gmail.com>
> > wrote:
> > > >
> > > > > Hey Avani,
> > > > >
> > > > > Kafka Connect <https://kafka.apache.org/documentation/#connect> is
> > the
> > > > > tool
> > > > > to use when you want to stream data to/from Kafka via external
> > systems.
> > > > One
> > > > > would typically configure connectors which allow streaming data
> > to/from
> > > > > Kafka. There are 2 types of connectors:
> > > > > 1) Source Connectors: Which stream data from external systems like
> > > > > databases etc to Kafka and
> > > > > 2) Sink Connectors: Which stream data from Kafka to external
> systems.
> > > > >
> > > > > Since you want to stream data from MySQL to SQL Server, with Kafka
> > > > Connect
> > > > > it would be a 2 step process:
> > > > >
> > > > > 1) Capture changes from MySQL to Kafka using connectors like JDBC
> > > source
> > > > > connector or Debezium MySQL connector.
> > > > > 2) Once the data is in Kafka, you can use JDBC sink connectors to
> > > stream
> > > > > data from Kafka topics to the tables in SQL Server.
> > > > >
> > > > > Note that this is a very simplified view of how you can achieve
> your
> > > goal
> > > > > of streaming changes from MySQL to SQL Server and I would recommend
> > > > reading
> > > > > the documentation of the individual connectors and the Kafka
> Connect
> > > > > framework to understand how to make it work for your usecase.
> > > > >
> > > > > Thanks for your interest on Apache Kafka!
> > > > >
> > > > > Thanks!
> > > > > Sagar.
> > > > >
> > > > >
> > > > > On Mon, Jun 26, 2023 at 11:42 AM Avani Panchal
> > > > > <av...@adit.com.invalid> wrote:
> > > > >
> > > > > > Hi,
> > > > > > In my application I want to sync my client's data to my SQL
> > server.
> > > at
> > > > > > client place the database is MYSQL.
> > > > > >
> > > > > > How can I achieve this using Kafka? I read a lot of documents
> but I
> > > > don't
> > > > > > understand which setup I need and how I can achieve it.
> > > > > >
> > > > > > I was also wondering about "Book a demo with Kafka" but didn't
> find
> > > it.
> > > > > >
> > > > > > Please help me.
> > > > > >
> > > > > > Thank you,
> > > > > > Avani
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: How to Integrate MySQL database with Kafka? Need Demo
Posted by Roman Schmitz <ro...@gmail.com>.
Hi Avani,
totally agree - depending on the way you'd like to integrate your source
that would be probably Debezium for CDC or JDBC source that can be used
either in a "full-load" as well as in an incremental way.
As a sink JDBC sink with the corresponding driver for the DB. Kindly have a
look at the Connect Documentation and some examples.
If you'd like to test with runnable Docker-based examples, you can have a
look at this Docker Playground Repo:
https://github.com/vdesabou/kafka-docker-playground/tree/master/connect
Kind regards,
Roman
Am Di., 27. Juni 2023 um 08:48 Uhr schrieb Avani Panchal
<av...@adit.com.invalid>:
> Hi Sagar,
>
> Thank you for your email.
>
> Thanks & Regards,
> Avani Panchal
>
>
> On Tue, Jun 27, 2023 at 12:15 PM Sagar <sa...@gmail.com> wrote:
>
> > Hi Avani,
> >
> > I already shared the documentation link for Kafka Connect. Let me share
> it
> > again: https://kafka.apache.org/documentation/#connect
> >
> > Regarding the connector documentation, you should be able to find them by
> > just searching for JDBC source connector, JDBC sink connector and
> Debezium
> > connector for MySQL.
> >
> > Let me know if that works.
> >
> > Thanks!
> > Sagar.
> >
> >
> >
> > On Mon, Jun 26, 2023 at 2:10 PM Avani Panchal
> > <av...@adit.com.invalid> wrote:
> >
> > > Hi Sagar,
> > >
> > > Thank you for the information, you solved our confusion.
> > > I also saw lots of links for documentation on Kafka, but I am confused
> > > which document I should use.
> > > So can you share the proper link from where I can read the documents.
> > >
> > > Thanks,
> > > Avani Panchal
> > >
> > >
> > > On Mon, Jun 26, 2023 at 1:48 PM Sagar <sa...@gmail.com>
> wrote:
> > >
> > > > Hey Avani,
> > > >
> > > > Kafka Connect <https://kafka.apache.org/documentation/#connect> is
> the
> > > > tool
> > > > to use when you want to stream data to/from Kafka via external
> systems.
> > > One
> > > > would typically configure connectors which allow streaming data
> to/from
> > > > Kafka. There are 2 types of connectors:
> > > > 1) Source Connectors: Which stream data from external systems like
> > > > databases etc to Kafka and
> > > > 2) Sink Connectors: Which stream data from Kafka to external systems.
> > > >
> > > > Since you want to stream data from MySQL to SQL Server, with Kafka
> > > Connect
> > > > it would be a 2 step process:
> > > >
> > > > 1) Capture changes from MySQL to Kafka using connectors like JDBC
> > source
> > > > connector or Debezium MySQL connector.
> > > > 2) Once the data is in Kafka, you can use JDBC sink connectors to
> > stream
> > > > data from Kafka topics to the tables in SQL Server.
> > > >
> > > > Note that this is a very simplified view of how you can achieve your
> > goal
> > > > of streaming changes from MySQL to SQL Server and I would recommend
> > > reading
> > > > the documentation of the individual connectors and the Kafka Connect
> > > > framework to understand how to make it work for your usecase.
> > > >
> > > > Thanks for your interest on Apache Kafka!
> > > >
> > > > Thanks!
> > > > Sagar.
> > > >
> > > >
> > > > On Mon, Jun 26, 2023 at 11:42 AM Avani Panchal
> > > > <av...@adit.com.invalid> wrote:
> > > >
> > > > > Hi,
> > > > > In my application I want to sync my client's data to my SQL
> server.
> > at
> > > > > client place the database is MYSQL.
> > > > >
> > > > > How can I achieve this using Kafka? I read a lot of documents but I
> > > don't
> > > > > understand which setup I need and how I can achieve it.
> > > > >
> > > > > I was also wondering about "Book a demo with Kafka" but didn't find
> > it.
> > > > >
> > > > > Please help me.
> > > > >
> > > > > Thank you,
> > > > > Avani
> > > > >
> > > >
> > >
> >
>
Re: How to Integrate MySQL database with Kafka? Need Demo
Posted by Avani Panchal <av...@adit.com.INVALID>.
Hi Sagar,
Thank you for your email.
Thanks & Regards,
Avani Panchal
On Tue, Jun 27, 2023 at 12:15 PM Sagar <sa...@gmail.com> wrote:
> Hi Avani,
>
> I already shared the documentation link for Kafka Connect. Let me share it
> again: https://kafka.apache.org/documentation/#connect
>
> Regarding the connector documentation, you should be able to find them by
> just searching for JDBC source connector, JDBC sink connector and Debezium
> connector for MySQL.
>
> Let me know if that works.
>
> Thanks!
> Sagar.
>
>
>
> On Mon, Jun 26, 2023 at 2:10 PM Avani Panchal
> <av...@adit.com.invalid> wrote:
>
> > Hi Sagar,
> >
> > Thank you for the information, you solved our confusion.
> > I also saw lots of links for documentation on Kafka, but I am confused
> > which document I should use.
> > So can you share the proper link from where I can read the documents.
> >
> > Thanks,
> > Avani Panchal
> >
> >
> > On Mon, Jun 26, 2023 at 1:48 PM Sagar <sa...@gmail.com> wrote:
> >
> > > Hey Avani,
> > >
> > > Kafka Connect <https://kafka.apache.org/documentation/#connect> is the
> > > tool
> > > to use when you want to stream data to/from Kafka via external systems.
> > One
> > > would typically configure connectors which allow streaming data to/from
> > > Kafka. There are 2 types of connectors:
> > > 1) Source Connectors: Which stream data from external systems like
> > > databases etc to Kafka and
> > > 2) Sink Connectors: Which stream data from Kafka to external systems.
> > >
> > > Since you want to stream data from MySQL to SQL Server, with Kafka
> > Connect
> > > it would be a 2 step process:
> > >
> > > 1) Capture changes from MySQL to Kafka using connectors like JDBC
> source
> > > connector or Debezium MySQL connector.
> > > 2) Once the data is in Kafka, you can use JDBC sink connectors to
> stream
> > > data from Kafka topics to the tables in SQL Server.
> > >
> > > Note that this is a very simplified view of how you can achieve your
> goal
> > > of streaming changes from MySQL to SQL Server and I would recommend
> > reading
> > > the documentation of the individual connectors and the Kafka Connect
> > > framework to understand how to make it work for your usecase.
> > >
> > > Thanks for your interest on Apache Kafka!
> > >
> > > Thanks!
> > > Sagar.
> > >
> > >
> > > On Mon, Jun 26, 2023 at 11:42 AM Avani Panchal
> > > <av...@adit.com.invalid> wrote:
> > >
> > > > Hi,
> > > > In my application I want to sync my client's data to my SQL server.
> at
> > > > client place the database is MYSQL.
> > > >
> > > > How can I achieve this using Kafka? I read a lot of documents but I
> > don't
> > > > understand which setup I need and how I can achieve it.
> > > >
> > > > I was also wondering about "Book a demo with Kafka" but didn't find
> it.
> > > >
> > > > Please help me.
> > > >
> > > > Thank you,
> > > > Avani
> > > >
> > >
> >
>
Re: How to Integrate MySQL database with Kafka? Need Demo
Posted by Sagar <sa...@gmail.com>.
Hi Avani,
I already shared the documentation link for Kafka Connect. Let me share it
again: https://kafka.apache.org/documentation/#connect
Regarding the connector documentation, you should be able to find them by
just searching for JDBC source connector, JDBC sink connector and Debezium
connector for MySQL.
Let me know if that works.
Thanks!
Sagar.
On Mon, Jun 26, 2023 at 2:10 PM Avani Panchal
<av...@adit.com.invalid> wrote:
> Hi Sagar,
>
> Thank you for the information, you solved our confusion.
> I also saw lots of links for documentation on Kafka, but I am confused
> which document I should use.
> So can you share the proper link from where I can read the documents.
>
> Thanks,
> Avani Panchal
>
>
> On Mon, Jun 26, 2023 at 1:48 PM Sagar <sa...@gmail.com> wrote:
>
> > Hey Avani,
> >
> > Kafka Connect <https://kafka.apache.org/documentation/#connect> is the
> > tool
> > to use when you want to stream data to/from Kafka via external systems.
> One
> > would typically configure connectors which allow streaming data to/from
> > Kafka. There are 2 types of connectors:
> > 1) Source Connectors: Which stream data from external systems like
> > databases etc to Kafka and
> > 2) Sink Connectors: Which stream data from Kafka to external systems.
> >
> > Since you want to stream data from MySQL to SQL Server, with Kafka
> Connect
> > it would be a 2 step process:
> >
> > 1) Capture changes from MySQL to Kafka using connectors like JDBC source
> > connector or Debezium MySQL connector.
> > 2) Once the data is in Kafka, you can use JDBC sink connectors to stream
> > data from Kafka topics to the tables in SQL Server.
> >
> > Note that this is a very simplified view of how you can achieve your goal
> > of streaming changes from MySQL to SQL Server and I would recommend
> reading
> > the documentation of the individual connectors and the Kafka Connect
> > framework to understand how to make it work for your usecase.
> >
> > Thanks for your interest on Apache Kafka!
> >
> > Thanks!
> > Sagar.
> >
> >
> > On Mon, Jun 26, 2023 at 11:42 AM Avani Panchal
> > <av...@adit.com.invalid> wrote:
> >
> > > Hi,
> > > In my application I want to sync my client's data to my SQL server. at
> > > client place the database is MYSQL.
> > >
> > > How can I achieve this using Kafka? I read a lot of documents but I
> don't
> > > understand which setup I need and how I can achieve it.
> > >
> > > I was also wondering about "Book a demo with Kafka" but didn't find it.
> > >
> > > Please help me.
> > >
> > > Thank you,
> > > Avani
> > >
> >
>
Re: How to Integrate MySQL database with Kafka? Need Demo
Posted by Avani Panchal <av...@adit.com.INVALID>.
Hi Sagar,
Thank you for the information, you solved our confusion.
I also saw lots of links for documentation on Kafka, but I am confused
which document I should use.
So can you share the proper link from where I can read the documents.
Thanks,
Avani Panchal
On Mon, Jun 26, 2023 at 1:48 PM Sagar <sa...@gmail.com> wrote:
> Hey Avani,
>
> Kafka Connect <https://kafka.apache.org/documentation/#connect> is the
> tool
> to use when you want to stream data to/from Kafka via external systems. One
> would typically configure connectors which allow streaming data to/from
> Kafka. There are 2 types of connectors:
> 1) Source Connectors: Which stream data from external systems like
> databases etc to Kafka and
> 2) Sink Connectors: Which stream data from Kafka to external systems.
>
> Since you want to stream data from MySQL to SQL Server, with Kafka Connect
> it would be a 2 step process:
>
> 1) Capture changes from MySQL to Kafka using connectors like JDBC source
> connector or Debezium MySQL connector.
> 2) Once the data is in Kafka, you can use JDBC sink connectors to stream
> data from Kafka topics to the tables in SQL Server.
>
> Note that this is a very simplified view of how you can achieve your goal
> of streaming changes from MySQL to SQL Server and I would recommend reading
> the documentation of the individual connectors and the Kafka Connect
> framework to understand how to make it work for your usecase.
>
> Thanks for your interest on Apache Kafka!
>
> Thanks!
> Sagar.
>
>
> On Mon, Jun 26, 2023 at 11:42 AM Avani Panchal
> <av...@adit.com.invalid> wrote:
>
> > Hi,
> > In my application I want to sync my client's data to my SQL server. at
> > client place the database is MYSQL.
> >
> > How can I achieve this using Kafka? I read a lot of documents but I don't
> > understand which setup I need and how I can achieve it.
> >
> > I was also wondering about "Book a demo with Kafka" but didn't find it.
> >
> > Please help me.
> >
> > Thank you,
> > Avani
> >
>
Re: How to Integrate MySQL database with Kafka? Need Demo
Posted by Sagar <sa...@gmail.com>.
Hey Avani,
Kafka Connect <https://kafka.apache.org/documentation/#connect> is the tool
to use when you want to stream data to/from Kafka via external systems. One
would typically configure connectors which allow streaming data to/from
Kafka. There are 2 types of connectors:
1) Source Connectors: Which stream data from external systems like
databases etc to Kafka and
2) Sink Connectors: Which stream data from Kafka to external systems.
Since you want to stream data from MySQL to SQL Server, with Kafka Connect
it would be a 2 step process:
1) Capture changes from MySQL to Kafka using connectors like JDBC source
connector or Debezium MySQL connector.
2) Once the data is in Kafka, you can use JDBC sink connectors to stream
data from Kafka topics to the tables in SQL Server.
Note that this is a very simplified view of how you can achieve your goal
of streaming changes from MySQL to SQL Server and I would recommend reading
the documentation of the individual connectors and the Kafka Connect
framework to understand how to make it work for your usecase.
Thanks for your interest on Apache Kafka!
Thanks!
Sagar.
On Mon, Jun 26, 2023 at 11:42 AM Avani Panchal
<av...@adit.com.invalid> wrote:
> Hi,
> In my application I want to sync my client's data to my SQL server. at
> client place the database is MYSQL.
>
> How can I achieve this using Kafka? I read a lot of documents but I don't
> understand which setup I need and how I can achieve it.
>
> I was also wondering about "Book a demo with Kafka" but didn't find it.
>
> Please help me.
>
> Thank you,
> Avani
>
Re: How to Integrate MySQL database with Kafka? Need Demo
Posted by Avani Panchal <av...@adit.com.INVALID>.
Hi Josep Prat,
Thank you for your reply, I was wondering from so many days you solved
my problem.
So you are from "Aiven", your company is providing a demo for Kafka?
Thanks,
Avani Panchal
On Mon, Jun 26, 2023 at 1:46 PM Josep Prat <jo...@aiven.io.invalid>
wrote:
> Hi Avani,
>
> Thanks for taking interest in Apache Kafka.
> Regarding the example code involving a MySQL database as a sink you can
> take a look at this blog post:
>
> https://aiven.io/developer/db-technology-migration-with-apache-kafka-and-kafka-connect
> The example in there contains more moving pieces, but I believe it contains
> your use case. Disclaimer, I'm employed by Aiven.
>
> Regarding the "Book a demo with Kafka" you won't find this on the
> kafka.apache.org page, because this is the project's website. For
> commercial offerings you should head over to the websites of the copmanies
> that offer Apache Kafka as a service. aiven.io, confluent.io and
> instaclustr.com should have an option to request a demo.
>
> Best,
>
> On Mon, Jun 26, 2023 at 8:12 AM Avani Panchal
> <av...@adit.com.invalid> wrote:
>
> > Hi,
> > In my application I want to sync my client's data to my SQL server. at
> > client place the database is MYSQL.
> >
> > How can I achieve this using Kafka? I read a lot of documents but I don't
> > understand which setup I need and how I can achieve it.
> >
> > I was also wondering about "Book a demo with Kafka" but didn't find it.
> >
> > Please help me.
> >
> > Thank you,
> > Avani
> >
>
>
> --
> [image: Aiven] <https://www.aiven.io>
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.prat@aiven.io | +491715557497
> aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud
> >
> <https://www.linkedin.com/company/aiven/> <
> https://twitter.com/aiven_io>
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>
Re: How to Integrate MySQL database with Kafka? Need Demo
Posted by Josep Prat <jo...@aiven.io.INVALID>.
Hi Avani,
Thanks for taking interest in Apache Kafka.
Regarding the example code involving a MySQL database as a sink you can
take a look at this blog post:
https://aiven.io/developer/db-technology-migration-with-apache-kafka-and-kafka-connect
The example in there contains more moving pieces, but I believe it contains
your use case. Disclaimer, I'm employed by Aiven.
Regarding the "Book a demo with Kafka" you won't find this on the
kafka.apache.org page, because this is the project's website. For
commercial offerings you should head over to the websites of the copmanies
that offer Apache Kafka as a service. aiven.io, confluent.io and
instaclustr.com should have an option to request a demo.
Best,
On Mon, Jun 26, 2023 at 8:12 AM Avani Panchal
<av...@adit.com.invalid> wrote:
> Hi,
> In my application I want to sync my client's data to my SQL server. at
> client place the database is MYSQL.
>
> How can I achieve this using Kafka? I read a lot of documents but I don't
> understand which setup I need and how I can achieve it.
>
> I was also wondering about "Book a demo with Kafka" but didn't find it.
>
> Please help me.
>
> Thank you,
> Avani
>
--
[image: Aiven] <https://www.aiven.io>
*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.prat@aiven.io | +491715557497
aiven.io <https://www.aiven.io> | <https://www.facebook.com/aivencloud>
<https://www.linkedin.com/company/aiven/> <https://twitter.com/aiven_io>
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
Amtsgericht Charlottenburg, HRB 209739 B