You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by VIVEK KUMAR MISHRA 13BIT0066 <vi...@vit.ac.in> on 2017/02/27 13:35:27 UTC

Kafka Connect

How to use kafka connect using python to get information  about
update,delete and insertion  of data at various data sources?

RE: Kafka Connect

Posted by "Tauzell, Dave" <Da...@surescripts.com>.
Are you specifically talking about relational databases?    Kafka Connect has  a JDBC source (http://docs.confluent.io/3.1.1/connect/connect-jdbc/docs/source_connector.html) which can push data changes to kafka.  It can only run sql queries, though, so out of the box it will just get you updated and new rows.   If you want to get a list of changes you'll either need to build that into your schema or use something else that does CDC (Change Data Capture) on your source.

-Dave

-----Original Message-----
From: VIVEK KUMAR MISHRA 13BIT0066 [mailto:vivekkumar.mishra2013@vit.ac.in]
Sent: Monday, February 27, 2017 7:35 AM
To: users@kafka.apache.org
Subject: Kafka Connect

How to use kafka connect using python to get information  about update,delete and insertion  of data at various data sources?
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.

Re: Kafka Connect

Posted by Hans Jespersen <ha...@confluent.io>.
Maybe look at this Kafka source connector for salesforce https://github.com/jcustenborder/kafka-connect-salesforce

-hans

Sent from my iPhone

> On Feb 27, 2017, at 4:06 PM, VIVEK KUMAR MISHRA 13BIT0066 <vi...@vit.ac.in> wrote:
> 
> Actually my data sources are salesforce and mailchimp. i have developed an
> api that will fetch data from there but now i want that if any changes are
> happening in data of salesforce and mailchimp sources then that changes
> should be reflected in my topic data.
> 
> On Mon, Feb 27, 2017 at 8:54 PM, Tauzell, Dave <Dave.Tauzell@surescripts.com
>> wrote:
> 
>> Also, see this article on streaming changes from MySQL to kafka:
>> https://wecode.wepay.com/posts/streaming-databases-in-
>> realtime-with-mysql-debezium-kafka
>> 
>> -----Original Message-----
>> From: Tauzell, Dave
>> Sent: Monday, February 27, 2017 9:07 AM
>> To: users@kafka.apache.org
>> Subject: RE: Kafka Connect
>> 
>> Are you specifically talking about relational databases?    Kafka Connect
>> has  a JDBC source (http://docs.confluent.io/3.1.
>> 1/connect/connect-jdbc/docs/source_connector.html) which can push data
>> changes to kafka.  It can only run sql queries, though, so out of the box
>> it will just get you updated and new rows.   If you want to get a list of
>> changes you'll either need to build that into your schema or use something
>> else that does CDC (Change Data Capture) on your source.
>> 
>> -Dave
>> 
>> -----Original Message-----
>> From: VIVEK KUMAR MISHRA 13BIT0066 [mailto:vivekkumar.mishra2013@vit.ac.in
>> ]
>> Sent: Monday, February 27, 2017 7:35 AM
>> To: users@kafka.apache.org
>> Subject: Kafka Connect
>> 
>> How to use kafka connect using python to get information  about
>> update,delete and insertion  of data at various data sources?
>> This e-mail and any files transmitted with it are confidential, may
>> contain sensitive information, and are intended solely for the use of the
>> individual or entity to whom they are addressed. If you have received this
>> e-mail in error, please notify the sender by reply e-mail immediately and
>> destroy all copies of the e-mail and any attachments.
>> 

Re: Kafka Connect

Posted by VIVEK KUMAR MISHRA 13BIT0066 <vi...@vit.ac.in>.
Actually my data sources are salesforce and mailchimp. i have developed an
api that will fetch data from there but now i want that if any changes are
happening in data of salesforce and mailchimp sources then that changes
should be reflected in my topic data.

On Mon, Feb 27, 2017 at 8:54 PM, Tauzell, Dave <Dave.Tauzell@surescripts.com
> wrote:

> Also, see this article on streaming changes from MySQL to kafka:
> https://wecode.wepay.com/posts/streaming-databases-in-
> realtime-with-mysql-debezium-kafka
>
> -----Original Message-----
> From: Tauzell, Dave
> Sent: Monday, February 27, 2017 9:07 AM
> To: users@kafka.apache.org
> Subject: RE: Kafka Connect
>
> Are you specifically talking about relational databases?    Kafka Connect
> has  a JDBC source (http://docs.confluent.io/3.1.
> 1/connect/connect-jdbc/docs/source_connector.html) which can push data
> changes to kafka.  It can only run sql queries, though, so out of the box
> it will just get you updated and new rows.   If you want to get a list of
> changes you'll either need to build that into your schema or use something
> else that does CDC (Change Data Capture) on your source.
>
> -Dave
>
> -----Original Message-----
> From: VIVEK KUMAR MISHRA 13BIT0066 [mailto:vivekkumar.mishra2013@vit.ac.in
> ]
> Sent: Monday, February 27, 2017 7:35 AM
> To: users@kafka.apache.org
> Subject: Kafka Connect
>
> How to use kafka connect using python to get information  about
> update,delete and insertion  of data at various data sources?
> This e-mail and any files transmitted with it are confidential, may
> contain sensitive information, and are intended solely for the use of the
> individual or entity to whom they are addressed. If you have received this
> e-mail in error, please notify the sender by reply e-mail immediately and
> destroy all copies of the e-mail and any attachments.
>

RE: Kafka Connect

Posted by "Tauzell, Dave" <Da...@surescripts.com>.
Also, see this article on streaming changes from MySQL to kafka:  https://wecode.wepay.com/posts/streaming-databases-in-realtime-with-mysql-debezium-kafka

-----Original Message-----
From: Tauzell, Dave
Sent: Monday, February 27, 2017 9:07 AM
To: users@kafka.apache.org
Subject: RE: Kafka Connect

Are you specifically talking about relational databases?    Kafka Connect has  a JDBC source (http://docs.confluent.io/3.1.1/connect/connect-jdbc/docs/source_connector.html) which can push data changes to kafka.  It can only run sql queries, though, so out of the box it will just get you updated and new rows.   If you want to get a list of changes you'll either need to build that into your schema or use something else that does CDC (Change Data Capture) on your source.

-Dave

-----Original Message-----
From: VIVEK KUMAR MISHRA 13BIT0066 [mailto:vivekkumar.mishra2013@vit.ac.in]
Sent: Monday, February 27, 2017 7:35 AM
To: users@kafka.apache.org
Subject: Kafka Connect

How to use kafka connect using python to get information  about update,delete and insertion  of data at various data sources?
This e-mail and any files transmitted with it are confidential, may contain sensitive information, and are intended solely for the use of the individual or entity to whom they are addressed. If you have received this e-mail in error, please notify the sender by reply e-mail immediately and destroy all copies of the e-mail and any attachments.