MySQL CDC solution with nifi

[xm...@gmail.com]

I am researching to implement   Change Data Capture(CDC) solution  by observing MySQL binlogs and routing that data through NiFi to detect inserts , updates and deletes and finally storing/sync the data into NoSQL(MongoDB) and HDFS.
For this use case the data in source system is normalized (OLTP) and sink system is denormalized  (OLAP), so I have to accumulate some data in NiFi to compose aggregated record and store it into MongoDB.

I would like to know if any  NiFi users building  CDC solution using NiFi?
Is NiFi a good choice for this kind of dataflow / real-time ETL? 

I found some Java based CDC frameworks here:
https://github.com/wushujames/mysql-cdc-projects/wiki

Thanks 
Sumanth

Re: MySQL CDC solution with nifi

[Joe Witt <jo...@gmail.com>]

Sumanth, Uli,

I think from a framework perspective NiFi is well suited for many ETL
use cases.  Where I think we fall short today is in the existing set
of processors/extensions to help you with it.  I write this knowing
full well ETL means a lot of different things to a lot of different
people.  My answer is in the traditional DB to DB type ETL mangling
cases (not the broader sense it is sometimes used in).  This is coming
up quite often though so I expect we'll end up seeing development
within the community here pretty soon.  Perhaps there are good open
source options we can integrate with.  Will look more at the list you
link to.

Thanks
Joe

On Wed, Oct 14, 2015 at 4:58 PM, Uli Bethke <ul...@sonra.io> wrote:
> I would also be interested in the question below.
> Anyone?
>
>
> On 14/10/2015 05:17, xmlking@gmail.com wrote:
>>
>> I am researching to implement   Change Data Capture(CDC) solution  by
>> observing MySQL binlogs and routing that data through NiFi to detect inserts
>> , updates and deletes and finally storing/sync the data into NoSQL(MongoDB)
>> and HDFS.
>> For this use case the data in source system is normalized (OLTP) and sink
>> system is denormalized  (OLAP), so I have to accumulate some data in NiFi to
>> compose aggregated record and store it into MongoDB.
>>
>> I would like to know if any  NiFi users building  CDC solution using NiFi?
>> Is NiFi a good choice for this kind of dataflow / real-time ETL?
>>
>> I found some Java based CDC frameworks here:
>> https://github.com/wushujames/mysql-cdc-projects/wiki
>>
>> Thanks
>> Sumanth
>>
>>
>
>

Re: MySQL CDC solution with nifi

[Uli Bethke <ul...@sonra.io>]

I would also be interested in the question below.
Anyone?

On 14/10/2015 05:17, xmlking@gmail.com wrote:
> I am researching to implement   Change Data Capture(CDC) solution  by observing MySQL binlogs and routing that data through NiFi to detect inserts , updates and deletes and finally storing/sync the data into NoSQL(MongoDB) and HDFS.
> For this use case the data in source system is normalized (OLTP) and sink system is denormalized  (OLAP), so I have to accumulate some data in NiFi to compose aggregated record and store it into MongoDB.
>
> I would like to know if any  NiFi users building  CDC solution using NiFi?
> Is NiFi a good choice for this kind of dataflow / real-time ETL?
>
> I found some Java based CDC frameworks here:
> https://github.com/wushujames/mysql-cdc-projects/wiki
>
> Thanks
> Sumanth
>
>