You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by KhajaAsmath Mohammed <md...@gmail.com> on 2020/02/21 13:16:45 UTC

Nifi -Incremental load from database 

Hi,

I am looking for solution to do incremental load from database once the initial load is done. Incremental should happen automatically with the flow . Any suggestions 

Does capturechangesql help in this case ? 

Thanks,
Asmath

Sent from my iPhone

Re: Nifi -Incremental load from database

Posted by Pierre Villard <pi...@gmail.com>.
Ok,

When it comes to only managing new inserts, it's something you can easily
do with QueryDatabaseTable as long as their is a column that is
incrementing with your newly inserted lines like an insert timestamp or a
key ID.

When it's about any kind of operation on your source database, then it's
really a CDC problem and you have to look at more specific solutions based
on your input database. As of now, there is a processor for MySQL [1], and
there are pull requests for Microsoft SQL [2] and PostgreSQL. You'd have to
look into that direction and check if it matches your requirements.

[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-cdc-mysql-nar/1.11.2/org.apache.nifi.cdc.mysql.processors.CaptureChangeMySQL/index.html
[2] https://github.com/apache/nifi/pull/2231
[3] https://github.com/apache/nifi/pull/4065

Thanks,
Pierre

Le ven. 21 févr. 2020 à 13:19, KhajaAsmath Mohammed <md...@gmail.com>
a écrit :

> Hi ,
>
> I am looking for both solutions actually.
>
> Thanks,
> Asmath
>
> Sent from my iPhone
>
> On Feb 21, 2020, at 1:47 PM, Pierre Villard <pi...@gmail.com>
> wrote:
>
> 
> Hi,
>
> When you mean incremental load, do you mean only newly inserted lines in
> the source database? or do you also want to capture deletes and updates?
>
> Thanks,
> Pierre
>
> Le ven. 21 févr. 2020 à 05:16, KhajaAsmath Mohammed <
> mdkhajaasmath@gmail.com> a écrit :
>
>> Hi,
>>
>> I am looking for solution to do incremental load from database once the
>> initial load is done. Incremental should happen automatically with the flow
>> . Any suggestions
>>
>> Does capturechangesql help in this case ?
>>
>> Thanks,
>> Asmath
>>
>> Sent from my iPhone
>
>

Re: Nifi -Incremental load from database

Posted by KhajaAsmath Mohammed <md...@gmail.com>.
Hi ,

I am looking for both solutions actually.

Thanks,
Asmath

Sent from my iPhone

> On Feb 21, 2020, at 1:47 PM, Pierre Villard <pi...@gmail.com> wrote:
> 
> 
> Hi,
> 
> When you mean incremental load, do you mean only newly inserted lines in the source database? or do you also want to capture deletes and updates?
> 
> Thanks,
> Pierre
> 
>> Le ven. 21 févr. 2020 à 05:16, KhajaAsmath Mohammed <md...@gmail.com> a écrit :
>> Hi,
>> 
>> I am looking for solution to do incremental load from database once the initial load is done. Incremental should happen automatically with the flow . Any suggestions 
>> 
>> Does capturechangesql help in this case ? 
>> 
>> Thanks,
>> Asmath
>> 
>> Sent from my iPhone

Re: Nifi -Incremental load from database

Posted by Pierre Villard <pi...@gmail.com>.
Hi,

When you mean incremental load, do you mean only newly inserted lines in
the source database? or do you also want to capture deletes and updates?

Thanks,
Pierre

Le ven. 21 févr. 2020 à 05:16, KhajaAsmath Mohammed <md...@gmail.com>
a écrit :

> Hi,
>
> I am looking for solution to do incremental load from database once the
> initial load is done. Incremental should happen automatically with the flow
> . Any suggestions
>
> Does capturechangesql help in this case ?
>
> Thanks,
> Asmath
>
> Sent from my iPhone