You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Manik <ma...@pinnacleseven.com> on 2016/09/29 10:15:27 UTC
nifi incremental fetch
Hi Team,
I am confused to use which processor is fit for incremental fetch in nifi.
In nifi My task is like i need to fetch the data from sql server and doing
Incremental fetch data and load into hadoop and i need to compare two datas
i.e(incremantal and normal datas) merged and load into hive .In this task
i already used some processor it didnt work properly .please give some about
this task .I already use this processor
QueryDatabasetable--->>--SplitAvro-->>--ConvertAvroToJSON-->>--ReplaceText-->>--LogAttributes-->>--PutHdfs-->>--MergeContent-->>UpdateAttribute-->>--PutHiveStreaming
In this processor I didnt get exact Incremental data as well as Base data
.please guide me to solve this tasks..
Thanks&Advance,
Manikandan
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/nifi-incremental-fetch-tp13489.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: nifi incremental fetch
Posted by Manik <ma...@pinnacleseven.com>.
Hi Matt,
Thanks for your quick response!!I am using Maximum value property as "id" but it can't work incremental fetch could you please some idea to finesh my task
Thanks & Regards,Manikandan Kolanjinathan
Junior Software Engineer
Pinnacle Seven Technologies 0422-4208736, 4506535 www.pinnacleseven.comDelivering your business apps on cloud!
From: Matt Burgess-2 [via Apache NiFi Developer List] <ml...@n7.nabble.com>
To: Manik <ma...@pinnacleseven.com>
Sent: Thursday, 29 September 2016 6:03 PM
Subject: Re: nifi incremental fetch
Manikandan,
QueryDatabaseTable (and GenerateTableFetch) should work for
incremental fetching of rows, provided that you have at least one
column that strictly increases (like a primary key with integer ID for
example). In that case you put the column name into the processor's
Maximum Value Columns property. The first time the processor runs, it
will get all rows from the table, but then it will keep track of the
largest value it has seen for the maximum value column. The next time
it runs, it will only retrieve rows whose values in that column are
greater than the currently observed maximum.
For your example, did you select a maximum value column and include it
in the processor's Maximum Value Columns property?
Regards,
Matt
On Thu, Sep 29, 2016 at 6:15 AM, Manik <[hidden email]> wrote:
> Hi Team,
>
> I am confused to use which processor is fit for incremental fetch in nifi.
> In nifi My task is like i need to fetch the data from sql server and doing
> Incremental fetch data and load into hadoop and i need to compare two datas
> i.e(incremantal and normal datas) merged and load into hive .In this task
> i already used some processor it didnt work properly .please give some about
> this task .I already use this processor
>
> QueryDatabasetable--->>--SplitAvro-->>--ConvertAvroToJSON-->>--ReplaceText-->>--LogAttributes-->>--PutHdfs-->>--MergeContent-->>UpdateAttribute-->>--PutHiveStreaming
>
> In this processor I didnt get exact Incremental data as well as Base data
> .please guide me to solve this tasks..
>
>
> Thanks&Advance,
>
> Manikandan
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/nifi-incremental-fetch-tp13489.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
If you reply to this email, your message will be added to the discussion below: http://apache-nifi-developer-list.39713.n7.nabble.com/nifi-incremental-fetch-tp13489p13490.html To unsubscribe from nifi incremental fetch, click here.
NAML
-----
Thanks,
Manee
--
View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/nifi-incremental-fetch-tp13489p13493.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: nifi incremental fetch
Posted by Matt Burgess <ma...@apache.org>.
Manikandan,
QueryDatabaseTable (and GenerateTableFetch) should work for
incremental fetching of rows, provided that you have at least one
column that strictly increases (like a primary key with integer ID for
example). In that case you put the column name into the processor's
Maximum Value Columns property. The first time the processor runs, it
will get all rows from the table, but then it will keep track of the
largest value it has seen for the maximum value column. The next time
it runs, it will only retrieve rows whose values in that column are
greater than the currently observed maximum.
For your example, did you select a maximum value column and include it
in the processor's Maximum Value Columns property?
Regards,
Matt
On Thu, Sep 29, 2016 at 6:15 AM, Manik <ma...@pinnacleseven.com> wrote:
> Hi Team,
>
> I am confused to use which processor is fit for incremental fetch in nifi.
> In nifi My task is like i need to fetch the data from sql server and doing
> Incremental fetch data and load into hadoop and i need to compare two datas
> i.e(incremantal and normal datas) merged and load into hive .In this task
> i already used some processor it didnt work properly .please give some about
> this task .I already use this processor
>
> QueryDatabasetable--->>--SplitAvro-->>--ConvertAvroToJSON-->>--ReplaceText-->>--LogAttributes-->>--PutHdfs-->>--MergeContent-->>UpdateAttribute-->>--PutHiveStreaming
>
> In this processor I didnt get exact Incremental data as well as Base data
> .please guide me to solve this tasks..
>
>
> Thanks&Advance,
>
> Manikandan
>
>
>
> --
> View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/nifi-incremental-fetch-tp13489.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.