You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Selvam Raman <se...@gmail.com> on 2016/09/19 11:17:23 UTC
Periodic delta pulls from a data source
Hi,
We have a requirement to pull data periodically from a data source. For
this to work we would like nifi to keep track of the last id or time stamp
that was pulled successfully so that the next pull starts from that point.
Let me know if nifi supports this? If yes how do we configure?
--
Selvam Raman
"லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
Re: Periodic delta pulls from a data source
Posted by Selvam Raman <se...@gmail.com>.
Hi,
We are making OAI-PMH requests invoked within Http.
On Tue, Sep 20, 2016 at 9:31 AM, Pierre Villard <pierre.villard.fr@gmail.com
> wrote:
> Hi Selvam,
>
> Supposing that your source if a SQL-like source, you should have a look at
> QueryDatabaseTable [1] processor. It proposes a 'Maximum-value Columns'
> that gives you the possibility to specify the column containing an ID and
> or timestamp. The processor will keep track of the maximum value for each
> column that has been returned since the processor started running. This can
> be used to retrieve only those rows that have been added/updated since the
> last retrieval. Note that some JDBC types such as bit/boolean are not
> conducive to maintaining maximum value, so columns of these types should
> not be listed in this property, and will result in error(s) during
> processing. If no columns are provided, all rows from the table will be
> considered, which could have a performance impact.
>
> [1] https://nifi.apache.org/docs/nifi-docs/
>
> Pierre
>
>
> 2016-09-20 8:50 GMT+02:00 Selvam Raman <se...@gmail.com>:
>
>>
>> > Hi,
>> >
>> > We have a requirement to pull data periodically from a data source. For
>> this to work we would like nifi to keep track of the last id or time stamp
>> that was pulled successfully so that the next pull starts from that point.
>> >
>> > Let me know if nifi supports this? If yes how do we configure?
>> >
>> > --
>> > Selvam Raman
>> > "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>>
>
>
--
Selvam Raman
"லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
Re: Periodic delta pulls from a data source
Posted by Pierre Villard <pi...@gmail.com>.
Hi Selvam,
Supposing that your source if a SQL-like source, you should have a look at
QueryDatabaseTable [1] processor. It proposes a 'Maximum-value Columns'
that gives you the possibility to specify the column containing an ID and
or timestamp. The processor will keep track of the maximum value for each
column that has been returned since the processor started running. This can
be used to retrieve only those rows that have been added/updated since the
last retrieval. Note that some JDBC types such as bit/boolean are not
conducive to maintaining maximum value, so columns of these types should
not be listed in this property, and will result in error(s) during
processing. If no columns are provided, all rows from the table will be
considered, which could have a performance impact.
[1] https://nifi.apache.org/docs/nifi-docs/
Pierre
2016-09-20 8:50 GMT+02:00 Selvam Raman <se...@gmail.com>:
>
> > Hi,
> >
> > We have a requirement to pull data periodically from a data source. For
> this to work we would like nifi to keep track of the last id or time stamp
> that was pulled successfully so that the next pull starts from that point.
> >
> > Let me know if nifi supports this? If yes how do we configure?
> >
> > --
> > Selvam Raman
> > "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>
Periodic delta pulls from a data source
Posted by Selvam Raman <se...@gmail.com>.
> Hi,
>
> We have a requirement to pull data periodically from a data source. For
this to work we would like nifi to keep track of the last id or time stamp
that was pulled successfully so that the next pull starts from that point.
>
> Let me know if nifi supports this? If yes how do we configure?
>
> --
> Selvam Raman
> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
Re: Periodic delta pulls from a data source
Posted by Andre <an...@fucs.org>.
Selvam,
Usually this sort of conversation is better addressed within the NIFI users
list.
Have you tried asking over there?
Cheers
On Mon, Sep 19, 2016 at 9:17 PM, Selvam Raman <se...@gmail.com> wrote:
> Hi,
>
> We have a requirement to pull data periodically from a data source. For
> this to work we would like nifi to keep track of the last id or time stamp
> that was pulled successfully so that the next pull starts from that point.
>
> Let me know if nifi supports this? If yes how do we configure?
>
> --
> Selvam Raman
> "லஞ்சம் தவிர்த்து நெஞ்சம் நிமிர்த்து"
>