You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Manojkumar Ravichandran <se...@gmail.com> on 2017/06/08 06:43:03 UTC
Set priority to files based on date time value stored on attribute
Hi All,
I need to process the files based on the date time value stored on the
attribute
*For example:*
If the incoming files contains the following date time attribute values
*2017/06/07 16:57:02*
*2017/06/06 12:49:49*
*2017/06/06 11:09:28*
*2017/06/06 06:37:45*
I need to process the files based on the order of time that is oldest one
from the current time
First I want to access the file that contains below date time attribute
which is the oldest one among them from the current time
*i.e 2017/06/06 06:37:45*
and then below one,
*2017/06/06 11:09:28*
and then this
*2017/06/06 12:49:49*
so on* .... *
How can I achieve the above mentioned scenario ?
Regards,
Manoj kumar R
Re: Set priority to files based on date time value stored on attribute
Posted by Andre <an...@fucs.org>.
Koji,
One could convert date to epoch format which is incremental in nature.
Would that help?
On 8 Jun 2017 19:33, "Koji Kawamura" <ij...@gmail.com> wrote:
> Hi Manoj,
>
> I think EnforceOrder would not be useful in your case, as it expects
> the order to increases one by one (without skip).
> As Pierre suggested, I'd suggest using PriorityAttributePrioritizer.
>
> Thanks,
> Koji
>
> On Thu, Jun 8, 2017 at 3:50 PM, Pierre Villard
> <pi...@gmail.com> wrote:
> > Hi Manoj,
> >
> > You may want ot have a look at EnforceOrder processor [1] or simply the
> > prioritizers [2] of the connections (it depends of how your workflow is
> > working). The idea would be to extract the date as an attribute of your
> flow
> > file, convert into an integer (using expression language) and use it to
> > ensure order.
> >
> > [1]
> > https://nifi.apache.org/docs/nifi-docs/components/org.
> apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.
> EnforceOrder/index.html
> > [2]
> > https://nifi.apache.org/docs/nifi-docs/html/user-guide.
> html#prioritization
> >
> > Hope this helps.
> >
> >
> > 2017-06-08 8:43 GMT+02:00 Manojkumar Ravichandran <
> sendmailtomj@gmail.com>:
> >>
> >> Hi All,
> >>
> >> I need to process the files based on the date time value stored on the
> >> attribute
> >>
> >> For example:
> >>
> >> If the incoming files contains the following date time attribute values
> >>
> >> 2017/06/07 16:57:02
> >> 2017/06/06 12:49:49
> >> 2017/06/06 11:09:28
> >> 2017/06/06 06:37:45
> >>
> >> I need to process the files based on the order of time that is oldest
> one
> >> from the current time
> >>
> >> First I want to access the file that contains below date time attribute
> >> which is the oldest one among them from the current time
> >> i.e 2017/06/06 06:37:45
> >> and then below one,
> >> 2017/06/06 11:09:28
> >> and then this
> >> 2017/06/06 12:49:49
> >> so on ....
> >>
> >> How can I achieve the above mentioned scenario ?
> >>
> >> Regards,
> >> Manoj kumar R
> >
> >
>
Re: Set priority to files based on date time value stored on attribute
Posted by Koji Kawamura <ij...@gmail.com>.
Hi Manoj,
I think EnforceOrder would not be useful in your case, as it expects
the order to increases one by one (without skip).
As Pierre suggested, I'd suggest using PriorityAttributePrioritizer.
Thanks,
Koji
On Thu, Jun 8, 2017 at 3:50 PM, Pierre Villard
<pi...@gmail.com> wrote:
> Hi Manoj,
>
> You may want ot have a look at EnforceOrder processor [1] or simply the
> prioritizers [2] of the connections (it depends of how your workflow is
> working). The idea would be to extract the date as an attribute of your flow
> file, convert into an integer (using expression language) and use it to
> ensure order.
>
> [1]
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.EnforceOrder/index.html
> [2]
> https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#prioritization
>
> Hope this helps.
>
>
> 2017-06-08 8:43 GMT+02:00 Manojkumar Ravichandran <se...@gmail.com>:
>>
>> Hi All,
>>
>> I need to process the files based on the date time value stored on the
>> attribute
>>
>> For example:
>>
>> If the incoming files contains the following date time attribute values
>>
>> 2017/06/07 16:57:02
>> 2017/06/06 12:49:49
>> 2017/06/06 11:09:28
>> 2017/06/06 06:37:45
>>
>> I need to process the files based on the order of time that is oldest one
>> from the current time
>>
>> First I want to access the file that contains below date time attribute
>> which is the oldest one among them from the current time
>> i.e 2017/06/06 06:37:45
>> and then below one,
>> 2017/06/06 11:09:28
>> and then this
>> 2017/06/06 12:49:49
>> so on ....
>>
>> How can I achieve the above mentioned scenario ?
>>
>> Regards,
>> Manoj kumar R
>
>
Re: Set priority to files based on date time value stored on attribute
Posted by Koji Kawamura <ij...@gmail.com>.
Hi Andre,
Epoch format is indeed incremental, but EnforceOrder needs every
number without skip.
Unless the use-case guarantees incoming FlowFiles has a perfect set of
incremental numbers in successful flow, EnforceOrder is not helpful.
E.g. 1, 5, 4, 2, 3 can be enforced by EnforceOrder
but 100, 500, 400, 200, 300 can NOT. But PriorityAttributePrioritizer
can sort these FlowFiles.
However, a caveat is, PriorityAttributePrioritizer does not wait if
some incoming FlowFiles are delayed.
If 1, 5 are queued when a processor is scheduled,
PriorityAttributePrioritizer picks 1 and 5. It does not care missing
numbers.
On the other hand, EnforceOrder does wait.
If 1, 5 are queued, EnforceOrder transfers 1, then keep 5 wait.
Then if 4, 2, 3 arrive, EnforceOrder transfers 2, 3, 4, then 5
Thanks,
Koji
On Thu, Jun 8, 2017 at 9:28 PM, Manojkumar Ravichandran
<se...@gmail.com> wrote:
> Hi Pierre,
>
>
> After converting those date time format in to integer (using expression
> language),I can able to process the file as per the requirement
> by setting those integer values to the priority attribute and process those
> files based on that priority.
>
> Thanks for your guidance
>
>
> Regards,
> Manoj kumar R
>
>
> On Thu, Jun 8, 2017 at 8:50 AM, Pierre Villard <pi...@gmail.com>
> wrote:
>>
>> Hi Manoj,
>>
>> You may want ot have a look at EnforceOrder processor [1] or simply the
>> prioritizers [2] of the connections (it depends of how your workflow is
>> working). The idea would be to extract the date as an attribute of your flow
>> file, convert into an integer (using expression language) and use it to
>> ensure order.
>>
>> [1]
>> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.EnforceOrder/index.html
>> [2]
>> https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#prioritization
>>
>> Hope this helps.
>>
>>
>> 2017-06-08 8:43 GMT+02:00 Manojkumar Ravichandran
>> <se...@gmail.com>:
>>>
>>> Hi All,
>>>
>>> I need to process the files based on the date time value stored on the
>>> attribute
>>>
>>> For example:
>>>
>>> If the incoming files contains the following date time attribute values
>>>
>>> 2017/06/07 16:57:02
>>> 2017/06/06 12:49:49
>>> 2017/06/06 11:09:28
>>> 2017/06/06 06:37:45
>>>
>>> I need to process the files based on the order of time that is oldest one
>>> from the current time
>>>
>>> First I want to access the file that contains below date time attribute
>>> which is the oldest one among them from the current time
>>> i.e 2017/06/06 06:37:45
>>> and then below one,
>>> 2017/06/06 11:09:28
>>> and then this
>>> 2017/06/06 12:49:49
>>> so on ....
>>>
>>> How can I achieve the above mentioned scenario ?
>>>
>>> Regards,
>>> Manoj kumar R
>>
>>
>
Re: Set priority to files based on date time value stored on attribute
Posted by Manojkumar Ravichandran <se...@gmail.com>.
Hi Pierre,
After converting those date time format in to integer (using expression
language),I can able to process the file as per the requirement
by setting those integer values to the priority attribute and process those
files based on that priority.
Thanks for your guidance
Regards,
Manoj kumar R
On Thu, Jun 8, 2017 at 8:50 AM, Pierre Villard <pi...@gmail.com>
wrote:
> Hi Manoj,
>
> You may want ot have a look at EnforceOrder processor [1] or simply the
> prioritizers [2] of the connections (it depends of how your workflow is
> working). The idea would be to extract the date as an attribute of your
> flow file, convert into an integer (using expression language) and use it
> to ensure order.
>
> [1] https://nifi.apache.org/docs/nifi-docs/components/org.
> apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.
> EnforceOrder/index.html
> [2] https://nifi.apache.org/docs/nifi-docs/html/user-guide.
> html#prioritization
>
> Hope this helps.
>
>
> 2017-06-08 8:43 GMT+02:00 Manojkumar Ravichandran <se...@gmail.com>
> :
>
>> Hi All,
>>
>> I need to process the files based on the date time value stored on the
>> attribute
>>
>> *For example:*
>>
>> If the incoming files contains the following date time attribute values
>>
>> *2017/06/07 16:57:02*
>> *2017/06/06 12:49:49*
>> *2017/06/06 11:09:28*
>> *2017/06/06 06:37:45*
>>
>> I need to process the files based on the order of time that is oldest one
>> from the current time
>>
>> First I want to access the file that contains below date time attribute
>> which is the oldest one among them from the current time
>> *i.e 2017/06/06 06:37:45*
>> and then below one,
>> *2017/06/06 11:09:28*
>> and then this
>> *2017/06/06 12:49:49*
>> so on* .... *
>>
>> How can I achieve the above mentioned scenario ?
>>
>> Regards,
>> Manoj kumar R
>>
>
>
Re: Set priority to files based on date time value stored on attribute
Posted by Pierre Villard <pi...@gmail.com>.
Hi Manoj,
You may want ot have a look at EnforceOrder processor [1] or simply the
prioritizers [2] of the connections (it depends of how your workflow is
working). The idea would be to extract the date as an attribute of your
flow file, convert into an integer (using expression language) and use it
to ensure order.
[1]
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-standard-nar/1.2.0/org.apache.nifi.processors.standard.EnforceOrder/index.html
[2]
https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#prioritization
Hope this helps.
2017-06-08 8:43 GMT+02:00 Manojkumar Ravichandran <se...@gmail.com>:
> Hi All,
>
> I need to process the files based on the date time value stored on the
> attribute
>
> *For example:*
>
> If the incoming files contains the following date time attribute values
>
> *2017/06/07 16:57:02*
> *2017/06/06 12:49:49*
> *2017/06/06 11:09:28*
> *2017/06/06 06:37:45*
>
> I need to process the files based on the order of time that is oldest one
> from the current time
>
> First I want to access the file that contains below date time attribute
> which is the oldest one among them from the current time
> *i.e 2017/06/06 06:37:45*
> and then below one,
> *2017/06/06 11:09:28*
> and then this
> *2017/06/06 12:49:49*
> so on* .... *
>
> How can I achieve the above mentioned scenario ?
>
> Regards,
> Manoj kumar R
>