You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Tomislav Novosel <to...@clearpeaks.com> on 2021/02/26 15:41:58 UTC

Stopping processor after MAX number of retries

Hi guys,

I want to stop the processor after exceeding maximum number of retries.
For that I'm using RetryFlowFile processor, after 5 times of retry, it routes
flowfile to retries_exceeded.

When that kicks in, I want to stop the processor which was retried 5 times.

What is the best approach? I have few ones:


  *   Execute shell script which sends request to nifi-api to set processor state to STOPPED
  *   Put InvokeHTTP processor to send request

The downside is, what if processor-id changes, e.g. deploying to another env or nifi restart, not sure about that.
Also, it is nifi cluster with authentication and SSL, so it complicates the things.

Maybe someone has much simpler approach, with backpressure or something.

Regards,
Tom

Re: Stopping processor after MAX number of retries

Posted by Matt Burgess <ma...@apache.org>.
There's an example template on the Example Dataflow Templates page [1]
called Retry_Count_Loop.xml [2], not sure what components it uses
though.

Regards,
Matt

[1] https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
[2] https://cwiki.apache.org/confluence/download/attachments/57904847/Retry_Count_Loop.xml?version=1&modificationDate=1433271239000&api=v2

On Mon, Mar 1, 2021 at 1:07 PM James McMahon <js...@gmail.com> wrote:
>
> Any recommended examples of successful DistributedMapCache implementations?
>
> On Fri, Feb 26, 2021 at 11:30 AM Andrew Grande <ap...@gmail.com> wrote:
>>
>> I saw it several times and I have a strong conviction this is an anti-pattern. A dataflow must not mess with start/stop state of processors or process groups.
>>
>> Instead, a flow is always running and one puts a conditional check to either not get the data in or reroute/deny it.
>>
>> If you need to coordinate between processors, use wait/notify.
>>
>> If you need to coordinate across nodes, consider DistributedMapCache with a variety of implementations.
>>
>> Finally, can use external stores directly if need to coordinate with other systems.
>>
>> These become part of the flow design.
>>
>> Andrew
>>
>> On Fri, Feb 26, 2021, 7:42 AM Tomislav Novosel <to...@clearpeaks.com> wrote:
>>>
>>> Hi guys,
>>>
>>>
>>>
>>> I want to stop the processor after exceeding maximum number of retries.
>>>
>>> For that I'm using RetryFlowFile processor, after 5 times of retry, it routes
>>>
>>> flowfile to retries_exceeded.
>>>
>>>
>>>
>>> When that kicks in, I want to stop the processor which was retried 5 times.
>>>
>>>
>>>
>>> What is the best approach? I have few ones:
>>>
>>>
>>>
>>> Execute shell script which sends request to nifi-api to set processor state to STOPPED
>>> Put InvokeHTTP processor to send request
>>>
>>>
>>>
>>> The downside is, what if processor-id changes, e.g. deploying to another env or nifi restart, not sure about that.
>>>
>>> Also, it is nifi cluster with authentication and SSL, so it complicates the things.
>>>
>>>
>>>
>>> Maybe someone has much simpler approach, with backpressure or something.
>>>
>>>
>>>
>>> Regards,
>>>
>>> Tom

Re: Stopping processor after MAX number of retries

Posted by James McMahon <js...@gmail.com>.
Any recommended examples of successful DistributedMapCache implementations?

On Fri, Feb 26, 2021 at 11:30 AM Andrew Grande <ap...@gmail.com> wrote:

> I saw it several times and I have a strong conviction this is an
> anti-pattern. A dataflow must not mess with start/stop state of processors
> or process groups.
>
> Instead, a flow is always running and one puts a conditional check to
> either not get the data in or reroute/deny it.
>
> If you need to coordinate between processors, use wait/notify.
>
> If you need to coordinate across nodes, consider DistributedMapCache with
> a variety of implementations.
>
> Finally, can use external stores directly if need to coordinate with other
> systems.
>
> These become part of the flow design.
>
> Andrew
>
> On Fri, Feb 26, 2021, 7:42 AM Tomislav Novosel <
> tomislav.novosel@clearpeaks.com> wrote:
>
>> Hi guys,
>>
>>
>>
>> I want to stop the processor after exceeding maximum number of retries.
>>
>> For that I'm using RetryFlowFile processor, after 5 times of retry, it
>> routes
>>
>> flowfile to retries_exceeded.
>>
>>
>>
>> When that kicks in, I want to stop the processor which was retried 5
>> times.
>>
>>
>>
>> What is the best approach? I have few ones:
>>
>>
>>
>>    - Execute shell script which sends request to nifi-api to set
>>    processor state to STOPPED
>>    - Put InvokeHTTP processor to send request
>>
>>
>>
>> The downside is, what if processor-id changes, e.g. deploying to another
>> env or nifi restart, not sure about that.
>>
>> Also, it is nifi cluster with authentication and SSL, so it complicates
>> the things.
>>
>>
>>
>> Maybe someone has much simpler approach, with backpressure or something.
>>
>>
>>
>> Regards,
>>
>> Tom
>>
>

Re: Stopping processor after MAX number of retries

Posted by Andrew Grande <ap...@gmail.com>.
I saw it several times and I have a strong conviction this is an
anti-pattern. A dataflow must not mess with start/stop state of processors
or process groups.

Instead, a flow is always running and one puts a conditional check to
either not get the data in or reroute/deny it.

If you need to coordinate between processors, use wait/notify.

If you need to coordinate across nodes, consider DistributedMapCache with a
variety of implementations.

Finally, can use external stores directly if need to coordinate with other
systems.

These become part of the flow design.

Andrew

On Fri, Feb 26, 2021, 7:42 AM Tomislav Novosel <
tomislav.novosel@clearpeaks.com> wrote:

> Hi guys,
>
>
>
> I want to stop the processor after exceeding maximum number of retries.
>
> For that I'm using RetryFlowFile processor, after 5 times of retry, it
> routes
>
> flowfile to retries_exceeded.
>
>
>
> When that kicks in, I want to stop the processor which was retried 5 times.
>
>
>
> What is the best approach? I have few ones:
>
>
>
>    - Execute shell script which sends request to nifi-api to set
>    processor state to STOPPED
>    - Put InvokeHTTP processor to send request
>
>
>
> The downside is, what if processor-id changes, e.g. deploying to another
> env or nifi restart, not sure about that.
>
> Also, it is nifi cluster with authentication and SSL, so it complicates
> the things.
>
>
>
> Maybe someone has much simpler approach, with backpressure or something.
>
>
>
> Regards,
>
> Tom
>