You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Joe Obernberger <jo...@gmail.com> on 2021/06/17 16:59:19 UTC
RetryFlowFile
Hi All - I'm wondering if there is an approach to using the
RetryFlowFile that doesn't get 'stuck' if there are lots of failures.
I'm using InvokeHTTP and if it fails, the failure goes to a
RetryFlowFile process. The retry from the RetryFlowFile goes back to
InvokeHTTP. If the InvokeHTTP process fails for a long time, the
failure queue fills, the retry queue fills, and when InvokeHTTP is
brought back up, it won't start since both queues are full.
Any ideas?
Thank you!
-Joe Obernberger
Re: RetryFlowFile
Posted by Joe Obernberger <jo...@gmail.com>.
Oh - well - wow. That makes a lot of sense! Thank you Mark, I will try
that.
-Joe
On 6/18/2021 1:54 PM, Mark Payne wrote:
> Joe,
>
> The Retry processor should only be used if your intent is to do
> something like “try 5 times and then do something else.” If the intent
> is to just keep trying and avoid data loss, you can just route the
> failure relationship from InvokeHTTP back to InvokeHTTP with nothing
> in between. This is the typical pattern used in nifi.
>
> Thanks
> -Mark
>
> Sent from my iPhone
>
>> On Jun 18, 2021, at 1:37 PM, Joe Obernberger
>> <jo...@gmail.com> wrote:
>>
>>
>>
>> Thank you Arpad. We are using NiFi to call many REST endpoints with
>> lots of records. If one of the end points fails, we may not discover
>> it for a while, in which case these queues fill up. Then when the
>> endpoint comes back up, it won't process anything until we adjust the
>> queue size in NiFi.
>> Would love a better solution. I don't want to lose data, so I don't
>> want to drop the flowfiles. Basically, I just want it to stop
>> processing while the service (endpoint) is down, and then start up
>> again when the service comes back. I'm not familar with minifi; will
>> take a look.
>>
>> On 6/18/2021 11:57 AM, Arpad Boda wrote:
>>> Joe,
>>>
>>> I think that's currently a limitation in backpressure handling of NiFi.
>>> Single loopback connections are handled properly (those are
>>> excluded), although more complex loopbacks (such as this retry
>>> scenario) are not.
>>>
>>> Although my gut feeling says that your configuration leaves space
>>> for improvement. In case your both connections can fill up, you:
>>> -May have too many retries
>>> -Too big penalties for failures
>>> -Simply the capacity of the connections are not big enough
>>>
>>> What I would like to highlight here is that based on your average
>>> data throughput and the maximum time you would like to keep retrying
>>> sending the same flowfile, you should have a good estimation for the
>>> number of flowfiles you can have in your loopback connections in
>>> such a scenario. Apply some buffers and you should be good.
>>> You can also apply maximum age for flowfiles, so they get dropped
>>> after a while from these connections.
>>>
>>> The long term solution would be to introduce a more complex logic to
>>> handle loops when it comes to backpressure (such as the one in
>>> minifi cpp), but I guess that will take a while. I'm pretty sure
>>> that with a good flow design you can avoid this issue without it as
>>> well.
>>>
>>> Regards,
>>> Arpad
>>>
>>> On Thu, Jun 17, 2021 at 6:59 PM Joe Obernberger
>>> <jo...@gmail.com> wrote:
>>>
>>> Hi All - I'm wondering if there is an approach to using the
>>> RetryFlowFile that doesn't get 'stuck' if there are lots of
>>> failures.
>>> I'm using InvokeHTTP and if it fails, the failure goes to a
>>> RetryFlowFile process. The retry from the RetryFlowFile goes
>>> back to
>>> InvokeHTTP. If the InvokeHTTP process fails for a long time, the
>>> failure queue fills, the retry queue fills, and when InvokeHTTP is
>>> brought back up, it won't start since both queues are full.
>>>
>>> Any ideas?
>>> Thank you!
>>>
>>> -Joe Obernberger
>>>
>>>
>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>> Virus-free. www.avg.com
>>> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>>>
>>>
Re: RetryFlowFile
Posted by Mark Payne <ma...@hotmail.com>.
Joe,
The Retry processor should only be used if your intent is to do something like “try 5 times and then do something else.” If the intent is to just keep trying and avoid data loss, you can just route the failure relationship from InvokeHTTP back to InvokeHTTP with nothing in between. This is the typical pattern used in nifi.
Thanks
-Mark
Sent from my iPhone
On Jun 18, 2021, at 1:37 PM, Joe Obernberger <jo...@gmail.com> wrote:
Thank you Arpad. We are using NiFi to call many REST endpoints with lots of records. If one of the end points fails, we may not discover it for a while, in which case these queues fill up. Then when the endpoint comes back up, it won't process anything until we adjust the queue size in NiFi.
Would love a better solution. I don't want to lose data, so I don't want to drop the flowfiles. Basically, I just want it to stop processing while the service (endpoint) is down, and then start up again when the service comes back. I'm not familar with minifi; will take a look.
On 6/18/2021 11:57 AM, Arpad Boda wrote:
Joe,
I think that's currently a limitation in backpressure handling of NiFi.
Single loopback connections are handled properly (those are excluded), although more complex loopbacks (such as this retry scenario) are not.
Although my gut feeling says that your configuration leaves space for improvement. In case your both connections can fill up, you:
-May have too many retries
-Too big penalties for failures
-Simply the capacity of the connections are not big enough
What I would like to highlight here is that based on your average data throughput and the maximum time you would like to keep retrying sending the same flowfile, you should have a good estimation for the number of flowfiles you can have in your loopback connections in such a scenario. Apply some buffers and you should be good.
You can also apply maximum age for flowfiles, so they get dropped after a while from these connections.
The long term solution would be to introduce a more complex logic to handle loops when it comes to backpressure (such as the one in minifi cpp), but I guess that will take a while. I'm pretty sure that with a good flow design you can avoid this issue without it as well.
Regards,
Arpad
On Thu, Jun 17, 2021 at 6:59 PM Joe Obernberger <jo...@gmail.com>> wrote:
Hi All - I'm wondering if there is an approach to using the
RetryFlowFile that doesn't get 'stuck' if there are lots of failures.
I'm using InvokeHTTP and if it fails, the failure goes to a
RetryFlowFile process. The retry from the RetryFlowFile goes back to
InvokeHTTP. If the InvokeHTTP process fails for a long time, the
failure queue fills, the retry queue fills, and when InvokeHTTP is
brought back up, it won't start since both queues are full.
Any ideas?
Thank you!
-Joe Obernberger
[https://ipmcdn.avast.com/images/icons/icon-envelope-tick-green-avg-v1.png]<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free. www.avg.com<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
Re: RetryFlowFile
Posted by Joe Obernberger <jo...@gmail.com>.
Thank you Arpad. We are using NiFi to call many REST endpoints with
lots of records. If one of the end points fails, we may not discover it
for a while, in which case these queues fill up. Then when the endpoint
comes back up, it won't process anything until we adjust the queue size
in NiFi.
Would love a better solution. I don't want to lose data, so I don't
want to drop the flowfiles. Basically, I just want it to stop
processing while the service (endpoint) is down, and then start up again
when the service comes back. I'm not familar with minifi; will take a look.
On 6/18/2021 11:57 AM, Arpad Boda wrote:
> Joe,
>
> I think that's currently a limitation in backpressure handling of NiFi.
> Single loopback connections are handled properly (those are excluded),
> although more complex loopbacks (such as this retry scenario) are not.
>
> Although my gut feeling says that your configuration leaves space for
> improvement. In case your both connections can fill up, you:
> -May have too many retries
> -Too big penalties for failures
> -Simply the capacity of the connections are not big enough
>
> What I would like to highlight here is that based on your average data
> throughput and the maximum time you would like to keep retrying
> sending the same flowfile, you should have a good estimation for the
> number of flowfiles you can have in your loopback connections in such
> a scenario. Apply some buffers and you should be good.
> You can also apply maximum age for flowfiles, so they get dropped
> after a while from these connections.
>
> The long term solution would be to introduce a more complex logic to
> handle loops when it comes to backpressure (such as the one in minifi
> cpp), but I guess that will take a while. I'm pretty sure that with a
> good flow design you can avoid this issue without it as well.
>
> Regards,
> Arpad
>
> On Thu, Jun 17, 2021 at 6:59 PM Joe Obernberger
> <jo...@gmail.com> wrote:
>
> Hi All - I'm wondering if there is an approach to using the
> RetryFlowFile that doesn't get 'stuck' if there are lots of failures.
> I'm using InvokeHTTP and if it fails, the failure goes to a
> RetryFlowFile process. The retry from the RetryFlowFile goes back to
> InvokeHTTP. If the InvokeHTTP process fails for a long time, the
> failure queue fills, the retry queue fills, and when InvokeHTTP is
> brought back up, it won't start since both queues are full.
>
> Any ideas?
> Thank you!
>
> -Joe Obernberger
>
>
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
> Virus-free. www.avg.com
> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient>
>
>
> <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
Re: RetryFlowFile
Posted by Arpad Boda <ab...@apache.org>.
Joe,
I think that's currently a limitation in backpressure handling of NiFi.
Single loopback connections are handled properly (those are excluded),
although more complex loopbacks (such as this retry scenario) are not.
Although my gut feeling says that your configuration leaves space for
improvement. In case your both connections can fill up, you:
-May have too many retries
-Too big penalties for failures
-Simply the capacity of the connections are not big enough
What I would like to highlight here is that based on your average data
throughput and the maximum time you would like to keep retrying sending the
same flowfile, you should have a good estimation for the number of
flowfiles you can have in your loopback connections in such a scenario.
Apply some buffers and you should be good.
You can also apply maximum age for flowfiles, so they get dropped after a
while from these connections.
The long term solution would be to introduce a more complex logic to handle
loops when it comes to backpressure (such as the one in minifi cpp), but I
guess that will take a while. I'm pretty sure that with a good flow design
you can avoid this issue without it as well.
Regards,
Arpad
On Thu, Jun 17, 2021 at 6:59 PM Joe Obernberger <
joseph.obernberger@gmail.com> wrote:
> Hi All - I'm wondering if there is an approach to using the
> RetryFlowFile that doesn't get 'stuck' if there are lots of failures.
> I'm using InvokeHTTP and if it fails, the failure goes to a
> RetryFlowFile process. The retry from the RetryFlowFile goes back to
> InvokeHTTP. If the InvokeHTTP process fails for a long time, the
> failure queue fills, the retry queue fills, and when InvokeHTTP is
> brought back up, it won't start since both queues are full.
>
> Any ideas?
> Thank you!
>
> -Joe Obernberger
>
>