Re: Retry

[Adam Taft <ad...@adamtaft.com>]

Sorry for late reply on this thread ...

One of the old-school solutions for this problem is to use a disk buffer
outside of NiFi. Basically use PutFile to dump your failed items instead of
putting them into a retry loop that will eventually get shutdown due to
back-pressure. At some convenient interval, you can pick up the files from
your disk buffer to retry them again later (and if they fail again, they
just get written back to disk again). With the external disk buffer, you
can manage/avoid the backpressure situation and allow your flows to keep
running. Writing the flowfiles to disk as flowfile-v3 format (via
MergeContent) is probably the best.

As another take on this solution, I once wrote a
NonBlockingRouteOnAttribute processor that worked pretty much like
RouteOnAttribute, except that the destinations wouldn't block if
backpressured. Instead, flowfiles would be routed to a "blocked"
relationship to indicate the the selected downstream route wasn't
available, but allowed other routes to continue to function even if one
relationship was back-pressured. This was a convenient solution for a heavy
workflow that could intermittently fail on one of the downstream routes but
the other relationships could be kept up. It was kind of like the
"overflow" distribution strategy of DistributeLoad, but more explicit about
the downstream relationships (rather than just the "first" relationship).
Maybe that could inspire you if you can write your own processor (or maybe
I should dust off mine and contribute it here).

Cheers,

/Adam


On Fri, Jul 29, 2022 at 1:26 PM Joe Witt <jo...@gmail.com> wrote:

> Ah, I get what you mean.  The back-pressure has to be releasable somewhere
> under that model when the flow is a closed loop.  That could be routing
> things out when the destination queue is full using BalanceLoad (i think)
> or expiry of flowfiles (not desirable in some cases) or allowing some queue
> to grow quite large...
>
> It was a pain point both to setup and to understand.  This new mechanism
> certainly improves the game.  If you can't have the nice things just yet
> then the above is the mentality to work from.  Others might have some good
> tips to share ...
>
> Thanks
>
> On Fri, Jul 29, 2022 at 12:22 PM Joe Obernberger <
> joseph.obernberger@gmail.com> wrote:
>
>> Thank you Joe.  The type of logic I was going with doesn't work though -
>> hence the original question.  If things fail, it gets stuck and the
>> flowfiles don't more.  If I change the Back Pressure Object Threshold to a
>> larger value, then they will start running again; was hoping there was a
>> best practice to retrying.  If things fail, it will fill up those queues,
>> and stop running.
>>
>> -Joe
>> On 7/29/2022 3:18 PM, Joe Witt wrote:
>>
>> The alternative is setting the flow up with the type of logic you were
>> going with.  Using UpdateAttribute/RouteOnAttribute processors.  It was
>> pretty annoying for folks.  This new built-in capability has been met
>> with much joy.
>>
>> Thanks
>> Joe
>>
>> On Fri, Jul 29, 2022 at 12:16 PM Joe Obernberger <
>> joseph.obernberger@gmail.com> wrote:
>>
>>> Ah.  Thank you for this.  I didn't know about this new feature.  We have
>>> another system on an older NiFi that doesn't have the relationships tab; is
>>> there an alternative?
>>> On 7/29/2022 1:24 PM, Joe Witt wrote:
>>>
>>> Joe
>>>
>>> I'd ditch that processor entirely and just use the built-in retry
>>> capability on all components now.
>>>
>>>
>>> https://nifi.apache.org/docs/nifi-docs/html/user-guide.html#relationships-tab
>>>
>>> Make the failure relationship retry against the policies you'd like to
>>> see there.
>>>
>>> Thanks
>>>
>>> On Fri, Jul 29, 2022 at 10:17 AM Joe Obernberger <
>>> joseph.obernberger@gmail.com> wrote:
>>>
>>>> Hi All - I'm having an issue with RetryFlowFile where it will get stuck
>>>> if there are a lot of failures.  I think I'm using this thing wrong!  Is
>>>> there a best practice?
>>>>
>>>> I'm reading from a Kafka queue and calling a REST service.  If the REST
>>>> service goes down, then I'd like it to retry.
>>>>
>>>> Thank you!
>>>>
>>>> -Joe
>>>>
>>>>
>>>>
>>>> ------------------------------
>>>> [image: AVG logo] <https://www.avg.com/internet-security>
>>>>
>>>> This email has been checked for viruses by AVG antivirus software.
>>>> www.avg.com <https://www.avg.com/internet-security>
>>>>
>>>>
>>>> <#m_2475063508637957987_m_2823392557795393223_m_-4327657386256172964_m_1251243267382400656_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
>>>>
>>>