Retry deadlock

[Tomislav Novosel <to...@clearpeaks.com>]

Hi guys,

I have situation with retrying failed flowfile, but with much bigger number of the flowfiles.
The implementation is with RetryFlowfile processor after publishing flowfile(message) to AMQ queue.

If I set backpressure of failure relationship to 10 flowfiles and retry relationship from RetryFlowfile to 1000
flowfiles both queues will be sooner or later filled up and deadlock will kick in.

For example I'm sending this picture:

[Graphical user interface, application  Description automatically generated]

If failure is set to 10 files threshold and retry is 1000 files, they will eventually fill up and even when
Kafka broker is fixed, files will be in dead loop, failure filled up preventing PublishKafka to retry files from retry relationship.

I don't know how is this pattern good if we have milions of files and let's say Kafka broker is down - deadlock will kick in.

Is there some part I'm missing?

Thanks in advance,
Tom