You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Luca Giovannini <Lu...@dedagroup.it> on 2020/12/16 16:36:19 UTC
Wait processor doesn't route expired flowfiles to 'expired'
relationship
Hi All,
I am using the Wait/Notify couple of processors, with a setting of the Wait processor as follows:
* Wait mode = keep in the upstream connection
* Expiration duration = 10 min
* Wait penalty duration = 30 sec
I use the Wait/Notify in a split/merge situation where al the split parts have to be processed before the Wait is released.
Occasionally, one of the branches of the split fails or takes too much time and that is taken care with an “expired” branch of the Wait processor.
The flow runs well for days but then slowly the incoming connection to the Wait processor starts to grow, and checking the “queued duration” and “lineage duration” of the flowfiles in the queue list I see that they are much older than the “Expiration duration” setting (days vs. 10 minutes). I have also noticed that, when this happens, if I change the “expiration duration” setting to a smaller value the Wait processor starts to clean up the queue by routing the flowfiles to expired. I have seen this problem happening with all sorts of different values for “Expiration duration” and “Wait penalty”, the ones that I provided above are just an example.
I am using NiFi 1.11.4.
What can be happening?
This unpredictable behaviour is blocking the whole flow and making the Wait processor unusable for me ☹
Thank you very much for your help and support!
Luca
Le informazioni contenute in questo messaggio di posta elettronica sono riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o forma. Qualora Lei non fosse la persona destinataria del presente messaggio, La invitiamo a non diffonderlo e ad eliminarlo, dandone gentilmente comunicazione al mittente.
The information included in this e-mail and any attachments are confidential and may also be privileged. If you are not the correct recipient, you are kindly requested to notify the sender immediately, to cancel it and not to disclose the contents to any other person.
R: Wait processor doesn't route expired flowfiles to 'expired'
relationship
Posted by Luca Giovannini <Lu...@dedagroup.it>.
Thanks a lot, Koji, for your insight!
I will definitely look into it and see if it solves my problem.
Best regards,
Luca
-----Messaggio originale-----
Da: Koji Kawamura <ij...@gmail.com>
Inviato: lunedì 21 dicembre 2020 03:50
A: users@nifi.apache.org
Oggetto: Re: Wait processor doesn't route expired flowfiles to 'expired' relationship
**ATTENZIONE** Questo messaggio proviene da un ACCOUNT ESTERNO, presta attenzione ad eventuali link o allegati al suo interno.
Hi Luca,
Sorry to hear that you are having an issue with Wait processor.
By looking at the code and testing it locally, I couldn't find a cause of the issue nor reproduce it.
However, theoretically such a situation can happen when there are too many queued FlowFiles in the connection in front of the Wait processor exceeding the processor's throughput. Especially where the Wait processor is busy processing the recently added FlowFiles and not being able to take care of older FlowFiles.
By default, the order of processing queued FlowFiles is undefined.
To eliminate one of the possible uncertainties, I'd recommend using OldestFlowFileFirstPrioritizer at the incoming connection, so that older FlowFiles can be processed first.
Hope this helps,
Koji
On Thu, Dec 17, 2020 at 1:36 AM Luca Giovannini <Lu...@dedagroup.it> wrote:
>
>
>
> Hi All,
>
>
>
> I am using the Wait/Notify couple of processors, with a setting of the Wait processor as follows:
>
> Wait mode = keep in the upstream connection Expiration duration = 10
> min Wait penalty duration = 30 sec
>
>
>
> I use the Wait/Notify in a split/merge situation where al the split parts have to be processed before the Wait is released.
>
> Occasionally, one of the branches of the split fails or takes too much time and that is taken care with an “expired” branch of the Wait processor.
>
>
>
> The flow runs well for days but then slowly the incoming connection to the Wait processor starts to grow, and checking the “queued duration” and “lineage duration” of the flowfiles in the queue list I see that they are much older than the “Expiration duration” setting (days vs. 10 minutes). I have also noticed that, when this happens, if I change the “expiration duration” setting to a smaller value the Wait processor starts to clean up the queue by routing the flowfiles to expired. I have seen this problem happening with all sorts of different values for “Expiration duration” and “Wait penalty”, the ones that I provided above are just an example.
>
> I am using NiFi 1.11.4.
>
>
>
> What can be happening?
>
> This unpredictable behaviour is blocking the whole flow and making the
> Wait processor unusable for me ☹
>
>
>
> Thank you very much for your help and support!
>
>
>
> Luca
>
>
>
> Le informazioni contenute in questo messaggio di posta elettronica sono riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o forma. Qualora Lei non fosse la persona destinataria del presente messaggio, La invitiamo a non diffonderlo e ad eliminarlo, dandone gentilmente comunicazione al mittente.
>
> The information included in this e-mail and any attachments are confidential and may also be privileged. If you are not the correct recipient, you are kindly requested to notify the sender immediately, to cancel it and not to disclose the contents to any other person.
Re: Wait processor doesn't route expired flowfiles to 'expired' relationship
Posted by Koji Kawamura <ij...@gmail.com>.
Hi Luca,
Sorry to hear that you are having an issue with Wait processor.
By looking at the code and testing it locally, I couldn't find a cause
of the issue nor reproduce it.
However, theoretically such a situation can happen when there are too
many queued FlowFiles in the connection in front of the Wait processor
exceeding the processor's throughput. Especially where the Wait
processor is busy processing the recently added FlowFiles and not
being able to take care of older FlowFiles.
By default, the order of processing queued FlowFiles is undefined.
To eliminate one of the possible uncertainties, I'd recommend using
OldestFlowFileFirstPrioritizer at the incoming connection, so that
older FlowFiles can be processed first.
Hope this helps,
Koji
On Thu, Dec 17, 2020 at 1:36 AM Luca Giovannini
<Lu...@dedagroup.it> wrote:
>
>
>
> Hi All,
>
>
>
> I am using the Wait/Notify couple of processors, with a setting of the Wait processor as follows:
>
> Wait mode = keep in the upstream connection
> Expiration duration = 10 min
> Wait penalty duration = 30 sec
>
>
>
> I use the Wait/Notify in a split/merge situation where al the split parts have to be processed before the Wait is released.
>
> Occasionally, one of the branches of the split fails or takes too much time and that is taken care with an “expired” branch of the Wait processor.
>
>
>
> The flow runs well for days but then slowly the incoming connection to the Wait processor starts to grow, and checking the “queued duration” and “lineage duration” of the flowfiles in the queue list I see that they are much older than the “Expiration duration” setting (days vs. 10 minutes). I have also noticed that, when this happens, if I change the “expiration duration” setting to a smaller value the Wait processor starts to clean up the queue by routing the flowfiles to expired. I have seen this problem happening with all sorts of different values for “Expiration duration” and “Wait penalty”, the ones that I provided above are just an example.
>
> I am using NiFi 1.11.4.
>
>
>
> What can be happening?
>
> This unpredictable behaviour is blocking the whole flow and making the Wait processor unusable for me ☹
>
>
>
> Thank you very much for your help and support!
>
>
>
> Luca
>
>
>
> Le informazioni contenute in questo messaggio di posta elettronica sono riservate e confidenziali e ne e' vietata la diffusione in qualsiasi modo o forma. Qualora Lei non fosse la persona destinataria del presente messaggio, La invitiamo a non diffonderlo e ad eliminarlo, dandone gentilmente comunicazione al mittente.
>
> The information included in this e-mail and any attachments are confidential and may also be privileged. If you are not the correct recipient, you are kindly requested to notify the sender immediately, to cancel it and not to disclose the contents to any other person.