You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Akash Kumar Rai <ak...@gmail.com> on 2025/05/09 01:49:47 UTC
Queue getting shuffled when processed by nifi processors.
Hello,
In Nifi i have added priority to all the flowfiles, i can see it in
queue that the flowfiles are in sequence but when it is getting
processed it gets shuffled. I have added
1. PriorityAttributePrioritizer
2. FirstInFirstOut .
I can see the files in sequence in queues. but in data provenance of
processors somewhere it is in sequence somewhere it shuffled.
Sometimes it is in sequence everywhere and sometimes it get shuffled.
I am using Nifi version 1.22
Please let me know what else can i check in configuration, same
template works in another environment.
Regards,
Akash
Re: Queue getting shuffled when processed by nifi processors.
Posted by Mark Payne <ma...@hotmail.com>.
Hi Akash,
A couple of important considerations. In general, prioritizers are just that - they assign priority to the data processing. This is not a strict guarantee of ordering. If data is swapped to disk (a queue exceeds some configurable value that defaults to 20,000 FlowFiles), for example. FirstInFirstOut is special in that even if data is swapped out, we guarantee that it still applies. But others do not provide that guarantee.
You also have to consider Concurrent Tasks. If a processor has multiple concurrent tasks, it’s working data in parallel so ordering may change.
Thanks
-Mark
> On May 8, 2025, at 9:49 PM, Akash Kumar Rai <ak...@gmail.com> wrote:
>
> Hello,
> In Nifi i have added priority to all the flowfiles, i can see it in
> queue that the flowfiles are in sequence but when it is getting
> processed it gets shuffled. I have added
> 1. PriorityAttributePrioritizer
> 2. FirstInFirstOut .
>
> I can see the files in sequence in queues. but in data provenance of
> processors somewhere it is in sequence somewhere it shuffled.
>
> Sometimes it is in sequence everywhere and sometimes it get shuffled.
> I am using Nifi version 1.22
> Please let me know what else can i check in configuration, same
> template works in another environment.
>
> Regards,
> Akash