Back pressure in a NiFi cluster

[Alexis Sarda-Espinosa <sa...@gmail.com>]

Hello,

If I have a NiFi cluster and a processor that is scheduled to run on all
nodes, a queue for said processor basically represents a queue per node,
right? And since the configured back pressure thresholds are also per node,
it could be that 1 node is slower and reaches the threshold while the
others run normally. If the processor is only back pressured in 1 node,
does that also halt scheduling for the same processor in the other nodes?

Regards,
Alexis.

RE: Back pressure in a NiFi cluster

["stephen.hindmarch.bt.com via users" <us...@nifi.apache.org>]

Alexis,

I see this often in some of the clusters I maintain. The restricted scheduling only applies to the node where the back pressure is occurring. The red and amber colours on the UI are there as a clue to the user that back pressure is occurring.

You can test this for example by using the queue history to see that the back pressured node has a high queue count and a low throughput, whereas the other nodes will be the opposite.

Regards

Steve Hindmarch

From: Alexis Sarda-Espinosa <sa...@gmail.com>
Sent: Tuesday, June 4, 2024 1:49 PM
To: users@nifi.apache.org
Subject: Back pressure in a NiFi cluster

You don't often get email from sarda.espinosa@gmail.com<ma...@gmail.com>. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification>
Hello,

If I have a NiFi cluster and a processor that is scheduled to run on all nodes, a queue for said processor basically represents a queue per node, right? And since the configured back pressure thresholds are also per node, it could be that 1 node is slower and reaches the threshold while the others run normally. If the processor is only back pressured in 1 node, does that also halt scheduling for the same processor in the other nodes?

Regards,
Alexis.

Re: Back pressure in a NiFi cluster

[Alexis Sarda-Espinosa <sa...@gmail.com>]

Thanks for the confirmation!

Am Di., 4. Juni 2024 um 14:57 Uhr schrieb Pierre Villard <
pierre.villard.fr@gmail.com>:

> Hi Alexis,
>
> Backpressure is a node-level concept. If there is backpressure on one
> node, it does not impact other nodes. So the statement "it could be that 1
> node is slower and reaches the threshold while the others run normally" is
> correct.
>
> HTH,
> Pierre
>
> Le mar. 4 juin 2024 à 14:54, Alexis Sarda-Espinosa <
> sarda.espinosa@gmail.com> a écrit :
>
>> Hello,
>>
>> If I have a NiFi cluster and a processor that is scheduled to run on all
>> nodes, a queue for said processor basically represents a queue per node,
>> right? And since the configured back pressure thresholds are also per node,
>> it could be that 1 node is slower and reaches the threshold while the
>> others run normally. If the processor is only back pressured in 1 node,
>> does that also halt scheduling for the same processor in the other nodes?
>>
>> Regards,
>> Alexis.
>>
>>

Re: Back pressure in a NiFi cluster

[Pierre Villard <pi...@gmail.com>]

Hi Alexis,

Backpressure is a node-level concept. If there is backpressure on one node,
it does not impact other nodes. So the statement "it could be that 1 node
is slower and reaches the threshold while the others run normally" is
correct.

HTH,
Pierre

Le mar. 4 juin 2024 à 14:54, Alexis Sarda-Espinosa <sa...@gmail.com>
a écrit :

> Hello,
>
> If I have a NiFi cluster and a processor that is scheduled to run on all
> nodes, a queue for said processor basically represents a queue per node,
> right? And since the configured back pressure thresholds are also per node,
> it could be that 1 node is slower and reaches the threshold while the
> others run normally. If the processor is only back pressured in 1 node,
> does that also halt scheduling for the same processor in the other nodes?
>
> Regards,
> Alexis.
>
>