Acceptable number of processors

[Максим Римар <ri...@gmail.com>]

Hi everyone!

I have a question regarding the number of processors on the NiFi canvas.

Could you please share how many processors are considered acceptable within
a single NiFi instance? Can an excessive number of active (but idle) or
disabled processors negatively affect the processing time of flows that are
actively processing data?

I suppose that after a certain number of processors, it may also become
difficult to manage them effectively — regardless of whether they are idle
or not. Maybe there’s a practical limit, after which it’s better to split
different flows across separate NiFi clusters?

Best regards,
Maksym Rymar

Re: Acceptable number of processors

[Matt Burgess <ma...@apache.org>]

Another thing you can do with idle / unused process groups is to version
control them in NiFi Registry then remove them from the canvas. You can
always re-import them later.

On Fri, Nov 7, 2025 at 4:30 PM Phil Lord <ph...@gmail.com> wrote:

> Maksym,
>
> I echo Joe's comments as well...
>
> The only thing I'd add is that if you have a lot of idle processors it
> "could" impact performance.  That's why I always encourage disabling flows
> if they're not planned to be active at any point in time.  If you have a
> large flow that needs to stay turned on, but is largely idle.(example... a
> few files a day/etc) you can try increasing the "nifi.bored.yield.duration"
> property in the nifi.properties file.  This essentially tells idle
> processors how often to check for work when it has no data to process.
> I've seen small bumps in cpu consumption from modifying this setting
> before...
>
> On Fri, Nov 7, 2025 at 11:49 AM Максим Римар <ri...@gmail.com> wrote:
>
>> Thank you for sharing your experience! I’d appreciate it if anyone else
>> would like to share their experience as well.
>>
>> It’s really insightful for me to hear about nodes with over 30+ thousand
>> processors.
>>
>> пт, 7 лист. 2025 р., 18:22 користувач Joe Witt <jo...@gmail.com> пише:
>>
>>> Hello Maksym,
>>>
>>> I've seen enterprise level nifi nodes or clusters with flows having as
>>> few as two processors.  And I've seen 30+ thousand.
>>>
>>> Do idle processors matter?  In theory they could at extreme-ish levels
>>> on constrained machines in terms of CPU.  But in practice no.
>>>
>>> What really gets tricky after a while is having lots of flows on the
>>> same cluster at once who are all resource intensive and have strong SLAs.
>>> Here disk, cpu, and even network become a bottleneck and it does get
>>> admittedly hard to reason over the behavior then.   For this reason many of
>>> us have moved into making it as easy as possible for people to build lots
>>> of nifi clusters and manage them easily.  This has been extremely powerful
>>> because it addresses the resource isolation problem nicely.
>>> \
>>> Thanks
>>>
>>> On Fri, Nov 7, 2025 at 9:17 AM Максим Римар <ri...@gmail.com> wrote:
>>>
>>>> Hi everyone!
>>>>
>>>> I have a question regarding the number of processors on the NiFi canvas.
>>>>
>>>> Could you please share how many processors are considered acceptable
>>>> within a single NiFi instance? Can an excessive number of active (but idle)
>>>> or disabled processors negatively affect the processing time of flows that
>>>> are actively processing data?
>>>>
>>>> I suppose that after a certain number of processors, it may also become
>>>> difficult to manage them effectively — regardless of whether they are idle
>>>> or not. Maybe there’s a practical limit, after which it’s better to split
>>>> different flows across separate NiFi clusters?
>>>>
>>>> Best regards,
>>>> Maksym Rymar
>>>>
>>>

Re: Acceptable number of processors

[Phil Lord <ph...@gmail.com>]

Maksym,

I echo Joe's comments as well...

The only thing I'd add is that if you have a lot of idle processors it
"could" impact performance.  That's why I always encourage disabling flows
if they're not planned to be active at any point in time.  If you have a
large flow that needs to stay turned on, but is largely idle.(example... a
few files a day/etc) you can try increasing the "nifi.bored.yield.duration"
property in the nifi.properties file.  This essentially tells idle
processors how often to check for work when it has no data to process.
I've seen small bumps in cpu consumption from modifying this setting
before...

On Fri, Nov 7, 2025 at 11:49 AM Максим Римар <ri...@gmail.com> wrote:

> Thank you for sharing your experience! I’d appreciate it if anyone else
> would like to share their experience as well.
>
> It’s really insightful for me to hear about nodes with over 30+ thousand
> processors.
>
> пт, 7 лист. 2025 р., 18:22 користувач Joe Witt <jo...@gmail.com> пише:
>
>> Hello Maksym,
>>
>> I've seen enterprise level nifi nodes or clusters with flows having as
>> few as two processors.  And I've seen 30+ thousand.
>>
>> Do idle processors matter?  In theory they could at extreme-ish levels on
>> constrained machines in terms of CPU.  But in practice no.
>>
>> What really gets tricky after a while is having lots of flows on the same
>> cluster at once who are all resource intensive and have strong SLAs.  Here
>> disk, cpu, and even network become a bottleneck and it does get admittedly
>> hard to reason over the behavior then.   For this reason many of us have
>> moved into making it as easy as possible for people to build lots of nifi
>> clusters and manage them easily.  This has been extremely powerful because
>> it addresses the resource isolation problem nicely.
>> \
>> Thanks
>>
>> On Fri, Nov 7, 2025 at 9:17 AM Максим Римар <ri...@gmail.com> wrote:
>>
>>> Hi everyone!
>>>
>>> I have a question regarding the number of processors on the NiFi canvas.
>>>
>>> Could you please share how many processors are considered acceptable
>>> within a single NiFi instance? Can an excessive number of active (but idle)
>>> or disabled processors negatively affect the processing time of flows that
>>> are actively processing data?
>>>
>>> I suppose that after a certain number of processors, it may also become
>>> difficult to manage them effectively — regardless of whether they are idle
>>> or not. Maybe there’s a practical limit, after which it’s better to split
>>> different flows across separate NiFi clusters?
>>>
>>> Best regards,
>>> Maksym Rymar
>>>
>>

Re: Acceptable number of processors

[Максим Римар <ri...@gmail.com>]

Thank you for sharing your experience! I’d appreciate it if anyone else
would like to share their experience as well.

It’s really insightful for me to hear about nodes with over 30+ thousand
processors.

пт, 7 лист. 2025 р., 18:22 користувач Joe Witt <jo...@gmail.com> пише:

> Hello Maksym,
>
> I've seen enterprise level nifi nodes or clusters with flows having as few
> as two processors.  And I've seen 30+ thousand.
>
> Do idle processors matter?  In theory they could at extreme-ish levels on
> constrained machines in terms of CPU.  But in practice no.
>
> What really gets tricky after a while is having lots of flows on the same
> cluster at once who are all resource intensive and have strong SLAs.  Here
> disk, cpu, and even network become a bottleneck and it does get admittedly
> hard to reason over the behavior then.   For this reason many of us have
> moved into making it as easy as possible for people to build lots of nifi
> clusters and manage them easily.  This has been extremely powerful because
> it addresses the resource isolation problem nicely.
> \
> Thanks
>
> On Fri, Nov 7, 2025 at 9:17 AM Максим Римар <ri...@gmail.com> wrote:
>
>> Hi everyone!
>>
>> I have a question regarding the number of processors on the NiFi canvas.
>>
>> Could you please share how many processors are considered acceptable
>> within a single NiFi instance? Can an excessive number of active (but idle)
>> or disabled processors negatively affect the processing time of flows that
>> are actively processing data?
>>
>> I suppose that after a certain number of processors, it may also become
>> difficult to manage them effectively — regardless of whether they are idle
>> or not. Maybe there’s a practical limit, after which it’s better to split
>> different flows across separate NiFi clusters?
>>
>> Best regards,
>> Maksym Rymar
>>
>

Re: Acceptable number of processors

[Joe Witt <jo...@gmail.com>]

Hello Maksym,

I've seen enterprise level nifi nodes or clusters with flows having as few
as two processors.  And I've seen 30+ thousand.

Do idle processors matter?  In theory they could at extreme-ish levels on
constrained machines in terms of CPU.  But in practice no.

What really gets tricky after a while is having lots of flows on the same
cluster at once who are all resource intensive and have strong SLAs.  Here
disk, cpu, and even network become a bottleneck and it does get admittedly
hard to reason over the behavior then.   For this reason many of us have
moved into making it as easy as possible for people to build lots of nifi
clusters and manage them easily.  This has been extremely powerful because
it addresses the resource isolation problem nicely.
\
Thanks

On Fri, Nov 7, 2025 at 9:17 AM Максим Римар <ri...@gmail.com> wrote:

> Hi everyone!
>
> I have a question regarding the number of processors on the NiFi canvas.
>
> Could you please share how many processors are considered acceptable
> within a single NiFi instance? Can an excessive number of active (but idle)
> or disabled processors negatively affect the processing time of flows that
> are actively processing data?
>
> I suppose that after a certain number of processors, it may also become
> difficult to manage them effectively — regardless of whether they are idle
> or not. Maybe there’s a practical limit, after which it’s better to split
> different flows across separate NiFi clusters?
>
> Best regards,
> Maksym Rymar
>