You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn> on 2019/09/18 12:04:51 UTC
NiFi active thread count is no more than 10 ?
My NiFi server is 16 cores. I also configed some processor cocurrent tasks to 100.
But why the active thread count shown on the NiFi web ui is no more than 10?
wanglei2@geekplus.com.cn
Re: Re: NiFi active thread count is no more than 10 ?
Posted by "wanglei2@geekplus.com.cn" <wa...@geekplus.com.cn>.
Thanks very much Bryan, chaning the overall timer-driven thread pool value works.
wanglei2@geekplus.com.cn
From: Bryan Bende
Date: 2019-09-18 20:53
To: users
Subject: Re: NiFi active thread count is no more than 10 ?
The overall timer-driven thread pool defaults to 10 (configured from
the controller settings in top right menu).
So even if a processor has 100 concurrent tasks, there are still only
10 threads available.
On Wed, Sep 18, 2019 at 8:20 AM Joe Witt <jo...@gmail.com> wrote:
>
> Hello
>
> The 100 threads for the controller overall is the maximum number of threads that could run concurrently. On a 16 core system and a flow which is very I/O bound this is definitely achievable. Generally you want to look at some multiple of the number of physical cores such as 2,4,8, etc.. but in the end it isn't that important.
>
> So why do you only see at most 10 or so threads in active use? Generally this means your flow isn't demanding or configured to do more.
>
> How many processors do you have? How many tasks does each have? How much data is flowing through the system when it reaches 10? Are backlogs growing at that time? What are the run schedules? How does the load average on the system look? How does IO utilization/iowait look during those times?
>
> Thanks
> Joe
>
> On Wed, Sep 18, 2019 at 8:05 AM wanglei2@geekplus.com.cn <wa...@geekplus.com.cn> wrote:
>>
>>
>> My NiFi server is 16 cores. I also configed some processor cocurrent tasks to 100.
>> But why the active thread count shown on the NiFi web ui is no more than 10?
>>
>>
>> ________________________________
>> wanglei2@geekplus.com.cn
Re: NiFi active thread count is no more than 10 ?
Posted by Bryan Bende <bb...@gmail.com>.
The overall timer-driven thread pool defaults to 10 (configured from
the controller settings in top right menu).
So even if a processor has 100 concurrent tasks, there are still only
10 threads available.
On Wed, Sep 18, 2019 at 8:20 AM Joe Witt <jo...@gmail.com> wrote:
>
> Hello
>
> The 100 threads for the controller overall is the maximum number of threads that could run concurrently. On a 16 core system and a flow which is very I/O bound this is definitely achievable. Generally you want to look at some multiple of the number of physical cores such as 2,4,8, etc.. but in the end it isn't that important.
>
> So why do you only see at most 10 or so threads in active use? Generally this means your flow isn't demanding or configured to do more.
>
> How many processors do you have? How many tasks does each have? How much data is flowing through the system when it reaches 10? Are backlogs growing at that time? What are the run schedules? How does the load average on the system look? How does IO utilization/iowait look during those times?
>
> Thanks
> Joe
>
> On Wed, Sep 18, 2019 at 8:05 AM wanglei2@geekplus.com.cn <wa...@geekplus.com.cn> wrote:
>>
>>
>> My NiFi server is 16 cores. I also configed some processor cocurrent tasks to 100.
>> But why the active thread count shown on the NiFi web ui is no more than 10?
>>
>>
>> ________________________________
>> wanglei2@geekplus.com.cn
Re: NiFi active thread count is no more than 10 ?
Posted by Joe Witt <jo...@gmail.com>.
Hello
The 100 threads for the controller overall is the maximum number of threads
that could run concurrently. On a 16 core system and a flow which is very
I/O bound this is definitely achievable. Generally you want to look at
some multiple of the number of physical cores such as 2,4,8, etc.. but in
the end it isn't that important.
So why do you only see at most 10 or so threads in active use? Generally
this means your flow isn't demanding or configured to do more.
How many processors do you have? How many tasks does each have? How much
data is flowing through the system when it reaches 10? Are backlogs
growing at that time? What are the run schedules? How does the load
average on the system look? How does IO utilization/iowait look during
those times?
Thanks
Joe
On Wed, Sep 18, 2019 at 8:05 AM wanglei2@geekplus.com.cn <
wanglei2@geekplus.com.cn> wrote:
>
> My NiFi server is 16 cores. I also configed some processor cocurrent tasks
> to 100.
> But why the active thread count shown on the NiFi web ui is no more than
> 10?
>
>
> ------------------------------
> wanglei2@geekplus.com.cn
>