You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Phil H <gi...@gmail.com> on 2020/09/18 01:49:05 UTC
Maximum load
Hi there,
Regardless of what I set in the maximum thread count settings, I never see
more than 128 active threads. The CPU load of the system is only 5-8% when
NiFi is running at this rate. It’s a Solaris system with 64 cores.
Any help getting more performance would be appreciated!
Regards,
Phil
Re: Maximum load
Posted by Edward Armes <ed...@gmail.com>.
Hi Phil,
A few things that might be of help here.
It's been a while since I looked at the call chain for a processor
but....... I believe what happens is that when a processor is set to have
a number of concurrent processors set what happens is that per nifi node
the locks around the onTrigger method is set equal to that number, thus
this doesn't create any additional instances of a processor.
The implication of this being that if you have any instance variables in
your custom processor that are synchronized or protected by locks in some
way then this may cause a backlog. Now if you have any libraries that use a
singleton pattern or has resources that may also have instance locks (or
syncronised) internally this may also cause a backlog while each of the
instances race and block to get hold of that lock.
The other thing maybe is that, it could be a setting in either in the JVM,
or unix limit subsystem that may be blocking nifi from using more than 10%
of the CPU.e
Edward
On Mon, Sep 21, 2020 at 1:06 PM Mike Thomsen <mi...@gmail.com> wrote:
> Phil,
>
> > Basically I have two custom processors that are slow (data intensive)
> that create backlogs
>
> Perhaps I missed it, but you didn't say whether the processors are
> functionally I/O-bound or CPU-bound. Also, the nature of the output is
> important. How many flowfiles are you creating, what are their average
> sizes, etc.
>
> Thanks,
>
> Mike
>
> On Sun, Sep 20, 2020 at 11:02 PM Phil H <gi...@gmail.com> wrote:
> >
> > Hi there,
> >
> > The issue I have isn’t my processor, it’s the fact I can’t get Nifi to
> run
> > more threads. I’ve noticed that if I bump the max thread counts in the
> > controller settings to total over 256, then nifi as a whole locks up and
> I
> > have to clear the state and restart it.
> >
> > Thanks,
> > Phil
> >
> > On Mon, 21 Sep 2020 at 11:01, Mark Payne <ma...@hotmail.com> wrote:
> >
> > > Phil,
> > >
> > >
> > >
> > > It sounds like you’re dealing with a large amount of lock contention.
> > > Either using explicit java.util.concurrency.Lock or synchronized
> blocks.
> > > Would review the code for lock contention or run a profiler against
> nifi
> > > and see what that shows.
> > >
> > >
> > >
> > > Sent from my iPhone
> > >
> > >
> > >
> > > > On Sep 20, 2020, at 8:30 PM, Phil H <gi...@gmail.com> wrote:
> > >
> > > >
> > >
> > > > Various numbers - max of 96 so far.
> > >
> > > >
> > >
> > > >> On Sat, 19 Sep 2020 at 21:44, Pierre Villard <
> > > pierre.villard.fr@gmail.com>
> > >
> > > >> wrote:
> > >
> > > >>
> > >
> > > >> How many concurrent tasks did you set on the two custom processors?
> > >
> > > >>
> > >
> > > >>
> > >
> > > >>
> > >
> > > >> Le sam. 19 sept. 2020 à 02:16, Phil H <gi...@gmail.com> a
> écrit :
> > >
> > > >>
> > >
> > > >>
> > >
> > > >>
> > >
> > > >>> Basically I have two custom processors that are slow (data
> intensive)
> > >
> > > >> that
> > >
> > > >>
> > >
> > > >>> create backlogs. All other aspects of the flow work well - I am
> running
> > >
> > > >> SSD
> > >
> > > >>
> > >
> > > >>> disks so that sort of I/O is not biting me. The process itself is
> > > highly
> > >
> > > >>
> > >
> > > >>> ‘parallelisable’ but I can’t get NiFi to use more than about 10%
> of the
> > >
> > > >> CPU
> > >
> > > >>
> > >
> > > >>> load. This is the only purpose this machine has - I want it to work
> > > much
> > >
> > > >>
> > >
> > > >>> harder!
> > >
> > > >>
> > >
> > > >>>
> > >
> > > >>
> > >
> > > >>> On Fri, 18 Sep 2020 at 20:56, Pierre Villard <
> > >
> > > >> pierre.villard.fr@gmail.com>
> > >
> > > >>
> > >
> > > >>> wrote:
> > >
> > > >>
> > >
> > > >>>
> > >
> > > >>
> > >
> > > >>>> Hi,
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>> Do you have backpressure with flow files accumulating somewhere in
> > > your
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>> flow? If not, then you probably don't need more threads.
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>> If you do find bottlenecks in your flow design, then you can look
> at
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>> increasing the concurrent tasks on specific processors to have
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>> multi-threading. Note that it's not always the solution.
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>> Changing the pool size does not mean NiFi will use more resources.
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>> Actually, I usually recommend changing the pool size only when
> you see
> > >
> > > >>
> > >
> > > >>> that
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>> all the threads available are used.
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>> Overall we'd need more details about your flow and observations.
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>> Thanks,
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>> Pierre
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>> Le ven. 18 sept. 2020 à 03:49, Phil H <gi...@gmail.com> a
> écrit :
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>> Hi there,
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>>
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>> Regardless of what I set in the maximum thread count settings, I
> > >
> > > >> never
> > >
> > > >>
> > >
> > > >>>> see
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>> more than 128 active threads. The CPU load of the system is only
> 5-8%
> > >
> > > >>
> > >
> > > >>>> when
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>> NiFi is running at this rate. It’s a Solaris system with 64
> cores.
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>>
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>> Any help getting more performance would be appreciated!
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>>
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>> Regards,
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>> Phil
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>>
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>>
> > >
> > > >>
> > >
> > > >>>
> > >
> > > >>
> > >
> > > >>
> > >
> > >
>
Re: Maximum load
Posted by Mike Thomsen <mi...@gmail.com>.
Phil,
> Basically I have two custom processors that are slow (data intensive) that create backlogs
Perhaps I missed it, but you didn't say whether the processors are
functionally I/O-bound or CPU-bound. Also, the nature of the output is
important. How many flowfiles are you creating, what are their average
sizes, etc.
Thanks,
Mike
On Sun, Sep 20, 2020 at 11:02 PM Phil H <gi...@gmail.com> wrote:
>
> Hi there,
>
> The issue I have isn’t my processor, it’s the fact I can’t get Nifi to run
> more threads. I’ve noticed that if I bump the max thread counts in the
> controller settings to total over 256, then nifi as a whole locks up and I
> have to clear the state and restart it.
>
> Thanks,
> Phil
>
> On Mon, 21 Sep 2020 at 11:01, Mark Payne <ma...@hotmail.com> wrote:
>
> > Phil,
> >
> >
> >
> > It sounds like you’re dealing with a large amount of lock contention.
> > Either using explicit java.util.concurrency.Lock or synchronized blocks.
> > Would review the code for lock contention or run a profiler against nifi
> > and see what that shows.
> >
> >
> >
> > Sent from my iPhone
> >
> >
> >
> > > On Sep 20, 2020, at 8:30 PM, Phil H <gi...@gmail.com> wrote:
> >
> > >
> >
> > > Various numbers - max of 96 so far.
> >
> > >
> >
> > >> On Sat, 19 Sep 2020 at 21:44, Pierre Villard <
> > pierre.villard.fr@gmail.com>
> >
> > >> wrote:
> >
> > >>
> >
> > >> How many concurrent tasks did you set on the two custom processors?
> >
> > >>
> >
> > >>
> >
> > >>
> >
> > >> Le sam. 19 sept. 2020 à 02:16, Phil H <gi...@gmail.com> a écrit :
> >
> > >>
> >
> > >>
> >
> > >>
> >
> > >>> Basically I have two custom processors that are slow (data intensive)
> >
> > >> that
> >
> > >>
> >
> > >>> create backlogs. All other aspects of the flow work well - I am running
> >
> > >> SSD
> >
> > >>
> >
> > >>> disks so that sort of I/O is not biting me. The process itself is
> > highly
> >
> > >>
> >
> > >>> ‘parallelisable’ but I can’t get NiFi to use more than about 10% of the
> >
> > >> CPU
> >
> > >>
> >
> > >>> load. This is the only purpose this machine has - I want it to work
> > much
> >
> > >>
> >
> > >>> harder!
> >
> > >>
> >
> > >>>
> >
> > >>
> >
> > >>> On Fri, 18 Sep 2020 at 20:56, Pierre Villard <
> >
> > >> pierre.villard.fr@gmail.com>
> >
> > >>
> >
> > >>> wrote:
> >
> > >>
> >
> > >>>
> >
> > >>
> >
> > >>>> Hi,
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>> Do you have backpressure with flow files accumulating somewhere in
> > your
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>> flow? If not, then you probably don't need more threads.
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>> If you do find bottlenecks in your flow design, then you can look at
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>> increasing the concurrent tasks on specific processors to have
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>> multi-threading. Note that it's not always the solution.
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>> Changing the pool size does not mean NiFi will use more resources.
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>> Actually, I usually recommend changing the pool size only when you see
> >
> > >>
> >
> > >>> that
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>> all the threads available are used.
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>> Overall we'd need more details about your flow and observations.
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>> Thanks,
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>> Pierre
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>> Le ven. 18 sept. 2020 à 03:49, Phil H <gi...@gmail.com> a écrit :
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>> Hi there,
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>>
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>> Regardless of what I set in the maximum thread count settings, I
> >
> > >> never
> >
> > >>
> >
> > >>>> see
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>> more than 128 active threads. The CPU load of the system is only 5-8%
> >
> > >>
> >
> > >>>> when
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>> NiFi is running at this rate. It’s a Solaris system with 64 cores.
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>>
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>> Any help getting more performance would be appreciated!
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>>
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>> Regards,
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>> Phil
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>>
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>>
> >
> > >>
> >
> > >>>
> >
> > >>
> >
> > >>
> >
> >
Re: Maximum load
Posted by Phil H <gi...@gmail.com>.
Hi there,
The issue I have isn’t my processor, it’s the fact I can’t get Nifi to run
more threads. I’ve noticed that if I bump the max thread counts in the
controller settings to total over 256, then nifi as a whole locks up and I
have to clear the state and restart it.
Thanks,
Phil
On Mon, 21 Sep 2020 at 11:01, Mark Payne <ma...@hotmail.com> wrote:
> Phil,
>
>
>
> It sounds like you’re dealing with a large amount of lock contention.
> Either using explicit java.util.concurrency.Lock or synchronized blocks.
> Would review the code for lock contention or run a profiler against nifi
> and see what that shows.
>
>
>
> Sent from my iPhone
>
>
>
> > On Sep 20, 2020, at 8:30 PM, Phil H <gi...@gmail.com> wrote:
>
> >
>
> > Various numbers - max of 96 so far.
>
> >
>
> >> On Sat, 19 Sep 2020 at 21:44, Pierre Villard <
> pierre.villard.fr@gmail.com>
>
> >> wrote:
>
> >>
>
> >> How many concurrent tasks did you set on the two custom processors?
>
> >>
>
> >>
>
> >>
>
> >> Le sam. 19 sept. 2020 à 02:16, Phil H <gi...@gmail.com> a écrit :
>
> >>
>
> >>
>
> >>
>
> >>> Basically I have two custom processors that are slow (data intensive)
>
> >> that
>
> >>
>
> >>> create backlogs. All other aspects of the flow work well - I am running
>
> >> SSD
>
> >>
>
> >>> disks so that sort of I/O is not biting me. The process itself is
> highly
>
> >>
>
> >>> ‘parallelisable’ but I can’t get NiFi to use more than about 10% of the
>
> >> CPU
>
> >>
>
> >>> load. This is the only purpose this machine has - I want it to work
> much
>
> >>
>
> >>> harder!
>
> >>
>
> >>>
>
> >>
>
> >>> On Fri, 18 Sep 2020 at 20:56, Pierre Villard <
>
> >> pierre.villard.fr@gmail.com>
>
> >>
>
> >>> wrote:
>
> >>
>
> >>>
>
> >>
>
> >>>> Hi,
>
> >>
>
> >>>>
>
> >>
>
> >>>>
>
> >>
>
> >>>>
>
> >>
>
> >>>> Do you have backpressure with flow files accumulating somewhere in
> your
>
> >>
>
> >>>>
>
> >>
>
> >>>> flow? If not, then you probably don't need more threads.
>
> >>
>
> >>>>
>
> >>
>
> >>>> If you do find bottlenecks in your flow design, then you can look at
>
> >>
>
> >>>>
>
> >>
>
> >>>> increasing the concurrent tasks on specific processors to have
>
> >>
>
> >>>>
>
> >>
>
> >>>> multi-threading. Note that it's not always the solution.
>
> >>
>
> >>>>
>
> >>
>
> >>>> Changing the pool size does not mean NiFi will use more resources.
>
> >>
>
> >>>>
>
> >>
>
> >>>> Actually, I usually recommend changing the pool size only when you see
>
> >>
>
> >>> that
>
> >>
>
> >>>>
>
> >>
>
> >>>> all the threads available are used.
>
> >>
>
> >>>>
>
> >>
>
> >>>> Overall we'd need more details about your flow and observations.
>
> >>
>
> >>>>
>
> >>
>
> >>>>
>
> >>
>
> >>>>
>
> >>
>
> >>>> Thanks,
>
> >>
>
> >>>>
>
> >>
>
> >>>> Pierre
>
> >>
>
> >>>>
>
> >>
>
> >>>>
>
> >>
>
> >>>>
>
> >>
>
> >>>> Le ven. 18 sept. 2020 à 03:49, Phil H <gi...@gmail.com> a écrit :
>
> >>
>
> >>>>
>
> >>
>
> >>>>
>
> >>
>
> >>>>
>
> >>
>
> >>>>> Hi there,
>
> >>
>
> >>>>
>
> >>
>
> >>>>>
>
> >>
>
> >>>>
>
> >>
>
> >>>>> Regardless of what I set in the maximum thread count settings, I
>
> >> never
>
> >>
>
> >>>> see
>
> >>
>
> >>>>
>
> >>
>
> >>>>> more than 128 active threads. The CPU load of the system is only 5-8%
>
> >>
>
> >>>> when
>
> >>
>
> >>>>
>
> >>
>
> >>>>> NiFi is running at this rate. It’s a Solaris system with 64 cores.
>
> >>
>
> >>>>
>
> >>
>
> >>>>>
>
> >>
>
> >>>>
>
> >>
>
> >>>>> Any help getting more performance would be appreciated!
>
> >>
>
> >>>>
>
> >>
>
> >>>>>
>
> >>
>
> >>>>
>
> >>
>
> >>>>> Regards,
>
> >>
>
> >>>>
>
> >>
>
> >>>>> Phil
>
> >>
>
> >>>>
>
> >>
>
> >>>>>
>
> >>
>
> >>>>
>
> >>
>
> >>>>
>
> >>
>
> >>>
>
> >>
>
> >>
>
>
Re: Maximum load
Posted by Mark Payne <ma...@hotmail.com>.
Phil,
It sounds like you’re dealing with a large amount of lock contention. Either using explicit java.util.concurrency.Lock or synchronized blocks. Would review the code for lock contention or run a profiler against nifi and see what that shows.
Sent from my iPhone
> On Sep 20, 2020, at 8:30 PM, Phil H <gi...@gmail.com> wrote:
>
> Various numbers - max of 96 so far.
>
>> On Sat, 19 Sep 2020 at 21:44, Pierre Villard <pi...@gmail.com>
>> wrote:
>>
>> How many concurrent tasks did you set on the two custom processors?
>>
>>
>>
>> Le sam. 19 sept. 2020 à 02:16, Phil H <gi...@gmail.com> a écrit :
>>
>>
>>
>>> Basically I have two custom processors that are slow (data intensive)
>> that
>>
>>> create backlogs. All other aspects of the flow work well - I am running
>> SSD
>>
>>> disks so that sort of I/O is not biting me. The process itself is highly
>>
>>> ‘parallelisable’ but I can’t get NiFi to use more than about 10% of the
>> CPU
>>
>>> load. This is the only purpose this machine has - I want it to work much
>>
>>> harder!
>>
>>>
>>
>>> On Fri, 18 Sep 2020 at 20:56, Pierre Villard <
>> pierre.villard.fr@gmail.com>
>>
>>> wrote:
>>
>>>
>>
>>>> Hi,
>>
>>>>
>>
>>>>
>>
>>>>
>>
>>>> Do you have backpressure with flow files accumulating somewhere in your
>>
>>>>
>>
>>>> flow? If not, then you probably don't need more threads.
>>
>>>>
>>
>>>> If you do find bottlenecks in your flow design, then you can look at
>>
>>>>
>>
>>>> increasing the concurrent tasks on specific processors to have
>>
>>>>
>>
>>>> multi-threading. Note that it's not always the solution.
>>
>>>>
>>
>>>> Changing the pool size does not mean NiFi will use more resources.
>>
>>>>
>>
>>>> Actually, I usually recommend changing the pool size only when you see
>>
>>> that
>>
>>>>
>>
>>>> all the threads available are used.
>>
>>>>
>>
>>>> Overall we'd need more details about your flow and observations.
>>
>>>>
>>
>>>>
>>
>>>>
>>
>>>> Thanks,
>>
>>>>
>>
>>>> Pierre
>>
>>>>
>>
>>>>
>>
>>>>
>>
>>>> Le ven. 18 sept. 2020 à 03:49, Phil H <gi...@gmail.com> a écrit :
>>
>>>>
>>
>>>>
>>
>>>>
>>
>>>>> Hi there,
>>
>>>>
>>
>>>>>
>>
>>>>
>>
>>>>> Regardless of what I set in the maximum thread count settings, I
>> never
>>
>>>> see
>>
>>>>
>>
>>>>> more than 128 active threads. The CPU load of the system is only 5-8%
>>
>>>> when
>>
>>>>
>>
>>>>> NiFi is running at this rate. It’s a Solaris system with 64 cores.
>>
>>>>
>>
>>>>>
>>
>>>>
>>
>>>>> Any help getting more performance would be appreciated!
>>
>>>>
>>
>>>>>
>>
>>>>
>>
>>>>> Regards,
>>
>>>>
>>
>>>>> Phil
>>
>>>>
>>
>>>>>
>>
>>>>
>>
>>>>
>>
>>>
>>
>>
Re: Maximum load
Posted by Phil H <gi...@gmail.com>.
Various numbers - max of 96 so far.
On Sat, 19 Sep 2020 at 21:44, Pierre Villard <pi...@gmail.com>
wrote:
> How many concurrent tasks did you set on the two custom processors?
>
>
>
> Le sam. 19 sept. 2020 à 02:16, Phil H <gi...@gmail.com> a écrit :
>
>
>
> > Basically I have two custom processors that are slow (data intensive)
> that
>
> > create backlogs. All other aspects of the flow work well - I am running
> SSD
>
> > disks so that sort of I/O is not biting me. The process itself is highly
>
> > ‘parallelisable’ but I can’t get NiFi to use more than about 10% of the
> CPU
>
> > load. This is the only purpose this machine has - I want it to work much
>
> > harder!
>
> >
>
> > On Fri, 18 Sep 2020 at 20:56, Pierre Villard <
> pierre.villard.fr@gmail.com>
>
> > wrote:
>
> >
>
> > > Hi,
>
> > >
>
> > >
>
> > >
>
> > > Do you have backpressure with flow files accumulating somewhere in your
>
> > >
>
> > > flow? If not, then you probably don't need more threads.
>
> > >
>
> > > If you do find bottlenecks in your flow design, then you can look at
>
> > >
>
> > > increasing the concurrent tasks on specific processors to have
>
> > >
>
> > > multi-threading. Note that it's not always the solution.
>
> > >
>
> > > Changing the pool size does not mean NiFi will use more resources.
>
> > >
>
> > > Actually, I usually recommend changing the pool size only when you see
>
> > that
>
> > >
>
> > > all the threads available are used.
>
> > >
>
> > > Overall we'd need more details about your flow and observations.
>
> > >
>
> > >
>
> > >
>
> > > Thanks,
>
> > >
>
> > > Pierre
>
> > >
>
> > >
>
> > >
>
> > > Le ven. 18 sept. 2020 à 03:49, Phil H <gi...@gmail.com> a écrit :
>
> > >
>
> > >
>
> > >
>
> > > > Hi there,
>
> > >
>
> > > >
>
> > >
>
> > > > Regardless of what I set in the maximum thread count settings, I
> never
>
> > > see
>
> > >
>
> > > > more than 128 active threads. The CPU load of the system is only 5-8%
>
> > > when
>
> > >
>
> > > > NiFi is running at this rate. It’s a Solaris system with 64 cores.
>
> > >
>
> > > >
>
> > >
>
> > > > Any help getting more performance would be appreciated!
>
> > >
>
> > > >
>
> > >
>
> > > > Regards,
>
> > >
>
> > > > Phil
>
> > >
>
> > > >
>
> > >
>
> > >
>
> >
>
>
Re: Maximum load
Posted by Pierre Villard <pi...@gmail.com>.
How many concurrent tasks did you set on the two custom processors?
Le sam. 19 sept. 2020 à 02:16, Phil H <gi...@gmail.com> a écrit :
> Basically I have two custom processors that are slow (data intensive) that
> create backlogs. All other aspects of the flow work well - I am running SSD
> disks so that sort of I/O is not biting me. The process itself is highly
> ‘parallelisable’ but I can’t get NiFi to use more than about 10% of the CPU
> load. This is the only purpose this machine has - I want it to work much
> harder!
>
> On Fri, 18 Sep 2020 at 20:56, Pierre Villard <pi...@gmail.com>
> wrote:
>
> > Hi,
> >
> >
> >
> > Do you have backpressure with flow files accumulating somewhere in your
> >
> > flow? If not, then you probably don't need more threads.
> >
> > If you do find bottlenecks in your flow design, then you can look at
> >
> > increasing the concurrent tasks on specific processors to have
> >
> > multi-threading. Note that it's not always the solution.
> >
> > Changing the pool size does not mean NiFi will use more resources.
> >
> > Actually, I usually recommend changing the pool size only when you see
> that
> >
> > all the threads available are used.
> >
> > Overall we'd need more details about your flow and observations.
> >
> >
> >
> > Thanks,
> >
> > Pierre
> >
> >
> >
> > Le ven. 18 sept. 2020 à 03:49, Phil H <gi...@gmail.com> a écrit :
> >
> >
> >
> > > Hi there,
> >
> > >
> >
> > > Regardless of what I set in the maximum thread count settings, I never
> > see
> >
> > > more than 128 active threads. The CPU load of the system is only 5-8%
> > when
> >
> > > NiFi is running at this rate. It’s a Solaris system with 64 cores.
> >
> > >
> >
> > > Any help getting more performance would be appreciated!
> >
> > >
> >
> > > Regards,
> >
> > > Phil
> >
> > >
> >
> >
>
Re: Maximum load
Posted by Phil H <gi...@gmail.com>.
Basically I have two custom processors that are slow (data intensive) that
create backlogs. All other aspects of the flow work well - I am running SSD
disks so that sort of I/O is not biting me. The process itself is highly
‘parallelisable’ but I can’t get NiFi to use more than about 10% of the CPU
load. This is the only purpose this machine has - I want it to work much
harder!
On Fri, 18 Sep 2020 at 20:56, Pierre Villard <pi...@gmail.com>
wrote:
> Hi,
>
>
>
> Do you have backpressure with flow files accumulating somewhere in your
>
> flow? If not, then you probably don't need more threads.
>
> If you do find bottlenecks in your flow design, then you can look at
>
> increasing the concurrent tasks on specific processors to have
>
> multi-threading. Note that it's not always the solution.
>
> Changing the pool size does not mean NiFi will use more resources.
>
> Actually, I usually recommend changing the pool size only when you see that
>
> all the threads available are used.
>
> Overall we'd need more details about your flow and observations.
>
>
>
> Thanks,
>
> Pierre
>
>
>
> Le ven. 18 sept. 2020 à 03:49, Phil H <gi...@gmail.com> a écrit :
>
>
>
> > Hi there,
>
> >
>
> > Regardless of what I set in the maximum thread count settings, I never
> see
>
> > more than 128 active threads. The CPU load of the system is only 5-8%
> when
>
> > NiFi is running at this rate. It’s a Solaris system with 64 cores.
>
> >
>
> > Any help getting more performance would be appreciated!
>
> >
>
> > Regards,
>
> > Phil
>
> >
>
>
Re: Maximum load
Posted by Pierre Villard <pi...@gmail.com>.
Hi,
Do you have backpressure with flow files accumulating somewhere in your
flow? If not, then you probably don't need more threads.
If you do find bottlenecks in your flow design, then you can look at
increasing the concurrent tasks on specific processors to have
multi-threading. Note that it's not always the solution.
Changing the pool size does not mean NiFi will use more resources.
Actually, I usually recommend changing the pool size only when you see that
all the threads available are used.
Overall we'd need more details about your flow and observations.
Thanks,
Pierre
Le ven. 18 sept. 2020 à 03:49, Phil H <gi...@gmail.com> a écrit :
> Hi there,
>
> Regardless of what I set in the maximum thread count settings, I never see
> more than 128 active threads. The CPU load of the system is only 5-8% when
> NiFi is running at this rate. It’s a Solaris system with 64 cores.
>
> Any help getting more performance would be appreciated!
>
> Regards,
> Phil
>