You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Phil H <gi...@gmail.com> on 2018/10/24 00:26:18 UTC
Load issues
Hi team,
My NiFi is struggling to process data (lots of back pressure in various
queues) but it is only using a very small amount (2-5%) of CPU according to
top.
Any ideas?
Re: Load issues
Posted by Charlie Meyer <ch...@civitaslearning.com>.
Also, we've seen flows automatically throttled if provenance (or other
repositories) can't keep up. Tuning the configs and switching drives (gp2
-> io1) helped
On Tue, Oct 23, 2018, 19:38 Milan Das <md...@interset.com> wrote:
> Did you increase the no. of Concurrent Tasks and Run Duration on slow
> processors ?
>
> Thanks,
> Milan Das
>
> On 10/23/18, 8:26 PM, "Phil H" <gi...@gmail.com> wrote:
>
> Hi team,
>
> My NiFi is struggling to process data (lots of back pressure in various
> queues) but it is only using a very small amount (2-5%) of CPU
> according to
> top.
>
> Any ideas?
>
>
>
>
Re: Load issues
Posted by Phil H <gi...@gmail.com>.
It's a single node. Neither the data rate or machine spec has changed
though. It's just running a lot slower now. With no errors being reported
On Wed, 24 Oct 2018 at 13:18, Bryan Bende <bb...@gmail.com> wrote:
> You’ll probably need to provide some more information like hardware specs
> (cpus/cores, memory, disks), number of nodes in the cluster, JVM heap size,
> configuration of repositories, specific processors that are bottle necks,
> etc.
>
> On Tue, Oct 23, 2018 at 10:02 PM Phil H <gi...@gmail.com> wrote:
>
> > Yep - although it now has less throughput with 8 threads than it
> > previously had with one.
> >
> > I have seen this happen when you have a lot of errors going into the
> logs,
> > but that doesn’t appear to be the case this time around
> >
> > Run Schedule is set to 0 sec
> >
> > > On 24 Oct 2018, at 11:38 am, Milan Das <md...@interset.com> wrote:
> > >
> > > Did you increase the no. of Concurrent Tasks and Run Duration on slow
> > processors ?
> > >
> > > Thanks,
> > > Milan Das
> > >
> > > On 10/23/18, 8:26 PM, "Phil H" <gi...@gmail.com> wrote:
> > >
> > > Hi team,
> > >
> > > My NiFi is struggling to process data (lots of back pressure in
> > various
> > > queues) but it is only using a very small amount (2-5%) of CPU
> > according to
> > > top.
> > >
> > > Any ideas?
> > >
> > >
> > >
> >
> > --
> Sent from Gmail Mobile
>
Re: Load issues
Posted by Bryan Bende <bb...@gmail.com>.
You’ll probably need to provide some more information like hardware specs
(cpus/cores, memory, disks), number of nodes in the cluster, JVM heap size,
configuration of repositories, specific processors that are bottle necks,
etc.
On Tue, Oct 23, 2018 at 10:02 PM Phil H <gi...@gmail.com> wrote:
> Yep - although it now has less throughput with 8 threads than it
> previously had with one.
>
> I have seen this happen when you have a lot of errors going into the logs,
> but that doesn’t appear to be the case this time around
>
> Run Schedule is set to 0 sec
>
> > On 24 Oct 2018, at 11:38 am, Milan Das <md...@interset.com> wrote:
> >
> > Did you increase the no. of Concurrent Tasks and Run Duration on slow
> processors ?
> >
> > Thanks,
> > Milan Das
> >
> > On 10/23/18, 8:26 PM, "Phil H" <gi...@gmail.com> wrote:
> >
> > Hi team,
> >
> > My NiFi is struggling to process data (lots of back pressure in
> various
> > queues) but it is only using a very small amount (2-5%) of CPU
> according to
> > top.
> >
> > Any ideas?
> >
> >
> >
>
> --
Sent from Gmail Mobile
Re: Load issues
Posted by Phil H <gi...@gmail.com>.
Yep - although it now has less throughput with 8 threads than it previously had with one.
I have seen this happen when you have a lot of errors going into the logs, but that doesn’t appear to be the case this time around
Run Schedule is set to 0 sec
> On 24 Oct 2018, at 11:38 am, Milan Das <md...@interset.com> wrote:
>
> Did you increase the no. of Concurrent Tasks and Run Duration on slow processors ?
>
> Thanks,
> Milan Das
>
> On 10/23/18, 8:26 PM, "Phil H" <gi...@gmail.com> wrote:
>
> Hi team,
>
> My NiFi is struggling to process data (lots of back pressure in various
> queues) but it is only using a very small amount (2-5%) of CPU according to
> top.
>
> Any ideas?
>
>
>
Re: Load issues
Posted by Milan Das <md...@interset.com>.
Did you increase the no. of Concurrent Tasks and Run Duration on slow processors ?
Thanks,
Milan Das
On 10/23/18, 8:26 PM, "Phil H" <gi...@gmail.com> wrote:
Hi team,
My NiFi is struggling to process data (lots of back pressure in various
queues) but it is only using a very small amount (2-5%) of CPU according to
top.
Any ideas?