Re: Make invokehttp to process faster to process more than 20k records.

[Midhun Mohan <mi...@esginc.us>]

No overall thread pool I haven't changed. How can I change that?

 Nifi is running on a standalone aws instance. T2.medium I remember so that
leaves us with 2 cores I guess


On Wed, 18 Mar 2020, 1:29 am Pierre Villard, <pi...@gmail.com>
wrote:

> Did you change the size of the overall thread pool at NiFi level? If yes,
> to which number? How many cores do you have on NiFi?
>
> Thanks
>
> Le mar. 17 mars 2020 à 20:55, Midhun Mohan <mi...@esginc.us> a
> écrit :
>
> > Yeah the endpoint which am sending right now has plenty of resources
> > available. Only thing is I need to send more records
> >
> > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <cz...@gmail.com> wrote:
> >
> > > I have not test throughput of InvokeHTTP so am not sure what the
> maximum
> > > is, but can give some general guidance.
> > >
> > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure out
> > system
> > > resources usage while your flow is running.
> > >
> > > You should be able to increase both your nifi count by more and adjust
> > your
> > > processors until you are limited by your system resources.
> > >
> > > Have you verified the endpoint you are sending to is not the bottle
> neck?
> > >
> > > Chad
> > >
> > >
> > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > > > Size of file around kb not more than that,
> > > > How can I find the bottle neck,
> > > >
> > > > Yes I adjust the count to 11 then all other processors stopped but
> > > > processing improved.
> > > > Totally at present 6 is showing not more than that.
> > > >
> > > > I just need a way to post more records to endpoint to make it
> realtime
> > > >
> > > > Hope i was able to give more details
> > > >
> > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <cz...@gmail.com>
> > wrote:
> > > >
> > > > > Midhun,
> > > > > A little more information would help.
> > > > >
> > > > > What size files are you sending?
> > > > > Have you looked at resource usage to see what the bottleneck is?
> > > > > Did you adjust your nifi system thread count from the hamburger
> menu?
> > > > > How many threads are running total for nifi? It's the number in the
> > top
> > > > > left of the screen.
> > > > >
> > > > > Thanks,
> > > > > Chad
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > >
> > > > > > Do anyone have better idea on this
> > > > > >
> > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I have a queue which will be filling up with realtime records
> of
> > > > around
> > > > > > > 20k records. Which is posting to an endpoint using Invokehttp
> > > > > processor.
> > > > > > >
> > > > > > > currently the average throughput is for 10k records it takes
> > around
> > > > 20
> > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > I  Increased the concurrent thread to larger number which is
> > > > increasing
> > > > > > > the throughput, but it is causing other processor to
> terminate. I
> > > > think
> > > > > > > that can be solved by running on larger instances.
> > > > > > >
> > > > > > > What all things I need to take care to make sure I get a  real
> > time
> > > > > > > posting on endpoint.
> > > > > > >
> > > > > > > I need 20k records to be processed as they receive it
> > > > > > >
> > > > > > >
> > > > > > > thanks in advance
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > > Midhun Mohan
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>