You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Midhun Mohan <mi...@esginc.us> on 2020/03/17 13:02:30 UTC
Make invokehttp to process faster to process more than 20k records
Hi all,
I have a queue which will be filling up with realtime records of around 20k
records. Which is posting to an endpoint using Invokehttp processor.
currently the average throughput is for 10k records it takes around 20
minutes to complete the invokehttp processor posting.
I Increased the concurrent thread to larger number which is increasing the
throughput, but it is causing other processor to terminate. I think that
can be solved by running on larger instances.
What all things I need to take care to make sure I get a real time posting
on endpoint.
I need 20k records to be processed as they receive it
thanks in advance
--
Regards,
Midhun Mohan
Re: Make invokehttp to process faster to process more than 20k records.
Posted by Midhun Mohan <mi...@esginc.us>.
No overall thread pool I haven't changed. How can I change that?
Nifi is running on a standalone aws instance. T2.medium I remember so that
leaves us with 2 cores I guess
On Wed, 18 Mar 2020, 1:29 am Pierre Villard, <pi...@gmail.com>
wrote:
> Did you change the size of the overall thread pool at NiFi level? If yes,
> to which number? How many cores do you have on NiFi?
>
> Thanks
>
> Le mar. 17 mars 2020 à 20:55, Midhun Mohan <mi...@esginc.us> a
> écrit :
>
> > Yeah the endpoint which am sending right now has plenty of resources
> > available. Only thing is I need to send more records
> >
> > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <cz...@gmail.com> wrote:
> >
> > > I have not test throughput of InvokeHTTP so am not sure what the
> maximum
> > > is, but can give some general guidance.
> > >
> > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure out
> > system
> > > resources usage while your flow is running.
> > >
> > > You should be able to increase both your nifi count by more and adjust
> > your
> > > processors until you are limited by your system resources.
> > >
> > > Have you verified the endpoint you are sending to is not the bottle
> neck?
> > >
> > > Chad
> > >
> > >
> > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > > > Size of file around kb not more than that,
> > > > How can I find the bottle neck,
> > > >
> > > > Yes I adjust the count to 11 then all other processors stopped but
> > > > processing improved.
> > > > Totally at present 6 is showing not more than that.
> > > >
> > > > I just need a way to post more records to endpoint to make it
> realtime
> > > >
> > > > Hope i was able to give more details
> > > >
> > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <cz...@gmail.com>
> > wrote:
> > > >
> > > > > Midhun,
> > > > > A little more information would help.
> > > > >
> > > > > What size files are you sending?
> > > > > Have you looked at resource usage to see what the bottleneck is?
> > > > > Did you adjust your nifi system thread count from the hamburger
> menu?
> > > > > How many threads are running total for nifi? It's the number in the
> > top
> > > > > left of the screen.
> > > > >
> > > > > Thanks,
> > > > > Chad
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > >
> > > > > > Do anyone have better idea on this
> > > > > >
> > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I have a queue which will be filling up with realtime records
> of
> > > > around
> > > > > > > 20k records. Which is posting to an endpoint using Invokehttp
> > > > > processor.
> > > > > > >
> > > > > > > currently the average throughput is for 10k records it takes
> > around
> > > > 20
> > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > I Increased the concurrent thread to larger number which is
> > > > increasing
> > > > > > > the throughput, but it is causing other processor to
> terminate. I
> > > > think
> > > > > > > that can be solved by running on larger instances.
> > > > > > >
> > > > > > > What all things I need to take care to make sure I get a real
> > time
> > > > > > > posting on endpoint.
> > > > > > >
> > > > > > > I need 20k records to be processed as they receive it
> > > > > > >
> > > > > > >
> > > > > > > thanks in advance
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > > Midhun Mohan
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Midhun Mohan <mi...@esginc.us>.
I new question then ideally nifi should run on which instance, based on aws
On Wed, 18 Mar 2020, 1:58 am Chad Zobrisky, <cz...@gmail.com> wrote:
> Probably can't answer how much it would increase processing speed, but
> processor thread count is limited by the nifi system thread count.
>
> Normally you can increase the nifi system thread count a decent amount,
> 100+, depending on the system.
>
> On Tue, Mar 17, 2020 at 4:19 PM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > Yeah got it thank you.
> > How will Changing that values affect the processing speed
> >
> > On Wed, 18 Mar 2020, 1:45 am Chad Zobrisky, <cz...@gmail.com> wrote:
> >
> > > Hamburger menu in the top right -> Controller Settings -> under the
> > general
> > > tab -> Maximum Timer Driven Thread Count.
> > >
> > > On Tue, Mar 17, 2020 at 4:11 PM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > > > No overall thread pool I haven't changed. How can I change that?
> > > >
> > > > Nifi is running on a standalone aws instance. T2.medium I remember
> so
> > > that
> > > > leaves us with 2 cores I guess
> > > >
> > > > On Wed, 18 Mar 2020, 1:29 am Pierre Villard, <
> > > pierre.villard.fr@gmail.com>
> > > > wrote:
> > > >
> > > > > Did you change the size of the overall thread pool at NiFi level?
> If
> > > yes,
> > > > > to which number? How many cores do you have on NiFi?
> > > > >
> > > > > Thanks
> > > > >
> > > > > Le mar. 17 mars 2020 à 20:55, Midhun Mohan <midhun.mohan@esginc.us
> >
> > a
> > > > > écrit :
> > > > >
> > > > > > Yeah the endpoint which am sending right now has plenty of
> > resources
> > > > > > available. Only thing is I need to send more records
> > > > > >
> > > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <czobrisky@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > I have not test throughput of InvokeHTTP so am not sure what
> the
> > > > > maximum
> > > > > > > is, but can give some general guidance.
> > > > > > >
> > > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to
> figure
> > > out
> > > > > > system
> > > > > > > resources usage while your flow is running.
> > > > > > >
> > > > > > > You should be able to increase both your nifi count by more and
> > > > adjust
> > > > > > your
> > > > > > > processors until you are limited by your system resources.
> > > > > > >
> > > > > > > Have you verified the endpoint you are sending to is not the
> > bottle
> > > > > neck?
> > > > > > >
> > > > > > > Chad
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <
> > > midhun.mohan@esginc.us
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Size of file around kb not more than that,
> > > > > > > > How can I find the bottle neck,
> > > > > > > >
> > > > > > > > Yes I adjust the count to 11 then all other processors
> stopped
> > > but
> > > > > > > > processing improved.
> > > > > > > > Totally at present 6 is showing not more than that.
> > > > > > > >
> > > > > > > > I just need a way to post more records to endpoint to make it
> > > > > realtime
> > > > > > > >
> > > > > > > > Hope i was able to give more details
> > > > > > > >
> > > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <
> > czobrisky@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Midhun,
> > > > > > > > > A little more information would help.
> > > > > > > > >
> > > > > > > > > What size files are you sending?
> > > > > > > > > Have you looked at resource usage to see what the
> bottleneck
> > > is?
> > > > > > > > > Did you adjust your nifi system thread count from the
> > hamburger
> > > > > menu?
> > > > > > > > > How many threads are running total for nifi? It's the
> number
> > in
> > > > the
> > > > > > top
> > > > > > > > > left of the screen.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Chad
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > > > > midhun.mohan@esginc.us
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Do anyone have better idea on this
> > > > > > > > > >
> > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > > > > midhun.mohan@esginc.us
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I have a queue which will be filling up with realtime
> > > records
> > > > > of
> > > > > > > > around
> > > > > > > > > > > 20k records. Which is posting to an endpoint using
> > > Invokehttp
> > > > > > > > > processor.
> > > > > > > > > > >
> > > > > > > > > > > currently the average throughput is for 10k records it
> > > takes
> > > > > > around
> > > > > > > > 20
> > > > > > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > > > > > I Increased the concurrent thread to larger number
> which
> > > is
> > > > > > > > increasing
> > > > > > > > > > > the throughput, but it is causing other processor to
> > > > > terminate. I
> > > > > > > > think
> > > > > > > > > > > that can be solved by running on larger instances.
> > > > > > > > > > >
> > > > > > > > > > > What all things I need to take care to make sure I get
> a
> > > > real
> > > > > > time
> > > > > > > > > > > posting on endpoint.
> > > > > > > > > > >
> > > > > > > > > > > I need 20k records to be processed as they receive it
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > thanks in advance
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Midhun Mohan
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Chad Zobrisky <cz...@gmail.com>.
Probably can't answer how much it would increase processing speed, but
processor thread count is limited by the nifi system thread count.
Normally you can increase the nifi system thread count a decent amount,
100+, depending on the system.
On Tue, Mar 17, 2020 at 4:19 PM Midhun Mohan <mi...@esginc.us> wrote:
> Yeah got it thank you.
> How will Changing that values affect the processing speed
>
> On Wed, 18 Mar 2020, 1:45 am Chad Zobrisky, <cz...@gmail.com> wrote:
>
> > Hamburger menu in the top right -> Controller Settings -> under the
> general
> > tab -> Maximum Timer Driven Thread Count.
> >
> > On Tue, Mar 17, 2020 at 4:11 PM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> > > No overall thread pool I haven't changed. How can I change that?
> > >
> > > Nifi is running on a standalone aws instance. T2.medium I remember so
> > that
> > > leaves us with 2 cores I guess
> > >
> > > On Wed, 18 Mar 2020, 1:29 am Pierre Villard, <
> > pierre.villard.fr@gmail.com>
> > > wrote:
> > >
> > > > Did you change the size of the overall thread pool at NiFi level? If
> > yes,
> > > > to which number? How many cores do you have on NiFi?
> > > >
> > > > Thanks
> > > >
> > > > Le mar. 17 mars 2020 à 20:55, Midhun Mohan <mi...@esginc.us>
> a
> > > > écrit :
> > > >
> > > > > Yeah the endpoint which am sending right now has plenty of
> resources
> > > > > available. Only thing is I need to send more records
> > > > >
> > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <cz...@gmail.com>
> > > wrote:
> > > > >
> > > > > > I have not test throughput of InvokeHTTP so am not sure what the
> > > > maximum
> > > > > > is, but can give some general guidance.
> > > > > >
> > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure
> > out
> > > > > system
> > > > > > resources usage while your flow is running.
> > > > > >
> > > > > > You should be able to increase both your nifi count by more and
> > > adjust
> > > > > your
> > > > > > processors until you are limited by your system resources.
> > > > > >
> > > > > > Have you verified the endpoint you are sending to is not the
> bottle
> > > > neck?
> > > > > >
> > > > > > Chad
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <
> > midhun.mohan@esginc.us
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Size of file around kb not more than that,
> > > > > > > How can I find the bottle neck,
> > > > > > >
> > > > > > > Yes I adjust the count to 11 then all other processors stopped
> > but
> > > > > > > processing improved.
> > > > > > > Totally at present 6 is showing not more than that.
> > > > > > >
> > > > > > > I just need a way to post more records to endpoint to make it
> > > > realtime
> > > > > > >
> > > > > > > Hope i was able to give more details
> > > > > > >
> > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <
> czobrisky@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Midhun,
> > > > > > > > A little more information would help.
> > > > > > > >
> > > > > > > > What size files are you sending?
> > > > > > > > Have you looked at resource usage to see what the bottleneck
> > is?
> > > > > > > > Did you adjust your nifi system thread count from the
> hamburger
> > > > menu?
> > > > > > > > How many threads are running total for nifi? It's the number
> in
> > > the
> > > > > top
> > > > > > > > left of the screen.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Chad
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > > > midhun.mohan@esginc.us
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Do anyone have better idea on this
> > > > > > > > >
> > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > > > midhun.mohan@esginc.us
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I have a queue which will be filling up with realtime
> > records
> > > > of
> > > > > > > around
> > > > > > > > > > 20k records. Which is posting to an endpoint using
> > Invokehttp
> > > > > > > > processor.
> > > > > > > > > >
> > > > > > > > > > currently the average throughput is for 10k records it
> > takes
> > > > > around
> > > > > > > 20
> > > > > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > > > > I Increased the concurrent thread to larger number which
> > is
> > > > > > > increasing
> > > > > > > > > > the throughput, but it is causing other processor to
> > > > terminate. I
> > > > > > > think
> > > > > > > > > > that can be solved by running on larger instances.
> > > > > > > > > >
> > > > > > > > > > What all things I need to take care to make sure I get a
> > > real
> > > > > time
> > > > > > > > > > posting on endpoint.
> > > > > > > > > >
> > > > > > > > > > I need 20k records to be processed as they receive it
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > thanks in advance
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Midhun Mohan
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Midhun Mohan <mi...@esginc.us>.
Yeah got it thank you.
How will Changing that values affect the processing speed
On Wed, 18 Mar 2020, 1:45 am Chad Zobrisky, <cz...@gmail.com> wrote:
> Hamburger menu in the top right -> Controller Settings -> under the general
> tab -> Maximum Timer Driven Thread Count.
>
> On Tue, Mar 17, 2020 at 4:11 PM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > No overall thread pool I haven't changed. How can I change that?
> >
> > Nifi is running on a standalone aws instance. T2.medium I remember so
> that
> > leaves us with 2 cores I guess
> >
> > On Wed, 18 Mar 2020, 1:29 am Pierre Villard, <
> pierre.villard.fr@gmail.com>
> > wrote:
> >
> > > Did you change the size of the overall thread pool at NiFi level? If
> yes,
> > > to which number? How many cores do you have on NiFi?
> > >
> > > Thanks
> > >
> > > Le mar. 17 mars 2020 à 20:55, Midhun Mohan <mi...@esginc.us> a
> > > écrit :
> > >
> > > > Yeah the endpoint which am sending right now has plenty of resources
> > > > available. Only thing is I need to send more records
> > > >
> > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <cz...@gmail.com>
> > wrote:
> > > >
> > > > > I have not test throughput of InvokeHTTP so am not sure what the
> > > maximum
> > > > > is, but can give some general guidance.
> > > > >
> > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure
> out
> > > > system
> > > > > resources usage while your flow is running.
> > > > >
> > > > > You should be able to increase both your nifi count by more and
> > adjust
> > > > your
> > > > > processors until you are limited by your system resources.
> > > > >
> > > > > Have you verified the endpoint you are sending to is not the bottle
> > > neck?
> > > > >
> > > > > Chad
> > > > >
> > > > >
> > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > >
> > > > > > Size of file around kb not more than that,
> > > > > > How can I find the bottle neck,
> > > > > >
> > > > > > Yes I adjust the count to 11 then all other processors stopped
> but
> > > > > > processing improved.
> > > > > > Totally at present 6 is showing not more than that.
> > > > > >
> > > > > > I just need a way to post more records to endpoint to make it
> > > realtime
> > > > > >
> > > > > > Hope i was able to give more details
> > > > > >
> > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <czobrisky@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Midhun,
> > > > > > > A little more information would help.
> > > > > > >
> > > > > > > What size files are you sending?
> > > > > > > Have you looked at resource usage to see what the bottleneck
> is?
> > > > > > > Did you adjust your nifi system thread count from the hamburger
> > > menu?
> > > > > > > How many threads are running total for nifi? It's the number in
> > the
> > > > top
> > > > > > > left of the screen.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Chad
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > > midhun.mohan@esginc.us
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Do anyone have better idea on this
> > > > > > > >
> > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > > midhun.mohan@esginc.us
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I have a queue which will be filling up with realtime
> records
> > > of
> > > > > > around
> > > > > > > > > 20k records. Which is posting to an endpoint using
> Invokehttp
> > > > > > > processor.
> > > > > > > > >
> > > > > > > > > currently the average throughput is for 10k records it
> takes
> > > > around
> > > > > > 20
> > > > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > > > I Increased the concurrent thread to larger number which
> is
> > > > > > increasing
> > > > > > > > > the throughput, but it is causing other processor to
> > > terminate. I
> > > > > > think
> > > > > > > > > that can be solved by running on larger instances.
> > > > > > > > >
> > > > > > > > > What all things I need to take care to make sure I get a
> > real
> > > > time
> > > > > > > > > posting on endpoint.
> > > > > > > > >
> > > > > > > > > I need 20k records to be processed as they receive it
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > thanks in advance
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Midhun Mohan
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Chad Zobrisky <cz...@gmail.com>.
Hamburger menu in the top right -> Controller Settings -> under the general
tab -> Maximum Timer Driven Thread Count.
On Tue, Mar 17, 2020 at 4:11 PM Midhun Mohan <mi...@esginc.us> wrote:
> No overall thread pool I haven't changed. How can I change that?
>
> Nifi is running on a standalone aws instance. T2.medium I remember so that
> leaves us with 2 cores I guess
>
> On Wed, 18 Mar 2020, 1:29 am Pierre Villard, <pi...@gmail.com>
> wrote:
>
> > Did you change the size of the overall thread pool at NiFi level? If yes,
> > to which number? How many cores do you have on NiFi?
> >
> > Thanks
> >
> > Le mar. 17 mars 2020 à 20:55, Midhun Mohan <mi...@esginc.us> a
> > écrit :
> >
> > > Yeah the endpoint which am sending right now has plenty of resources
> > > available. Only thing is I need to send more records
> > >
> > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <cz...@gmail.com>
> wrote:
> > >
> > > > I have not test throughput of InvokeHTTP so am not sure what the
> > maximum
> > > > is, but can give some general guidance.
> > > >
> > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure out
> > > system
> > > > resources usage while your flow is running.
> > > >
> > > > You should be able to increase both your nifi count by more and
> adjust
> > > your
> > > > processors until you are limited by your system resources.
> > > >
> > > > Have you verified the endpoint you are sending to is not the bottle
> > neck?
> > > >
> > > > Chad
> > > >
> > > >
> > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <midhun.mohan@esginc.us
> >
> > > > wrote:
> > > >
> > > > > Size of file around kb not more than that,
> > > > > How can I find the bottle neck,
> > > > >
> > > > > Yes I adjust the count to 11 then all other processors stopped but
> > > > > processing improved.
> > > > > Totally at present 6 is showing not more than that.
> > > > >
> > > > > I just need a way to post more records to endpoint to make it
> > realtime
> > > > >
> > > > > Hope i was able to give more details
> > > > >
> > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <cz...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Midhun,
> > > > > > A little more information would help.
> > > > > >
> > > > > > What size files are you sending?
> > > > > > Have you looked at resource usage to see what the bottleneck is?
> > > > > > Did you adjust your nifi system thread count from the hamburger
> > menu?
> > > > > > How many threads are running total for nifi? It's the number in
> the
> > > top
> > > > > > left of the screen.
> > > > > >
> > > > > > Thanks,
> > > > > > Chad
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > midhun.mohan@esginc.us
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Do anyone have better idea on this
> > > > > > >
> > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > midhun.mohan@esginc.us
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I have a queue which will be filling up with realtime records
> > of
> > > > > around
> > > > > > > > 20k records. Which is posting to an endpoint using Invokehttp
> > > > > > processor.
> > > > > > > >
> > > > > > > > currently the average throughput is for 10k records it takes
> > > around
> > > > > 20
> > > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > > I Increased the concurrent thread to larger number which is
> > > > > increasing
> > > > > > > > the throughput, but it is causing other processor to
> > terminate. I
> > > > > think
> > > > > > > > that can be solved by running on larger instances.
> > > > > > > >
> > > > > > > > What all things I need to take care to make sure I get a
> real
> > > time
> > > > > > > > posting on endpoint.
> > > > > > > >
> > > > > > > > I need 20k records to be processed as they receive it
> > > > > > > >
> > > > > > > >
> > > > > > > > thanks in advance
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Midhun Mohan
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Midhun Mohan <mi...@esginc.us>.
No overall thread pool I haven't changed. How can I change that?
Nifi is running on a standalone aws instance. T2.medium I remember so that
leaves us with 2 cores I guess
On Wed, 18 Mar 2020, 1:29 am Pierre Villard, <pi...@gmail.com>
wrote:
> Did you change the size of the overall thread pool at NiFi level? If yes,
> to which number? How many cores do you have on NiFi?
>
> Thanks
>
> Le mar. 17 mars 2020 à 20:55, Midhun Mohan <mi...@esginc.us> a
> écrit :
>
> > Yeah the endpoint which am sending right now has plenty of resources
> > available. Only thing is I need to send more records
> >
> > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <cz...@gmail.com> wrote:
> >
> > > I have not test throughput of InvokeHTTP so am not sure what the
> maximum
> > > is, but can give some general guidance.
> > >
> > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure out
> > system
> > > resources usage while your flow is running.
> > >
> > > You should be able to increase both your nifi count by more and adjust
> > your
> > > processors until you are limited by your system resources.
> > >
> > > Have you verified the endpoint you are sending to is not the bottle
> neck?
> > >
> > > Chad
> > >
> > >
> > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > > > Size of file around kb not more than that,
> > > > How can I find the bottle neck,
> > > >
> > > > Yes I adjust the count to 11 then all other processors stopped but
> > > > processing improved.
> > > > Totally at present 6 is showing not more than that.
> > > >
> > > > I just need a way to post more records to endpoint to make it
> realtime
> > > >
> > > > Hope i was able to give more details
> > > >
> > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <cz...@gmail.com>
> > wrote:
> > > >
> > > > > Midhun,
> > > > > A little more information would help.
> > > > >
> > > > > What size files are you sending?
> > > > > Have you looked at resource usage to see what the bottleneck is?
> > > > > Did you adjust your nifi system thread count from the hamburger
> menu?
> > > > > How many threads are running total for nifi? It's the number in the
> > top
> > > > > left of the screen.
> > > > >
> > > > > Thanks,
> > > > > Chad
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > >
> > > > > > Do anyone have better idea on this
> > > > > >
> > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I have a queue which will be filling up with realtime records
> of
> > > > around
> > > > > > > 20k records. Which is posting to an endpoint using Invokehttp
> > > > > processor.
> > > > > > >
> > > > > > > currently the average throughput is for 10k records it takes
> > around
> > > > 20
> > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > I Increased the concurrent thread to larger number which is
> > > > increasing
> > > > > > > the throughput, but it is causing other processor to
> terminate. I
> > > > think
> > > > > > > that can be solved by running on larger instances.
> > > > > > >
> > > > > > > What all things I need to take care to make sure I get a real
> > time
> > > > > > > posting on endpoint.
> > > > > > >
> > > > > > > I need 20k records to be processed as they receive it
> > > > > > >
> > > > > > >
> > > > > > > thanks in advance
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > > Midhun Mohan
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Pierre Villard <pi...@gmail.com>.
Did you change the size of the overall thread pool at NiFi level? If yes,
to which number? How many cores do you have on NiFi?
Thanks
Le mar. 17 mars 2020 à 20:55, Midhun Mohan <mi...@esginc.us> a
écrit :
> Yeah the endpoint which am sending right now has plenty of resources
> available. Only thing is I need to send more records
>
> On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <cz...@gmail.com> wrote:
>
> > I have not test throughput of InvokeHTTP so am not sure what the maximum
> > is, but can give some general guidance.
> >
> > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure out
> system
> > resources usage while your flow is running.
> >
> > You should be able to increase both your nifi count by more and adjust
> your
> > processors until you are limited by your system resources.
> >
> > Have you verified the endpoint you are sending to is not the bottle neck?
> >
> > Chad
> >
> >
> > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> > > Size of file around kb not more than that,
> > > How can I find the bottle neck,
> > >
> > > Yes I adjust the count to 11 then all other processors stopped but
> > > processing improved.
> > > Totally at present 6 is showing not more than that.
> > >
> > > I just need a way to post more records to endpoint to make it realtime
> > >
> > > Hope i was able to give more details
> > >
> > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <cz...@gmail.com>
> wrote:
> > >
> > > > Midhun,
> > > > A little more information would help.
> > > >
> > > > What size files are you sending?
> > > > Have you looked at resource usage to see what the bottleneck is?
> > > > Did you adjust your nifi system thread count from the hamburger menu?
> > > > How many threads are running total for nifi? It's the number in the
> top
> > > > left of the screen.
> > > >
> > > > Thanks,
> > > > Chad
> > > >
> > > >
> > > >
> > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <midhun.mohan@esginc.us
> >
> > > > wrote:
> > > >
> > > > > Do anyone have better idea on this
> > > > >
> > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <midhun.mohan@esginc.us
> >
> > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I have a queue which will be filling up with realtime records of
> > > around
> > > > > > 20k records. Which is posting to an endpoint using Invokehttp
> > > > processor.
> > > > > >
> > > > > > currently the average throughput is for 10k records it takes
> around
> > > 20
> > > > > > minutes to complete the invokehttp processor posting.
> > > > > > I Increased the concurrent thread to larger number which is
> > > increasing
> > > > > > the throughput, but it is causing other processor to terminate. I
> > > think
> > > > > > that can be solved by running on larger instances.
> > > > > >
> > > > > > What all things I need to take care to make sure I get a real
> time
> > > > > > posting on endpoint.
> > > > > >
> > > > > > I need 20k records to be processed as they receive it
> > > > > >
> > > > > >
> > > > > > thanks in advance
> > > > > >
> > > > > > --
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Midhun Mohan
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Mike Thomsen <mi...@gmail.com>.
Looping over curl with bash would work too.
On Wed, Mar 18, 2020 at 3:31 PM Midhun Mohan <mi...@esginc.us> wrote:
> Thanks for that suggestion will look into it as well
>
> On Thu, 19 Mar 2020 at 00:59, Mike Thomsen <mi...@gmail.com> wrote:
>
> > Quad core MacBook Pro that's about 3 years ago. So laptop grade i7 w/
> 16GB
> > of RAM and Xmx was the default of 512MB.
> >
> > Easy way to debug this is to write a python script to hit the endpoint
> > repeatedly. Something based on this:
> >
> > import json
> > import requests
> >
> > requests.post(url, json = json_data_var)
> >
> > And then wrap it like this:
> >
> > $> time python benchmark.py
> >
> > On Wed, Mar 18, 2020 at 9:29 AM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> > > Hey Mike , I meant like when I try posting that many records it is
> taking
> > > bit time. Just checking did you tweak around the thread count and what
> is
> > > your instance size. CPU and RAM
> > >
> > > On Wed, 18 Mar 2020 at 18:57, Mike Thomsen <mi...@gmail.com>
> > wrote:
> > >
> > > > By setting the HTTP verb to POST in InvokeHTTP.
> > > >
> > > > On Wed, Mar 18, 2020 at 1:29 AM Midhun Mohan <midhun.mohan@esginc.us
> >
> > > > wrote:
> > > >
> > > > > How did you post 50k flowfiles, that is what am looking for
> > > > >
> > > > > On Wed, 18 Mar 2020 at 02:36, Mike Thomsen <mikerthomsen@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > That's probably a lot of the issue, especially if it's an evented
> > > > service
> > > > > > like a Node service running with one thread. I just did a simple
> > test
> > > > by
> > > > > > posting 50k flowfiles with 4k of JSON in them to an Express hello
> > > world
> > > > > app
> > > > > > and it was able to respond to 50k flowfiles in under 10s using
> > > > InvokeHttp
> > > > > > with only 3 threads. All on my MacBook Pro.
> > > > > >
> > > > > > On Tue, Mar 17, 2020 at 4:54 PM Midhun Mohan <
> > midhun.mohan@esginc.us
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Endpoint does execute db query in a transaction, yes it is in
> the
> > > > same
> > > > > > > setup but different cluster
> > > > > > >
> > > > > > > On Wed, 18 Mar 2020, 2:20 am Mike Thomsen, <
> > mikerthomsen@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > How is the endpoint implemented and what does it do? Also, is
> > it
> > > > > > located
> > > > > > > in
> > > > > > > > the same data center as the EC2 instance running NiFi?
> > > > > > > >
> > > > > > > > On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan <
> > > > midhun.mohan@esginc.us
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Yeah the endpoint which am sending right now has plenty of
> > > > > resources
> > > > > > > > > available. Only thing is I need to send more records
> > > > > > > > >
> > > > > > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <
> > > czobrisky@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I have not test throughput of InvokeHTTP so am not sure
> > what
> > > > the
> > > > > > > > maximum
> > > > > > > > > > is, but can give some general guidance.
> > > > > > > > > >
> > > > > > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to
> > > > figure
> > > > > > out
> > > > > > > > > system
> > > > > > > > > > resources usage while your flow is running.
> > > > > > > > > >
> > > > > > > > > > You should be able to increase both your nifi count by
> more
> > > and
> > > > > > > adjust
> > > > > > > > > your
> > > > > > > > > > processors until you are limited by your system
> resources.
> > > > > > > > > >
> > > > > > > > > > Have you verified the endpoint you are sending to is not
> > the
> > > > > bottle
> > > > > > > > neck?
> > > > > > > > > >
> > > > > > > > > > Chad
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <
> > > > > > midhun.mohan@esginc.us
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Size of file around kb not more than that,
> > > > > > > > > > > How can I find the bottle neck,
> > > > > > > > > > >
> > > > > > > > > > > Yes I adjust the count to 11 then all other processors
> > > > stopped
> > > > > > but
> > > > > > > > > > > processing improved.
> > > > > > > > > > > Totally at present 6 is showing not more than that.
> > > > > > > > > > >
> > > > > > > > > > > I just need a way to post more records to endpoint to
> > make
> > > it
> > > > > > > > realtime
> > > > > > > > > > >
> > > > > > > > > > > Hope i was able to give more details
> > > > > > > > > > >
> > > > > > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <
> > > > > czobrisky@gmail.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Midhun,
> > > > > > > > > > > > A little more information would help.
> > > > > > > > > > > >
> > > > > > > > > > > > What size files are you sending?
> > > > > > > > > > > > Have you looked at resource usage to see what the
> > > > bottleneck
> > > > > > is?
> > > > > > > > > > > > Did you adjust your nifi system thread count from the
> > > > > hamburger
> > > > > > > > menu?
> > > > > > > > > > > > How many threads are running total for nifi? It's the
> > > > number
> > > > > in
> > > > > > > the
> > > > > > > > > top
> > > > > > > > > > > > left of the screen.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Chad
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > > > > > > > midhun.mohan@esginc.us
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Do anyone have better idea on this
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > > > > > > > midhun.mohan@esginc.us
> > > > > > > > > >
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I have a queue which will be filling up with
> > realtime
> > > > > > records
> > > > > > > > of
> > > > > > > > > > > around
> > > > > > > > > > > > > > 20k records. Which is posting to an endpoint
> using
> > > > > > Invokehttp
> > > > > > > > > > > > processor.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > currently the average throughput is for 10k
> records
> > > it
> > > > > > takes
> > > > > > > > > around
> > > > > > > > > > > 20
> > > > > > > > > > > > > > minutes to complete the invokehttp processor
> > posting.
> > > > > > > > > > > > > > I Increased the concurrent thread to larger
> number
> > > > which
> > > > > > is
> > > > > > > > > > > increasing
> > > > > > > > > > > > > > the throughput, but it is causing other processor
> > to
> > > > > > > > terminate. I
> > > > > > > > > > > think
> > > > > > > > > > > > > > that can be solved by running on larger
> instances.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > What all things I need to take care to make sure
> I
> > > get
> > > > a
> > > > > > > real
> > > > > > > > > time
> > > > > > > > > > > > > > posting on endpoint.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > I need 20k records to be processed as they
> receive
> > it
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > thanks in advance
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > --
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > Midhun Mohan
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > >
> > > > > Regards,
> > > > > Midhun Mohan
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > >
> > > Regards,
> > > Midhun Mohan
> > >
> >
>
>
> --
>
>
> Regards,
> Midhun Mohan
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Midhun Mohan <mi...@esginc.us>.
Thanks for that suggestion will look into it as well
On Thu, 19 Mar 2020 at 00:59, Mike Thomsen <mi...@gmail.com> wrote:
> Quad core MacBook Pro that's about 3 years ago. So laptop grade i7 w/ 16GB
> of RAM and Xmx was the default of 512MB.
>
> Easy way to debug this is to write a python script to hit the endpoint
> repeatedly. Something based on this:
>
> import json
> import requests
>
> requests.post(url, json = json_data_var)
>
> And then wrap it like this:
>
> $> time python benchmark.py
>
> On Wed, Mar 18, 2020 at 9:29 AM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > Hey Mike , I meant like when I try posting that many records it is taking
> > bit time. Just checking did you tweak around the thread count and what is
> > your instance size. CPU and RAM
> >
> > On Wed, 18 Mar 2020 at 18:57, Mike Thomsen <mi...@gmail.com>
> wrote:
> >
> > > By setting the HTTP verb to POST in InvokeHTTP.
> > >
> > > On Wed, Mar 18, 2020 at 1:29 AM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > > > How did you post 50k flowfiles, that is what am looking for
> > > >
> > > > On Wed, 18 Mar 2020 at 02:36, Mike Thomsen <mi...@gmail.com>
> > > wrote:
> > > >
> > > > > That's probably a lot of the issue, especially if it's an evented
> > > service
> > > > > like a Node service running with one thread. I just did a simple
> test
> > > by
> > > > > posting 50k flowfiles with 4k of JSON in them to an Express hello
> > world
> > > > app
> > > > > and it was able to respond to 50k flowfiles in under 10s using
> > > InvokeHttp
> > > > > with only 3 threads. All on my MacBook Pro.
> > > > >
> > > > > On Tue, Mar 17, 2020 at 4:54 PM Midhun Mohan <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > >
> > > > > > Endpoint does execute db query in a transaction, yes it is in the
> > > same
> > > > > > setup but different cluster
> > > > > >
> > > > > > On Wed, 18 Mar 2020, 2:20 am Mike Thomsen, <
> mikerthomsen@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > > How is the endpoint implemented and what does it do? Also, is
> it
> > > > > located
> > > > > > in
> > > > > > > the same data center as the EC2 instance running NiFi?
> > > > > > >
> > > > > > > On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan <
> > > midhun.mohan@esginc.us
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Yeah the endpoint which am sending right now has plenty of
> > > > resources
> > > > > > > > available. Only thing is I need to send more records
> > > > > > > >
> > > > > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <
> > czobrisky@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > I have not test throughput of InvokeHTTP so am not sure
> what
> > > the
> > > > > > > maximum
> > > > > > > > > is, but can give some general guidance.
> > > > > > > > >
> > > > > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to
> > > figure
> > > > > out
> > > > > > > > system
> > > > > > > > > resources usage while your flow is running.
> > > > > > > > >
> > > > > > > > > You should be able to increase both your nifi count by more
> > and
> > > > > > adjust
> > > > > > > > your
> > > > > > > > > processors until you are limited by your system resources.
> > > > > > > > >
> > > > > > > > > Have you verified the endpoint you are sending to is not
> the
> > > > bottle
> > > > > > > neck?
> > > > > > > > >
> > > > > > > > > Chad
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <
> > > > > midhun.mohan@esginc.us
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Size of file around kb not more than that,
> > > > > > > > > > How can I find the bottle neck,
> > > > > > > > > >
> > > > > > > > > > Yes I adjust the count to 11 then all other processors
> > > stopped
> > > > > but
> > > > > > > > > > processing improved.
> > > > > > > > > > Totally at present 6 is showing not more than that.
> > > > > > > > > >
> > > > > > > > > > I just need a way to post more records to endpoint to
> make
> > it
> > > > > > > realtime
> > > > > > > > > >
> > > > > > > > > > Hope i was able to give more details
> > > > > > > > > >
> > > > > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <
> > > > czobrisky@gmail.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Midhun,
> > > > > > > > > > > A little more information would help.
> > > > > > > > > > >
> > > > > > > > > > > What size files are you sending?
> > > > > > > > > > > Have you looked at resource usage to see what the
> > > bottleneck
> > > > > is?
> > > > > > > > > > > Did you adjust your nifi system thread count from the
> > > > hamburger
> > > > > > > menu?
> > > > > > > > > > > How many threads are running total for nifi? It's the
> > > number
> > > > in
> > > > > > the
> > > > > > > > top
> > > > > > > > > > > left of the screen.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Chad
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > > > > > > midhun.mohan@esginc.us
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Do anyone have better idea on this
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > > > > > > midhun.mohan@esginc.us
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi all,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I have a queue which will be filling up with
> realtime
> > > > > records
> > > > > > > of
> > > > > > > > > > around
> > > > > > > > > > > > > 20k records. Which is posting to an endpoint using
> > > > > Invokehttp
> > > > > > > > > > > processor.
> > > > > > > > > > > > >
> > > > > > > > > > > > > currently the average throughput is for 10k records
> > it
> > > > > takes
> > > > > > > > around
> > > > > > > > > > 20
> > > > > > > > > > > > > minutes to complete the invokehttp processor
> posting.
> > > > > > > > > > > > > I Increased the concurrent thread to larger number
> > > which
> > > > > is
> > > > > > > > > > increasing
> > > > > > > > > > > > > the throughput, but it is causing other processor
> to
> > > > > > > terminate. I
> > > > > > > > > > think
> > > > > > > > > > > > > that can be solved by running on larger instances.
> > > > > > > > > > > > >
> > > > > > > > > > > > > What all things I need to take care to make sure I
> > get
> > > a
> > > > > > real
> > > > > > > > time
> > > > > > > > > > > > > posting on endpoint.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I need 20k records to be processed as they receive
> it
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > thanks in advance
> > > > > > > > > > > > >
> > > > > > > > > > > > > --
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > Midhun Mohan
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > >
> > > >
> > > > Regards,
> > > > Midhun Mohan
> > > >
> > >
> >
> >
> > --
> >
> >
> > Regards,
> > Midhun Mohan
> >
>
--
Regards,
Midhun Mohan
Re: Make invokehttp to process faster to process more than 20k records
Posted by Mike Thomsen <mi...@gmail.com>.
Quad core MacBook Pro that's about 3 years ago. So laptop grade i7 w/ 16GB
of RAM and Xmx was the default of 512MB.
Easy way to debug this is to write a python script to hit the endpoint
repeatedly. Something based on this:
import json
import requests
requests.post(url, json = json_data_var)
And then wrap it like this:
$> time python benchmark.py
On Wed, Mar 18, 2020 at 9:29 AM Midhun Mohan <mi...@esginc.us> wrote:
> Hey Mike , I meant like when I try posting that many records it is taking
> bit time. Just checking did you tweak around the thread count and what is
> your instance size. CPU and RAM
>
> On Wed, 18 Mar 2020 at 18:57, Mike Thomsen <mi...@gmail.com> wrote:
>
> > By setting the HTTP verb to POST in InvokeHTTP.
> >
> > On Wed, Mar 18, 2020 at 1:29 AM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> > > How did you post 50k flowfiles, that is what am looking for
> > >
> > > On Wed, 18 Mar 2020 at 02:36, Mike Thomsen <mi...@gmail.com>
> > wrote:
> > >
> > > > That's probably a lot of the issue, especially if it's an evented
> > service
> > > > like a Node service running with one thread. I just did a simple test
> > by
> > > > posting 50k flowfiles with 4k of JSON in them to an Express hello
> world
> > > app
> > > > and it was able to respond to 50k flowfiles in under 10s using
> > InvokeHttp
> > > > with only 3 threads. All on my MacBook Pro.
> > > >
> > > > On Tue, Mar 17, 2020 at 4:54 PM Midhun Mohan <midhun.mohan@esginc.us
> >
> > > > wrote:
> > > >
> > > > > Endpoint does execute db query in a transaction, yes it is in the
> > same
> > > > > setup but different cluster
> > > > >
> > > > > On Wed, 18 Mar 2020, 2:20 am Mike Thomsen, <mikerthomsen@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > How is the endpoint implemented and what does it do? Also, is it
> > > > located
> > > > > in
> > > > > > the same data center as the EC2 instance running NiFi?
> > > > > >
> > > > > > On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan <
> > midhun.mohan@esginc.us
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Yeah the endpoint which am sending right now has plenty of
> > > resources
> > > > > > > available. Only thing is I need to send more records
> > > > > > >
> > > > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <
> czobrisky@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > I have not test throughput of InvokeHTTP so am not sure what
> > the
> > > > > > maximum
> > > > > > > > is, but can give some general guidance.
> > > > > > > >
> > > > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to
> > figure
> > > > out
> > > > > > > system
> > > > > > > > resources usage while your flow is running.
> > > > > > > >
> > > > > > > > You should be able to increase both your nifi count by more
> and
> > > > > adjust
> > > > > > > your
> > > > > > > > processors until you are limited by your system resources.
> > > > > > > >
> > > > > > > > Have you verified the endpoint you are sending to is not the
> > > bottle
> > > > > > neck?
> > > > > > > >
> > > > > > > > Chad
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <
> > > > midhun.mohan@esginc.us
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Size of file around kb not more than that,
> > > > > > > > > How can I find the bottle neck,
> > > > > > > > >
> > > > > > > > > Yes I adjust the count to 11 then all other processors
> > stopped
> > > > but
> > > > > > > > > processing improved.
> > > > > > > > > Totally at present 6 is showing not more than that.
> > > > > > > > >
> > > > > > > > > I just need a way to post more records to endpoint to make
> it
> > > > > > realtime
> > > > > > > > >
> > > > > > > > > Hope i was able to give more details
> > > > > > > > >
> > > > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <
> > > czobrisky@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Midhun,
> > > > > > > > > > A little more information would help.
> > > > > > > > > >
> > > > > > > > > > What size files are you sending?
> > > > > > > > > > Have you looked at resource usage to see what the
> > bottleneck
> > > > is?
> > > > > > > > > > Did you adjust your nifi system thread count from the
> > > hamburger
> > > > > > menu?
> > > > > > > > > > How many threads are running total for nifi? It's the
> > number
> > > in
> > > > > the
> > > > > > > top
> > > > > > > > > > left of the screen.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Chad
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > > > > > midhun.mohan@esginc.us
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Do anyone have better idea on this
> > > > > > > > > > >
> > > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > > > > > midhun.mohan@esginc.us
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I have a queue which will be filling up with realtime
> > > > records
> > > > > > of
> > > > > > > > > around
> > > > > > > > > > > > 20k records. Which is posting to an endpoint using
> > > > Invokehttp
> > > > > > > > > > processor.
> > > > > > > > > > > >
> > > > > > > > > > > > currently the average throughput is for 10k records
> it
> > > > takes
> > > > > > > around
> > > > > > > > > 20
> > > > > > > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > > > > > > I Increased the concurrent thread to larger number
> > which
> > > > is
> > > > > > > > > increasing
> > > > > > > > > > > > the throughput, but it is causing other processor to
> > > > > > terminate. I
> > > > > > > > > think
> > > > > > > > > > > > that can be solved by running on larger instances.
> > > > > > > > > > > >
> > > > > > > > > > > > What all things I need to take care to make sure I
> get
> > a
> > > > > real
> > > > > > > time
> > > > > > > > > > > > posting on endpoint.
> > > > > > > > > > > >
> > > > > > > > > > > > I need 20k records to be processed as they receive it
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > thanks in advance
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Midhun Mohan
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > >
> > > Regards,
> > > Midhun Mohan
> > >
> >
>
>
> --
>
>
> Regards,
> Midhun Mohan
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Midhun Mohan <mi...@esginc.us>.
I actually went a bit ahead and tested the database and api load and it
seems very low on current setup. it is around .6 percent for database and
less than 50 percent for endpoint.
I am thinking like since am running it on t2.medium instance the bandwidth
and threads are slowing down the processor.
I am planning to migrate the instance to m5.2xlarge to test if the issue is
still there. what do you think?
On Thu, 19 Mar 2020 at 00:45, Otto Fowler <ot...@gmail.com> wrote:
> Do you have some other client calling that proves your API endpoint isn’t
> the bottleneck? Sorry if I missed that part and you already provided that.
>
>
>
>
> On March 18, 2020 at 09:29:27, Midhun Mohan (midhun.mohan@esginc.us)
> wrote:
>
> Hey Mike , I meant like when I try posting that many records it is taking
> bit time. Just checking did you tweak around the thread count and what is
> your instance size. CPU and RAM
>
> On Wed, 18 Mar 2020 at 18:57, Mike Thomsen <mi...@gmail.com> wrote:
>
> > By setting the HTTP verb to POST in InvokeHTTP.
> >
> > On Wed, Mar 18, 2020 at 1:29 AM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> > > How did you post 50k flowfiles, that is what am looking for
> > >
> > > On Wed, 18 Mar 2020 at 02:36, Mike Thomsen <mi...@gmail.com>
> > wrote:
> > >
> > > > That's probably a lot of the issue, especially if it's an evented
> > service
> > > > like a Node service running with one thread. I just did a simple test
> > by
> > > > posting 50k flowfiles with 4k of JSON in them to an Express hello
> world
> > > app
> > > > and it was able to respond to 50k flowfiles in under 10s using
> > InvokeHttp
> > > > with only 3 threads. All on my MacBook Pro.
> > > >
> > > > On Tue, Mar 17, 2020 at 4:54 PM Midhun Mohan <midhun.mohan@esginc.us
> >
> > > > wrote:
> > > >
> > > > > Endpoint does execute db query in a transaction, yes it is in the
> > same
> > > > > setup but different cluster
> > > > >
> > > > > On Wed, 18 Mar 2020, 2:20 am Mike Thomsen, <mikerthomsen@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > How is the endpoint implemented and what does it do? Also, is it
> > > > located
> > > > > in
> > > > > > the same data center as the EC2 instance running NiFi?
> > > > > >
> > > > > > On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan <
> > midhun.mohan@esginc.us
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Yeah the endpoint which am sending right now has plenty of
> > > resources
> > > > > > > available. Only thing is I need to send more records
> > > > > > >
> > > > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <
> czobrisky@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > I have not test throughput of InvokeHTTP so am not sure what
> > the
> > > > > > maximum
> > > > > > > > is, but can give some general guidance.
> > > > > > > >
> > > > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to
> > figure
> > > > out
> > > > > > > system
> > > > > > > > resources usage while your flow is running.
> > > > > > > >
> > > > > > > > You should be able to increase both your nifi count by more
> and
> > > > > adjust
> > > > > > > your
> > > > > > > > processors until you are limited by your system resources.
> > > > > > > >
> > > > > > > > Have you verified the endpoint you are sending to is not the
> > > bottle
> > > > > > neck?
> > > > > > > >
> > > > > > > > Chad
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <
> > > > midhun.mohan@esginc.us
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Size of file around kb not more than that,
> > > > > > > > > How can I find the bottle neck,
> > > > > > > > >
> > > > > > > > > Yes I adjust the count to 11 then all other processors
> > stopped
> > > > but
> > > > > > > > > processing improved.
> > > > > > > > > Totally at present 6 is showing not more than that.
> > > > > > > > >
> > > > > > > > > I just need a way to post more records to endpoint to make
> it
> > > > > > realtime
> > > > > > > > >
> > > > > > > > > Hope i was able to give more details
> > > > > > > > >
> > > > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <
> > > czobrisky@gmail.com
> > > > >
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Midhun,
> > > > > > > > > > A little more information would help.
> > > > > > > > > >
> > > > > > > > > > What size files are you sending?
> > > > > > > > > > Have you looked at resource usage to see what the
> > bottleneck
> > > > is?
> > > > > > > > > > Did you adjust your nifi system thread count from the
> > > hamburger
> > > > > > menu?
> > > > > > > > > > How many threads are running total for nifi? It's the
> > number
> > > in
> > > > > the
> > > > > > > top
> > > > > > > > > > left of the screen.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Chad
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > > > > > midhun.mohan@esginc.us
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Do anyone have better idea on this
> > > > > > > > > > >
> > > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > > > > > midhun.mohan@esginc.us
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > I have a queue which will be filling up with realtime
> > > > records
> > > > > > of
> > > > > > > > > around
> > > > > > > > > > > > 20k records. Which is posting to an endpoint using
> > > > Invokehttp
> > > > > > > > > > processor.
> > > > > > > > > > > >
> > > > > > > > > > > > currently the average throughput is for 10k records
> it
> > > > takes
> > > > > > > around
> > > > > > > > > 20
> > > > > > > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > > > > > > I Increased the concurrent thread to larger number
> > which
> > > > is
> > > > > > > > > increasing
> > > > > > > > > > > > the throughput, but it is causing other processor to
> > > > > > terminate. I
> > > > > > > > > think
> > > > > > > > > > > > that can be solved by running on larger instances.
> > > > > > > > > > > >
> > > > > > > > > > > > What all things I need to take care to make sure I
> get
> > a
> > > > > real
> > > > > > > time
> > > > > > > > > > > > posting on endpoint.
> > > > > > > > > > > >
> > > > > > > > > > > > I need 20k records to be processed as they receive it
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > thanks in advance
> > > > > > > > > > > >
> > > > > > > > > > > > --
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Midhun Mohan
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > >
> > >
> > > Regards,
> > > Midhun Mohan
> > >
> >
>
>
> --
>
>
> Regards,
> Midhun Mohan
>
--
Regards,
Midhun Mohan
Re: Make invokehttp to process faster to process more than 20k records
Posted by Otto Fowler <ot...@gmail.com>.
Do you have some other client calling that proves your API endpoint isn’t
the bottleneck? Sorry if I missed that part and you already provided that.
On March 18, 2020 at 09:29:27, Midhun Mohan (midhun.mohan@esginc.us) wrote:
Hey Mike , I meant like when I try posting that many records it is taking
bit time. Just checking did you tweak around the thread count and what is
your instance size. CPU and RAM
On Wed, 18 Mar 2020 at 18:57, Mike Thomsen <mi...@gmail.com> wrote:
> By setting the HTTP verb to POST in InvokeHTTP.
>
> On Wed, Mar 18, 2020 at 1:29 AM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > How did you post 50k flowfiles, that is what am looking for
> >
> > On Wed, 18 Mar 2020 at 02:36, Mike Thomsen <mi...@gmail.com>
> wrote:
> >
> > > That's probably a lot of the issue, especially if it's an evented
> service
> > > like a Node service running with one thread. I just did a simple test
> by
> > > posting 50k flowfiles with 4k of JSON in them to an Express hello
world
> > app
> > > and it was able to respond to 50k flowfiles in under 10s using
> InvokeHttp
> > > with only 3 threads. All on my MacBook Pro.
> > >
> > > On Tue, Mar 17, 2020 at 4:54 PM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > > > Endpoint does execute db query in a transaction, yes it is in the
> same
> > > > setup but different cluster
> > > >
> > > > On Wed, 18 Mar 2020, 2:20 am Mike Thomsen, <mi...@gmail.com>
> > > wrote:
> > > >
> > > > > How is the endpoint implemented and what does it do? Also, is it
> > > located
> > > > in
> > > > > the same data center as the EC2 instance running NiFi?
> > > > >
> > > > > On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > >
> > > > > > Yeah the endpoint which am sending right now has plenty of
> > resources
> > > > > > available. Only thing is I need to send more records
> > > > > >
> > > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <czobrisky@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > I have not test throughput of InvokeHTTP so am not sure what
> the
> > > > > maximum
> > > > > > > is, but can give some general guidance.
> > > > > > >
> > > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to
> figure
> > > out
> > > > > > system
> > > > > > > resources usage while your flow is running.
> > > > > > >
> > > > > > > You should be able to increase both your nifi count by more
and
> > > > adjust
> > > > > > your
> > > > > > > processors until you are limited by your system resources.
> > > > > > >
> > > > > > > Have you verified the endpoint you are sending to is not the
> > bottle
> > > > > neck?
> > > > > > >
> > > > > > > Chad
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <
> > > midhun.mohan@esginc.us
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Size of file around kb not more than that,
> > > > > > > > How can I find the bottle neck,
> > > > > > > >
> > > > > > > > Yes I adjust the count to 11 then all other processors
> stopped
> > > but
> > > > > > > > processing improved.
> > > > > > > > Totally at present 6 is showing not more than that.
> > > > > > > >
> > > > > > > > I just need a way to post more records to endpoint to make
it
> > > > > realtime
> > > > > > > >
> > > > > > > > Hope i was able to give more details
> > > > > > > >
> > > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <
> > czobrisky@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Midhun,
> > > > > > > > > A little more information would help.
> > > > > > > > >
> > > > > > > > > What size files are you sending?
> > > > > > > > > Have you looked at resource usage to see what the
> bottleneck
> > > is?
> > > > > > > > > Did you adjust your nifi system thread count from the
> > hamburger
> > > > > menu?
> > > > > > > > > How many threads are running total for nifi? It's the
> number
> > in
> > > > the
> > > > > > top
> > > > > > > > > left of the screen.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Chad
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > > > > midhun.mohan@esginc.us
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Do anyone have better idea on this
> > > > > > > > > >
> > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > > > > midhun.mohan@esginc.us
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I have a queue which will be filling up with realtime
> > > records
> > > > > of
> > > > > > > > around
> > > > > > > > > > > 20k records. Which is posting to an endpoint using
> > > Invokehttp
> > > > > > > > > processor.
> > > > > > > > > > >
> > > > > > > > > > > currently the average throughput is for 10k records
it
> > > takes
> > > > > > around
> > > > > > > > 20
> > > > > > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > > > > > I Increased the concurrent thread to larger number
> which
> > > is
> > > > > > > > increasing
> > > > > > > > > > > the throughput, but it is causing other processor to
> > > > > terminate. I
> > > > > > > > think
> > > > > > > > > > > that can be solved by running on larger instances.
> > > > > > > > > > >
> > > > > > > > > > > What all things I need to take care to make sure I
get
> a
> > > > real
> > > > > > time
> > > > > > > > > > > posting on endpoint.
> > > > > > > > > > >
> > > > > > > > > > > I need 20k records to be processed as they receive it
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > thanks in advance
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Midhun Mohan
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> >
> > Regards,
> > Midhun Mohan
> >
>
--
Regards,
Midhun Mohan
RE: Make invokehttp to process faster to process more than 20k
records
Posted by Rick Braddy <rb...@softnas.com.INVALID>.
Unsubscribe
-----Original Message-----
From: Midhun Mohan <mi...@esginc.us>
Sent: Wednesday, March 18, 2020 8:29 AM
To: dev@nifi.apache.org
Subject: Re: Make invokehttp to process faster to process more than 20k records
Hey Mike , I meant like when I try posting that many records it is taking bit time. Just checking did you tweak around the thread count and what is your instance size. CPU and RAM
On Wed, 18 Mar 2020 at 18:57, Mike Thomsen <mi...@gmail.com> wrote:
> By setting the HTTP verb to POST in InvokeHTTP.
>
> On Wed, Mar 18, 2020 at 1:29 AM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > How did you post 50k flowfiles, that is what am looking for
> >
> > On Wed, 18 Mar 2020 at 02:36, Mike Thomsen <mi...@gmail.com>
> wrote:
> >
> > > That's probably a lot of the issue, especially if it's an evented
> service
> > > like a Node service running with one thread. I just did a simple
> > > test
> by
> > > posting 50k flowfiles with 4k of JSON in them to an Express hello
> > > world
> > app
> > > and it was able to respond to 50k flowfiles in under 10s using
> InvokeHttp
> > > with only 3 threads. All on my MacBook Pro.
> > >
> > > On Tue, Mar 17, 2020 at 4:54 PM Midhun Mohan
> > > <mi...@esginc.us>
> > > wrote:
> > >
> > > > Endpoint does execute db query in a transaction, yes it is in
> > > > the
> same
> > > > setup but different cluster
> > > >
> > > > On Wed, 18 Mar 2020, 2:20 am Mike Thomsen,
> > > > <mi...@gmail.com>
> > > wrote:
> > > >
> > > > > How is the endpoint implemented and what does it do? Also, is
> > > > > it
> > > located
> > > > in
> > > > > the same data center as the EC2 instance running NiFi?
> > > > >
> > > > > On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > >
> > > > > > Yeah the endpoint which am sending right now has plenty of
> > resources
> > > > > > available. Only thing is I need to send more records
> > > > > >
> > > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky,
> > > > > > <czobrisky@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > I have not test throughput of InvokeHTTP so am not sure
> > > > > > > what
> the
> > > > > maximum
> > > > > > > is, but can give some general guidance.
> > > > > > >
> > > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to
> figure
> > > out
> > > > > > system
> > > > > > > resources usage while your flow is running.
> > > > > > >
> > > > > > > You should be able to increase both your nifi count by
> > > > > > > more and
> > > > adjust
> > > > > > your
> > > > > > > processors until you are limited by your system resources.
> > > > > > >
> > > > > > > Have you verified the endpoint you are sending to is not
> > > > > > > the
> > bottle
> > > > > neck?
> > > > > > >
> > > > > > > Chad
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <
> > > midhun.mohan@esginc.us
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Size of file around kb not more than that, How can I
> > > > > > > > find the bottle neck,
> > > > > > > >
> > > > > > > > Yes I adjust the count to 11 then all other processors
> stopped
> > > but
> > > > > > > > processing improved.
> > > > > > > > Totally at present 6 is showing not more than that.
> > > > > > > >
> > > > > > > > I just need a way to post more records to endpoint to
> > > > > > > > make it
> > > > > realtime
> > > > > > > >
> > > > > > > > Hope i was able to give more details
> > > > > > > >
> > > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <
> > czobrisky@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Midhun,
> > > > > > > > > A little more information would help.
> > > > > > > > >
> > > > > > > > > What size files are you sending?
> > > > > > > > > Have you looked at resource usage to see what the
> bottleneck
> > > is?
> > > > > > > > > Did you adjust your nifi system thread count from the
> > hamburger
> > > > > menu?
> > > > > > > > > How many threads are running total for nifi? It's the
> number
> > in
> > > > the
> > > > > > top
> > > > > > > > > left of the screen.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Chad
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > > > > midhun.mohan@esginc.us
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Do anyone have better idea on this
> > > > > > > > > >
> > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > > > > midhun.mohan@esginc.us
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I have a queue which will be filling up with
> > > > > > > > > > > realtime
> > > records
> > > > > of
> > > > > > > > around
> > > > > > > > > > > 20k records. Which is posting to an endpoint using
> > > Invokehttp
> > > > > > > > > processor.
> > > > > > > > > > >
> > > > > > > > > > > currently the average throughput is for 10k
> > > > > > > > > > > records it
> > > takes
> > > > > > around
> > > > > > > > 20
> > > > > > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > > > > > I Increased the concurrent thread to larger
> > > > > > > > > > > number
> which
> > > is
> > > > > > > > increasing
> > > > > > > > > > > the throughput, but it is causing other processor
> > > > > > > > > > > to
> > > > > terminate. I
> > > > > > > > think
> > > > > > > > > > > that can be solved by running on larger instances.
> > > > > > > > > > >
> > > > > > > > > > > What all things I need to take care to make sure I
> > > > > > > > > > > get
> a
> > > > real
> > > > > > time
> > > > > > > > > > > posting on endpoint.
> > > > > > > > > > >
> > > > > > > > > > > I need 20k records to be processed as they receive
> > > > > > > > > > > it
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > thanks in advance
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Midhun Mohan
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> >
> > Regards,
> > Midhun Mohan
> >
>
--
Regards,
Midhun Mohan
Re: Make invokehttp to process faster to process more than 20k records
Posted by Midhun Mohan <mi...@esginc.us>.
Hey Mike , I meant like when I try posting that many records it is taking
bit time. Just checking did you tweak around the thread count and what is
your instance size. CPU and RAM
On Wed, 18 Mar 2020 at 18:57, Mike Thomsen <mi...@gmail.com> wrote:
> By setting the HTTP verb to POST in InvokeHTTP.
>
> On Wed, Mar 18, 2020 at 1:29 AM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > How did you post 50k flowfiles, that is what am looking for
> >
> > On Wed, 18 Mar 2020 at 02:36, Mike Thomsen <mi...@gmail.com>
> wrote:
> >
> > > That's probably a lot of the issue, especially if it's an evented
> service
> > > like a Node service running with one thread. I just did a simple test
> by
> > > posting 50k flowfiles with 4k of JSON in them to an Express hello world
> > app
> > > and it was able to respond to 50k flowfiles in under 10s using
> InvokeHttp
> > > with only 3 threads. All on my MacBook Pro.
> > >
> > > On Tue, Mar 17, 2020 at 4:54 PM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > > > Endpoint does execute db query in a transaction, yes it is in the
> same
> > > > setup but different cluster
> > > >
> > > > On Wed, 18 Mar 2020, 2:20 am Mike Thomsen, <mi...@gmail.com>
> > > wrote:
> > > >
> > > > > How is the endpoint implemented and what does it do? Also, is it
> > > located
> > > > in
> > > > > the same data center as the EC2 instance running NiFi?
> > > > >
> > > > > On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > >
> > > > > > Yeah the endpoint which am sending right now has plenty of
> > resources
> > > > > > available. Only thing is I need to send more records
> > > > > >
> > > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <czobrisky@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > I have not test throughput of InvokeHTTP so am not sure what
> the
> > > > > maximum
> > > > > > > is, but can give some general guidance.
> > > > > > >
> > > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to
> figure
> > > out
> > > > > > system
> > > > > > > resources usage while your flow is running.
> > > > > > >
> > > > > > > You should be able to increase both your nifi count by more and
> > > > adjust
> > > > > > your
> > > > > > > processors until you are limited by your system resources.
> > > > > > >
> > > > > > > Have you verified the endpoint you are sending to is not the
> > bottle
> > > > > neck?
> > > > > > >
> > > > > > > Chad
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <
> > > midhun.mohan@esginc.us
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Size of file around kb not more than that,
> > > > > > > > How can I find the bottle neck,
> > > > > > > >
> > > > > > > > Yes I adjust the count to 11 then all other processors
> stopped
> > > but
> > > > > > > > processing improved.
> > > > > > > > Totally at present 6 is showing not more than that.
> > > > > > > >
> > > > > > > > I just need a way to post more records to endpoint to make it
> > > > > realtime
> > > > > > > >
> > > > > > > > Hope i was able to give more details
> > > > > > > >
> > > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <
> > czobrisky@gmail.com
> > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Midhun,
> > > > > > > > > A little more information would help.
> > > > > > > > >
> > > > > > > > > What size files are you sending?
> > > > > > > > > Have you looked at resource usage to see what the
> bottleneck
> > > is?
> > > > > > > > > Did you adjust your nifi system thread count from the
> > hamburger
> > > > > menu?
> > > > > > > > > How many threads are running total for nifi? It's the
> number
> > in
> > > > the
> > > > > > top
> > > > > > > > > left of the screen.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Chad
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > > > > midhun.mohan@esginc.us
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Do anyone have better idea on this
> > > > > > > > > >
> > > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > > > > midhun.mohan@esginc.us
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > I have a queue which will be filling up with realtime
> > > records
> > > > > of
> > > > > > > > around
> > > > > > > > > > > 20k records. Which is posting to an endpoint using
> > > Invokehttp
> > > > > > > > > processor.
> > > > > > > > > > >
> > > > > > > > > > > currently the average throughput is for 10k records it
> > > takes
> > > > > > around
> > > > > > > > 20
> > > > > > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > > > > > I Increased the concurrent thread to larger number
> which
> > > is
> > > > > > > > increasing
> > > > > > > > > > > the throughput, but it is causing other processor to
> > > > > terminate. I
> > > > > > > > think
> > > > > > > > > > > that can be solved by running on larger instances.
> > > > > > > > > > >
> > > > > > > > > > > What all things I need to take care to make sure I get
> a
> > > > real
> > > > > > time
> > > > > > > > > > > posting on endpoint.
> > > > > > > > > > >
> > > > > > > > > > > I need 20k records to be processed as they receive it
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > thanks in advance
> > > > > > > > > > >
> > > > > > > > > > > --
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Midhun Mohan
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> > --
> >
> >
> > Regards,
> > Midhun Mohan
> >
>
--
Regards,
Midhun Mohan
Re: Make invokehttp to process faster to process more than 20k records
Posted by Mike Thomsen <mi...@gmail.com>.
By setting the HTTP verb to POST in InvokeHTTP.
On Wed, Mar 18, 2020 at 1:29 AM Midhun Mohan <mi...@esginc.us> wrote:
> How did you post 50k flowfiles, that is what am looking for
>
> On Wed, 18 Mar 2020 at 02:36, Mike Thomsen <mi...@gmail.com> wrote:
>
> > That's probably a lot of the issue, especially if it's an evented service
> > like a Node service running with one thread. I just did a simple test by
> > posting 50k flowfiles with 4k of JSON in them to an Express hello world
> app
> > and it was able to respond to 50k flowfiles in under 10s using InvokeHttp
> > with only 3 threads. All on my MacBook Pro.
> >
> > On Tue, Mar 17, 2020 at 4:54 PM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> > > Endpoint does execute db query in a transaction, yes it is in the same
> > > setup but different cluster
> > >
> > > On Wed, 18 Mar 2020, 2:20 am Mike Thomsen, <mi...@gmail.com>
> > wrote:
> > >
> > > > How is the endpoint implemented and what does it do? Also, is it
> > located
> > > in
> > > > the same data center as the EC2 instance running NiFi?
> > > >
> > > > On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan <midhun.mohan@esginc.us
> >
> > > > wrote:
> > > >
> > > > > Yeah the endpoint which am sending right now has plenty of
> resources
> > > > > available. Only thing is I need to send more records
> > > > >
> > > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <cz...@gmail.com>
> > > wrote:
> > > > >
> > > > > > I have not test throughput of InvokeHTTP so am not sure what the
> > > > maximum
> > > > > > is, but can give some general guidance.
> > > > > >
> > > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure
> > out
> > > > > system
> > > > > > resources usage while your flow is running.
> > > > > >
> > > > > > You should be able to increase both your nifi count by more and
> > > adjust
> > > > > your
> > > > > > processors until you are limited by your system resources.
> > > > > >
> > > > > > Have you verified the endpoint you are sending to is not the
> bottle
> > > > neck?
> > > > > >
> > > > > > Chad
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <
> > midhun.mohan@esginc.us
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Size of file around kb not more than that,
> > > > > > > How can I find the bottle neck,
> > > > > > >
> > > > > > > Yes I adjust the count to 11 then all other processors stopped
> > but
> > > > > > > processing improved.
> > > > > > > Totally at present 6 is showing not more than that.
> > > > > > >
> > > > > > > I just need a way to post more records to endpoint to make it
> > > > realtime
> > > > > > >
> > > > > > > Hope i was able to give more details
> > > > > > >
> > > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <
> czobrisky@gmail.com
> > >
> > > > > wrote:
> > > > > > >
> > > > > > > > Midhun,
> > > > > > > > A little more information would help.
> > > > > > > >
> > > > > > > > What size files are you sending?
> > > > > > > > Have you looked at resource usage to see what the bottleneck
> > is?
> > > > > > > > Did you adjust your nifi system thread count from the
> hamburger
> > > > menu?
> > > > > > > > How many threads are running total for nifi? It's the number
> in
> > > the
> > > > > top
> > > > > > > > left of the screen.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Chad
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > > > midhun.mohan@esginc.us
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Do anyone have better idea on this
> > > > > > > > >
> > > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > > > midhun.mohan@esginc.us
> > > > > >
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > I have a queue which will be filling up with realtime
> > records
> > > > of
> > > > > > > around
> > > > > > > > > > 20k records. Which is posting to an endpoint using
> > Invokehttp
> > > > > > > > processor.
> > > > > > > > > >
> > > > > > > > > > currently the average throughput is for 10k records it
> > takes
> > > > > around
> > > > > > > 20
> > > > > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > > > > I Increased the concurrent thread to larger number which
> > is
> > > > > > > increasing
> > > > > > > > > > the throughput, but it is causing other processor to
> > > > terminate. I
> > > > > > > think
> > > > > > > > > > that can be solved by running on larger instances.
> > > > > > > > > >
> > > > > > > > > > What all things I need to take care to make sure I get a
> > > real
> > > > > time
> > > > > > > > > > posting on endpoint.
> > > > > > > > > >
> > > > > > > > > > I need 20k records to be processed as they receive it
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > thanks in advance
> > > > > > > > > >
> > > > > > > > > > --
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Midhun Mohan
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
> --
>
>
> Regards,
> Midhun Mohan
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Midhun Mohan <mi...@esginc.us>.
How did you post 50k flowfiles, that is what am looking for
On Wed, 18 Mar 2020 at 02:36, Mike Thomsen <mi...@gmail.com> wrote:
> That's probably a lot of the issue, especially if it's an evented service
> like a Node service running with one thread. I just did a simple test by
> posting 50k flowfiles with 4k of JSON in them to an Express hello world app
> and it was able to respond to 50k flowfiles in under 10s using InvokeHttp
> with only 3 threads. All on my MacBook Pro.
>
> On Tue, Mar 17, 2020 at 4:54 PM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > Endpoint does execute db query in a transaction, yes it is in the same
> > setup but different cluster
> >
> > On Wed, 18 Mar 2020, 2:20 am Mike Thomsen, <mi...@gmail.com>
> wrote:
> >
> > > How is the endpoint implemented and what does it do? Also, is it
> located
> > in
> > > the same data center as the EC2 instance running NiFi?
> > >
> > > On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > > > Yeah the endpoint which am sending right now has plenty of resources
> > > > available. Only thing is I need to send more records
> > > >
> > > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <cz...@gmail.com>
> > wrote:
> > > >
> > > > > I have not test throughput of InvokeHTTP so am not sure what the
> > > maximum
> > > > > is, but can give some general guidance.
> > > > >
> > > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure
> out
> > > > system
> > > > > resources usage while your flow is running.
> > > > >
> > > > > You should be able to increase both your nifi count by more and
> > adjust
> > > > your
> > > > > processors until you are limited by your system resources.
> > > > >
> > > > > Have you verified the endpoint you are sending to is not the bottle
> > > neck?
> > > > >
> > > > > Chad
> > > > >
> > > > >
> > > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > >
> > > > > > Size of file around kb not more than that,
> > > > > > How can I find the bottle neck,
> > > > > >
> > > > > > Yes I adjust the count to 11 then all other processors stopped
> but
> > > > > > processing improved.
> > > > > > Totally at present 6 is showing not more than that.
> > > > > >
> > > > > > I just need a way to post more records to endpoint to make it
> > > realtime
> > > > > >
> > > > > > Hope i was able to give more details
> > > > > >
> > > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <czobrisky@gmail.com
> >
> > > > wrote:
> > > > > >
> > > > > > > Midhun,
> > > > > > > A little more information would help.
> > > > > > >
> > > > > > > What size files are you sending?
> > > > > > > Have you looked at resource usage to see what the bottleneck
> is?
> > > > > > > Did you adjust your nifi system thread count from the hamburger
> > > menu?
> > > > > > > How many threads are running total for nifi? It's the number in
> > the
> > > > top
> > > > > > > left of the screen.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Chad
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > > midhun.mohan@esginc.us
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Do anyone have better idea on this
> > > > > > > >
> > > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > > midhun.mohan@esginc.us
> > > > >
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > I have a queue which will be filling up with realtime
> records
> > > of
> > > > > > around
> > > > > > > > > 20k records. Which is posting to an endpoint using
> Invokehttp
> > > > > > > processor.
> > > > > > > > >
> > > > > > > > > currently the average throughput is for 10k records it
> takes
> > > > around
> > > > > > 20
> > > > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > > > I Increased the concurrent thread to larger number which
> is
> > > > > > increasing
> > > > > > > > > the throughput, but it is causing other processor to
> > > terminate. I
> > > > > > think
> > > > > > > > > that can be solved by running on larger instances.
> > > > > > > > >
> > > > > > > > > What all things I need to take care to make sure I get a
> > real
> > > > time
> > > > > > > > > posting on endpoint.
> > > > > > > > >
> > > > > > > > > I need 20k records to be processed as they receive it
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > thanks in advance
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Midhun Mohan
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
--
Regards,
Midhun Mohan
Re: Make invokehttp to process faster to process more than 20k records
Posted by Mike Thomsen <mi...@gmail.com>.
That's probably a lot of the issue, especially if it's an evented service
like a Node service running with one thread. I just did a simple test by
posting 50k flowfiles with 4k of JSON in them to an Express hello world app
and it was able to respond to 50k flowfiles in under 10s using InvokeHttp
with only 3 threads. All on my MacBook Pro.
On Tue, Mar 17, 2020 at 4:54 PM Midhun Mohan <mi...@esginc.us> wrote:
> Endpoint does execute db query in a transaction, yes it is in the same
> setup but different cluster
>
> On Wed, 18 Mar 2020, 2:20 am Mike Thomsen, <mi...@gmail.com> wrote:
>
> > How is the endpoint implemented and what does it do? Also, is it located
> in
> > the same data center as the EC2 instance running NiFi?
> >
> > On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> > > Yeah the endpoint which am sending right now has plenty of resources
> > > available. Only thing is I need to send more records
> > >
> > > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <cz...@gmail.com>
> wrote:
> > >
> > > > I have not test throughput of InvokeHTTP so am not sure what the
> > maximum
> > > > is, but can give some general guidance.
> > > >
> > > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure out
> > > system
> > > > resources usage while your flow is running.
> > > >
> > > > You should be able to increase both your nifi count by more and
> adjust
> > > your
> > > > processors until you are limited by your system resources.
> > > >
> > > > Have you verified the endpoint you are sending to is not the bottle
> > neck?
> > > >
> > > > Chad
> > > >
> > > >
> > > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <midhun.mohan@esginc.us
> >
> > > > wrote:
> > > >
> > > > > Size of file around kb not more than that,
> > > > > How can I find the bottle neck,
> > > > >
> > > > > Yes I adjust the count to 11 then all other processors stopped but
> > > > > processing improved.
> > > > > Totally at present 6 is showing not more than that.
> > > > >
> > > > > I just need a way to post more records to endpoint to make it
> > realtime
> > > > >
> > > > > Hope i was able to give more details
> > > > >
> > > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <cz...@gmail.com>
> > > wrote:
> > > > >
> > > > > > Midhun,
> > > > > > A little more information would help.
> > > > > >
> > > > > > What size files are you sending?
> > > > > > Have you looked at resource usage to see what the bottleneck is?
> > > > > > Did you adjust your nifi system thread count from the hamburger
> > menu?
> > > > > > How many threads are running total for nifi? It's the number in
> the
> > > top
> > > > > > left of the screen.
> > > > > >
> > > > > > Thanks,
> > > > > > Chad
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> > midhun.mohan@esginc.us
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Do anyone have better idea on this
> > > > > > >
> > > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> > midhun.mohan@esginc.us
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > I have a queue which will be filling up with realtime records
> > of
> > > > > around
> > > > > > > > 20k records. Which is posting to an endpoint using Invokehttp
> > > > > > processor.
> > > > > > > >
> > > > > > > > currently the average throughput is for 10k records it takes
> > > around
> > > > > 20
> > > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > > I Increased the concurrent thread to larger number which is
> > > > > increasing
> > > > > > > > the throughput, but it is causing other processor to
> > terminate. I
> > > > > think
> > > > > > > > that can be solved by running on larger instances.
> > > > > > > >
> > > > > > > > What all things I need to take care to make sure I get a
> real
> > > time
> > > > > > > > posting on endpoint.
> > > > > > > >
> > > > > > > > I need 20k records to be processed as they receive it
> > > > > > > >
> > > > > > > >
> > > > > > > > thanks in advance
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Midhun Mohan
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Midhun Mohan <mi...@esginc.us>.
Endpoint does execute db query in a transaction, yes it is in the same
setup but different cluster
On Wed, 18 Mar 2020, 2:20 am Mike Thomsen, <mi...@gmail.com> wrote:
> How is the endpoint implemented and what does it do? Also, is it located in
> the same data center as the EC2 instance running NiFi?
>
> On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > Yeah the endpoint which am sending right now has plenty of resources
> > available. Only thing is I need to send more records
> >
> > On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <cz...@gmail.com> wrote:
> >
> > > I have not test throughput of InvokeHTTP so am not sure what the
> maximum
> > > is, but can give some general guidance.
> > >
> > > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure out
> > system
> > > resources usage while your flow is running.
> > >
> > > You should be able to increase both your nifi count by more and adjust
> > your
> > > processors until you are limited by your system resources.
> > >
> > > Have you verified the endpoint you are sending to is not the bottle
> neck?
> > >
> > > Chad
> > >
> > >
> > > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > > > Size of file around kb not more than that,
> > > > How can I find the bottle neck,
> > > >
> > > > Yes I adjust the count to 11 then all other processors stopped but
> > > > processing improved.
> > > > Totally at present 6 is showing not more than that.
> > > >
> > > > I just need a way to post more records to endpoint to make it
> realtime
> > > >
> > > > Hope i was able to give more details
> > > >
> > > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <cz...@gmail.com>
> > wrote:
> > > >
> > > > > Midhun,
> > > > > A little more information would help.
> > > > >
> > > > > What size files are you sending?
> > > > > Have you looked at resource usage to see what the bottleneck is?
> > > > > Did you adjust your nifi system thread count from the hamburger
> menu?
> > > > > How many threads are running total for nifi? It's the number in the
> > top
> > > > > left of the screen.
> > > > >
> > > > > Thanks,
> > > > > Chad
> > > > >
> > > > >
> > > > >
> > > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > >
> > > > > > Do anyone have better idea on this
> > > > > >
> > > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <
> midhun.mohan@esginc.us
> > >
> > > > > wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > I have a queue which will be filling up with realtime records
> of
> > > > around
> > > > > > > 20k records. Which is posting to an endpoint using Invokehttp
> > > > > processor.
> > > > > > >
> > > > > > > currently the average throughput is for 10k records it takes
> > around
> > > > 20
> > > > > > > minutes to complete the invokehttp processor posting.
> > > > > > > I Increased the concurrent thread to larger number which is
> > > > increasing
> > > > > > > the throughput, but it is causing other processor to
> terminate. I
> > > > think
> > > > > > > that can be solved by running on larger instances.
> > > > > > >
> > > > > > > What all things I need to take care to make sure I get a real
> > time
> > > > > > > posting on endpoint.
> > > > > > >
> > > > > > > I need 20k records to be processed as they receive it
> > > > > > >
> > > > > > >
> > > > > > > thanks in advance
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > > Midhun Mohan
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Mike Thomsen <mi...@gmail.com>.
How is the endpoint implemented and what does it do? Also, is it located in
the same data center as the EC2 instance running NiFi?
On Tue, Mar 17, 2020 at 3:55 PM Midhun Mohan <mi...@esginc.us> wrote:
> Yeah the endpoint which am sending right now has plenty of resources
> available. Only thing is I need to send more records
>
> On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <cz...@gmail.com> wrote:
>
> > I have not test throughput of InvokeHTTP so am not sure what the maximum
> > is, but can give some general guidance.
> >
> > 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure out
> system
> > resources usage while your flow is running.
> >
> > You should be able to increase both your nifi count by more and adjust
> your
> > processors until you are limited by your system resources.
> >
> > Have you verified the endpoint you are sending to is not the bottle neck?
> >
> > Chad
> >
> >
> > On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> > > Size of file around kb not more than that,
> > > How can I find the bottle neck,
> > >
> > > Yes I adjust the count to 11 then all other processors stopped but
> > > processing improved.
> > > Totally at present 6 is showing not more than that.
> > >
> > > I just need a way to post more records to endpoint to make it realtime
> > >
> > > Hope i was able to give more details
> > >
> > > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <cz...@gmail.com>
> wrote:
> > >
> > > > Midhun,
> > > > A little more information would help.
> > > >
> > > > What size files are you sending?
> > > > Have you looked at resource usage to see what the bottleneck is?
> > > > Did you adjust your nifi system thread count from the hamburger menu?
> > > > How many threads are running total for nifi? It's the number in the
> top
> > > > left of the screen.
> > > >
> > > > Thanks,
> > > > Chad
> > > >
> > > >
> > > >
> > > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <midhun.mohan@esginc.us
> >
> > > > wrote:
> > > >
> > > > > Do anyone have better idea on this
> > > > >
> > > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <midhun.mohan@esginc.us
> >
> > > > wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > I have a queue which will be filling up with realtime records of
> > > around
> > > > > > 20k records. Which is posting to an endpoint using Invokehttp
> > > > processor.
> > > > > >
> > > > > > currently the average throughput is for 10k records it takes
> around
> > > 20
> > > > > > minutes to complete the invokehttp processor posting.
> > > > > > I Increased the concurrent thread to larger number which is
> > > increasing
> > > > > > the throughput, but it is causing other processor to terminate. I
> > > think
> > > > > > that can be solved by running on larger instances.
> > > > > >
> > > > > > What all things I need to take care to make sure I get a real
> time
> > > > > > posting on endpoint.
> > > > > >
> > > > > > I need 20k records to be processed as they receive it
> > > > > >
> > > > > >
> > > > > > thanks in advance
> > > > > >
> > > > > > --
> > > > > >
> > > > > >
> > > > > > Regards,
> > > > > > Midhun Mohan
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Midhun Mohan <mi...@esginc.us>.
Yeah the endpoint which am sending right now has plenty of resources
available. Only thing is I need to send more records
On Wed, 18 Mar 2020, 1:23 am Chad Zobrisky, <cz...@gmail.com> wrote:
> I have not test throughput of InvokeHTTP so am not sure what the maximum
> is, but can give some general guidance.
>
> 1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure out system
> resources usage while your flow is running.
>
> You should be able to increase both your nifi count by more and adjust your
> processors until you are limited by your system resources.
>
> Have you verified the endpoint you are sending to is not the bottle neck?
>
> Chad
>
>
> On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > Size of file around kb not more than that,
> > How can I find the bottle neck,
> >
> > Yes I adjust the count to 11 then all other processors stopped but
> > processing improved.
> > Totally at present 6 is showing not more than that.
> >
> > I just need a way to post more records to endpoint to make it realtime
> >
> > Hope i was able to give more details
> >
> > On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <cz...@gmail.com> wrote:
> >
> > > Midhun,
> > > A little more information would help.
> > >
> > > What size files are you sending?
> > > Have you looked at resource usage to see what the bottleneck is?
> > > Did you adjust your nifi system thread count from the hamburger menu?
> > > How many threads are running total for nifi? It's the number in the top
> > > left of the screen.
> > >
> > > Thanks,
> > > Chad
> > >
> > >
> > >
> > > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <mi...@esginc.us>
> > > wrote:
> > >
> > > > Do anyone have better idea on this
> > > >
> > > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <mi...@esginc.us>
> > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I have a queue which will be filling up with realtime records of
> > around
> > > > > 20k records. Which is posting to an endpoint using Invokehttp
> > > processor.
> > > > >
> > > > > currently the average throughput is for 10k records it takes around
> > 20
> > > > > minutes to complete the invokehttp processor posting.
> > > > > I Increased the concurrent thread to larger number which is
> > increasing
> > > > > the throughput, but it is causing other processor to terminate. I
> > think
> > > > > that can be solved by running on larger instances.
> > > > >
> > > > > What all things I need to take care to make sure I get a real time
> > > > > posting on endpoint.
> > > > >
> > > > > I need 20k records to be processed as they receive it
> > > > >
> > > > >
> > > > > thanks in advance
> > > > >
> > > > > --
> > > > >
> > > > >
> > > > > Regards,
> > > > > Midhun Mohan
> > > > >
> > > >
> > >
> >
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Chad Zobrisky <cz...@gmail.com>.
I have not test throughput of InvokeHTTP so am not sure what the maximum
is, but can give some general guidance.
1kb isn't bad. For bottleneck I'd use top, iotop, etc. to figure out system
resources usage while your flow is running.
You should be able to increase both your nifi count by more and adjust your
processors until you are limited by your system resources.
Have you verified the endpoint you are sending to is not the bottle neck?
Chad
On Tue, Mar 17, 2020 at 3:47 PM Midhun Mohan <mi...@esginc.us> wrote:
> Size of file around kb not more than that,
> How can I find the bottle neck,
>
> Yes I adjust the count to 11 then all other processors stopped but
> processing improved.
> Totally at present 6 is showing not more than that.
>
> I just need a way to post more records to endpoint to make it realtime
>
> Hope i was able to give more details
>
> On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <cz...@gmail.com> wrote:
>
> > Midhun,
> > A little more information would help.
> >
> > What size files are you sending?
> > Have you looked at resource usage to see what the bottleneck is?
> > Did you adjust your nifi system thread count from the hamburger menu?
> > How many threads are running total for nifi? It's the number in the top
> > left of the screen.
> >
> > Thanks,
> > Chad
> >
> >
> >
> > On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <mi...@esginc.us>
> > wrote:
> >
> > > Do anyone have better idea on this
> > >
> > > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <mi...@esginc.us>
> > wrote:
> > >
> > > > Hi all,
> > > >
> > > > I have a queue which will be filling up with realtime records of
> around
> > > > 20k records. Which is posting to an endpoint using Invokehttp
> > processor.
> > > >
> > > > currently the average throughput is for 10k records it takes around
> 20
> > > > minutes to complete the invokehttp processor posting.
> > > > I Increased the concurrent thread to larger number which is
> increasing
> > > > the throughput, but it is causing other processor to terminate. I
> think
> > > > that can be solved by running on larger instances.
> > > >
> > > > What all things I need to take care to make sure I get a real time
> > > > posting on endpoint.
> > > >
> > > > I need 20k records to be processed as they receive it
> > > >
> > > >
> > > > thanks in advance
> > > >
> > > > --
> > > >
> > > >
> > > > Regards,
> > > > Midhun Mohan
> > > >
> > >
> >
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Midhun Mohan <mi...@esginc.us>.
Size of file around kb not more than that,
How can I find the bottle neck,
Yes I adjust the count to 11 then all other processors stopped but
processing improved.
Totally at present 6 is showing not more than that.
I just need a way to post more records to endpoint to make it realtime
Hope i was able to give more details
On Wed, 18 Mar 2020, 1:05 am Chad Zobrisky, <cz...@gmail.com> wrote:
> Midhun,
> A little more information would help.
>
> What size files are you sending?
> Have you looked at resource usage to see what the bottleneck is?
> Did you adjust your nifi system thread count from the hamburger menu?
> How many threads are running total for nifi? It's the number in the top
> left of the screen.
>
> Thanks,
> Chad
>
>
>
> On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <mi...@esginc.us>
> wrote:
>
> > Do anyone have better idea on this
> >
> > On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <mi...@esginc.us>
> wrote:
> >
> > > Hi all,
> > >
> > > I have a queue which will be filling up with realtime records of around
> > > 20k records. Which is posting to an endpoint using Invokehttp
> processor.
> > >
> > > currently the average throughput is for 10k records it takes around 20
> > > minutes to complete the invokehttp processor posting.
> > > I Increased the concurrent thread to larger number which is increasing
> > > the throughput, but it is causing other processor to terminate. I think
> > > that can be solved by running on larger instances.
> > >
> > > What all things I need to take care to make sure I get a real time
> > > posting on endpoint.
> > >
> > > I need 20k records to be processed as they receive it
> > >
> > >
> > > thanks in advance
> > >
> > > --
> > >
> > >
> > > Regards,
> > > Midhun Mohan
> > >
> >
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Chad Zobrisky <cz...@gmail.com>.
Midhun,
A little more information would help.
What size files are you sending?
Have you looked at resource usage to see what the bottleneck is?
Did you adjust your nifi system thread count from the hamburger menu?
How many threads are running total for nifi? It's the number in the top
left of the screen.
Thanks,
Chad
On Tue, Mar 17, 2020 at 3:25 PM Midhun Mohan <mi...@esginc.us> wrote:
> Do anyone have better idea on this
>
> On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <mi...@esginc.us> wrote:
>
> > Hi all,
> >
> > I have a queue which will be filling up with realtime records of around
> > 20k records. Which is posting to an endpoint using Invokehttp processor.
> >
> > currently the average throughput is for 10k records it takes around 20
> > minutes to complete the invokehttp processor posting.
> > I Increased the concurrent thread to larger number which is increasing
> > the throughput, but it is causing other processor to terminate. I think
> > that can be solved by running on larger instances.
> >
> > What all things I need to take care to make sure I get a real time
> > posting on endpoint.
> >
> > I need 20k records to be processed as they receive it
> >
> >
> > thanks in advance
> >
> > --
> >
> >
> > Regards,
> > Midhun Mohan
> >
>
Re: Make invokehttp to process faster to process more than 20k records
Posted by Midhun Mohan <mi...@esginc.us>.
Do anyone have better idea on this
On Tue, 17 Mar 2020, 6:32 pm Midhun Mohan, <mi...@esginc.us> wrote:
> Hi all,
>
> I have a queue which will be filling up with realtime records of around
> 20k records. Which is posting to an endpoint using Invokehttp processor.
>
> currently the average throughput is for 10k records it takes around 20
> minutes to complete the invokehttp processor posting.
> I Increased the concurrent thread to larger number which is increasing
> the throughput, but it is causing other processor to terminate. I think
> that can be solved by running on larger instances.
>
> What all things I need to take care to make sure I get a real time
> posting on endpoint.
>
> I need 20k records to be processed as they receive it
>
>
> thanks in advance
>
> --
>
>
> Regards,
> Midhun Mohan
>