InvokeHTTP performance - max out at 50 post per sec.

["Jens M. Kofoed" <jm...@gmail.com>]

Hi

I have created some small test with invokeHTTP, and it seems like that
invokeHTTP only can handle 50 flowfiles per second. If I configure the
invokeHTTP to have x Concurrent Tasks, the output increase with x times.
 I created a very small flow
GenerateFlowFile
  File Size: 100kb
  Batch size: 100
  Run Schedule: 1 sec
->
InvokeHTTP:
  HTTP Method: POST

The receiving host, is another nifi node
with HandleHttpRequest/HandleHttpResponse. And it has no issue handling 4x
invokeHTTP.

So are there something in invokeHTTP which can be tweaked to perform more
than 50 files per sec?

Kind regards
Jens M. Kofoed

Re: InvokeHTTP performance - max out at 50 post per sec.

["Jens M. Kofoed" <jm...@gmail.com>]

Hi Isha

Thanks for your reply.
I have tried to play with the scheduling, and it doesn't help much. So I
will use the concurrent task. Had hoped there where other things which
could be tweaked to get it to run faster.

kind regards
Jens M. Kofoed


Den fre. 4. mar. 2022 kl. 10.16 skrev Isha Lamboo <
isha.lamboo@virtualsciences.nl>:

> Hi Jens,
>
>
>
> The behaviour you describe doesn’t seem abnormal if the round-trip time
> for a request is around 20ms in you test setup. A single InvokeHTTP would
> process requests one by one and that gives 1000 ms /20 ms = 50 requests per
> second. Increasing concurrent tasks to send requests in parallel is exactly
> the right solution here and you test results show as much.
>
>
>
> If the round trip time is much smaller than 20 ms, you should be able to
> tune the scheduling to get more throughput per concurrent task. I would
> test increasing the run duration to let the processor take on more tasks
> without yielding in between. This will impact the rest of your  flows if
> you set it too high though.
>
>
>
> Regards,
>
>
>
> Isha
>
>
>
> *Van:* Jens M. Kofoed <jm...@gmail.com>
> *Verzonden:* vrijdag 4 maart 2022 09:35
> *Aan:* users@nifi.apache.org
> *Onderwerp:* InvokeHTTP performance - max out at 50 post per sec.
>
>
>
> Hi
>
>
>
> I have created some small test with invokeHTTP, and it seems like that
> invokeHTTP only can handle 50 flowfiles per second. If I configure the
> invokeHTTP to have x Concurrent Tasks, the output increase with x times.
>
>  I created a very small flow
>
> GenerateFlowFile
>
>   File Size: 100kb
>
>   Batch size: 100
>
>   Run Schedule: 1 sec
>
> ->
>
> InvokeHTTP:
>
>   HTTP Method: POST
>
>
>
> The receiving host, is another nifi node
> with HandleHttpRequest/HandleHttpResponse. And it has no issue handling 4x
> invokeHTTP.
>
>
>
> So are there something in invokeHTTP which can be tweaked to perform more
> than 50 files per sec?
>
>
>
> Kind regards
>
> Jens M. Kofoed
>
>
>
>
>

RE: InvokeHTTP performance - max out at 50 post per sec.

[Isha Lamboo <is...@virtualsciences.nl>]

Hi Jens,

The behaviour you describe doesn’t seem abnormal if the round-trip time for a request is around 20ms in you test setup. A single InvokeHTTP would process requests one by one and that gives 1000 ms /20 ms = 50 requests per second. Increasing concurrent tasks to send requests in parallel is exactly the right solution here and you test results show as much.

If the round trip time is much smaller than 20 ms, you should be able to tune the scheduling to get more throughput per concurrent task. I would test increasing the run duration to let the processor take on more tasks without yielding in between. This will impact the rest of your  flows if you set it too high though.

Regards,

Isha

Van: Jens M. Kofoed <jm...@gmail.com>
Verzonden: vrijdag 4 maart 2022 09:35
Aan: users@nifi.apache.org
Onderwerp: InvokeHTTP performance - max out at 50 post per sec.

Hi

I have created some small test with invokeHTTP, and it seems like that invokeHTTP only can handle 50 flowfiles per second. If I configure the invokeHTTP to have x Concurrent Tasks, the output increase with x times.
 I created a very small flow
GenerateFlowFile
  File Size: 100kb
  Batch size: 100
  Run Schedule: 1 sec
->
InvokeHTTP:
  HTTP Method: POST

The receiving host, is another nifi node with HandleHttpRequest/HandleHttpResponse. And it has no issue handling 4x invokeHTTP.

So are there something in invokeHTTP which can be tweaked to perform more than 50 files per sec?

Kind regards
Jens M. Kofoed