You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Michael Moser <mo...@gmail.com> on 2025/03/13 19:05:51 UTC

[DISCUSS] ListenHTTP rate limiter

Team,

tl;dr I would like to remove the ListenHTTP "Max Data to Receive per
Second" capability because it has problems.  Thoughts?

I've been experimenting with the ListenHTTP "Max Data to Receive per
Second" feature and I've found some concerning results.
If I set "Max Data to Receive per Second" to a low value such as "500 KB"
and send a 5 MB file using InvokeHTTP, then ListenHTTP receives the file
after 10 seconds.  Great!
But if the size of the file would take longer than the InvokeHTTP "Socket
Read Timeout" (default 15 seconds), for example sending a 20 MB file (which
would take 40 seconds),
then after 15 seconds InvokeHTTP experiences a SocketTimeoutException and
routes the file to Failure.  After 40 seconds elapses, ListenHTTP completes
and receives the 20 MB file!  If the InvokeHTTP Failure relationship is
retried, then the receiving system gets duplicate files.

At larger "Max Data to Receive per Second" values such as "10 MB",
SocketTimeoutExceptions are harder to reproduce but I can still make it
happen by heavily loading ListenHTTP.

As an alternative to this property, if a ListenHTTP is followed by
ControlRate with tight back pressure in between, then a limited data rate
appears to work much better.
When InvokeHTTP is the client, and ListenHTTP experiences backpressure
and does things like send a 503 response or deny connections or close
connections,
InvokeHTTP experiences various errors.  This results in files going to
either Retry or Failure relationships.
When those are retried, then the number of files that InvokeHTTP wants to
send exactly equal the number of files that ListenHTTP receives.  Also
great!

So, I propose removing the ListenHTTP "Max Data to Receive per Second"
property, with proper migration of properties.  And documenting a UseCase
on ListenHTTP to explain
the recommended approach (backpressure) to rate limiting.

Thanks for your thoughts,
-- Mike

Re: [DISCUSS] ListenHTTP rate limiter

Posted by Mark Bean <ma...@gmail.com>.
Given the observed behavior, I am in support of removing this property
as well. In addition, the fact that this property is on the listening
side and not the posting side indicates this is a flow rate limiter.
It is not attempting to throttle the transfer rate. Therefore, the
appropriate - and arguably more desirable - approach here is to let
the data flow handle the rate throttle. As you've suggested,
ControlRate and possible back pressure that results is a great
solution.

The inclusion of a use case in the processor documentation is a great
idea and will assist users who may feel they lost functionality.

As Lucas mentions, this change should be clearly documented in the
migration guide when it is included in a release.

It's not always a great idea to change some base processors that are
very heavily utilized. However, the benefits here are significant
especially when possible duplication of data may occur.

+1 on recommended changes including documentation.

On Thu, Mar 13, 2025 at 3:45 PM Lucas Ottersbach <ot...@apache.org> wrote:
>
> Thanks for opening the discussion Michael.
>
> Separating the concerns of HTTP and rate limiting sounds reasonable to me.
> Especially when considering the limitations of the current implementation
> you've outlined.
> +1 from me on this issue.
>
> While I'm in favour, we should ensure this change is documented properly in
> the migration guide. Otherwise, users might experience undesired loads due
> to the limit being migrated away automatically but the corresponding
> ControlRate processor missing.
>
> Best regards,
> Lucas
>
> Michael Moser <mo...@gmail.com> schrieb am Do., 13. März 2025, 20:08:
>
> > Team,
> >
> > tl;dr I would like to remove the ListenHTTP "Max Data to Receive per
> > Second" capability because it has problems.  Thoughts?
> >
> > I've been experimenting with the ListenHTTP "Max Data to Receive per
> > Second" feature and I've found some concerning results.
> > If I set "Max Data to Receive per Second" to a low value such as "500 KB"
> > and send a 5 MB file using InvokeHTTP, then ListenHTTP receives the file
> > after 10 seconds.  Great!
> > But if the size of the file would take longer than the InvokeHTTP "Socket
> > Read Timeout" (default 15 seconds), for example sending a 20 MB file (which
> > would take 40 seconds),
> > then after 15 seconds InvokeHTTP experiences a SocketTimeoutException and
> > routes the file to Failure.  After 40 seconds elapses, ListenHTTP completes
> > and receives the 20 MB file!  If the InvokeHTTP Failure relationship is
> > retried, then the receiving system gets duplicate files.
> >
> > At larger "Max Data to Receive per Second" values such as "10 MB",
> > SocketTimeoutExceptions are harder to reproduce but I can still make it
> > happen by heavily loading ListenHTTP.
> >
> > As an alternative to this property, if a ListenHTTP is followed by
> > ControlRate with tight back pressure in between, then a limited data rate
> > appears to work much better.
> > When InvokeHTTP is the client, and ListenHTTP experiences backpressure
> > and does things like send a 503 response or deny connections or close
> > connections,
> > InvokeHTTP experiences various errors.  This results in files going to
> > either Retry or Failure relationships.
> > When those are retried, then the number of files that InvokeHTTP wants to
> > send exactly equal the number of files that ListenHTTP receives.  Also
> > great!
> >
> > So, I propose removing the ListenHTTP "Max Data to Receive per Second"
> > property, with proper migration of properties.  And documenting a UseCase
> > on ListenHTTP to explain
> > the recommended approach (backpressure) to rate limiting.
> >
> > Thanks for your thoughts,
> > -- Mike
> >

Re: [DISCUSS] ListenHTTP rate limiter

Posted by Lucas Ottersbach <ot...@apache.org>.
Thanks for opening the discussion Michael.

Separating the concerns of HTTP and rate limiting sounds reasonable to me.
Especially when considering the limitations of the current implementation
you've outlined.
+1 from me on this issue.

While I'm in favour, we should ensure this change is documented properly in
the migration guide. Otherwise, users might experience undesired loads due
to the limit being migrated away automatically but the corresponding
ControlRate processor missing.

Best regards,
Lucas

Michael Moser <mo...@gmail.com> schrieb am Do., 13. März 2025, 20:08:

> Team,
>
> tl;dr I would like to remove the ListenHTTP "Max Data to Receive per
> Second" capability because it has problems.  Thoughts?
>
> I've been experimenting with the ListenHTTP "Max Data to Receive per
> Second" feature and I've found some concerning results.
> If I set "Max Data to Receive per Second" to a low value such as "500 KB"
> and send a 5 MB file using InvokeHTTP, then ListenHTTP receives the file
> after 10 seconds.  Great!
> But if the size of the file would take longer than the InvokeHTTP "Socket
> Read Timeout" (default 15 seconds), for example sending a 20 MB file (which
> would take 40 seconds),
> then after 15 seconds InvokeHTTP experiences a SocketTimeoutException and
> routes the file to Failure.  After 40 seconds elapses, ListenHTTP completes
> and receives the 20 MB file!  If the InvokeHTTP Failure relationship is
> retried, then the receiving system gets duplicate files.
>
> At larger "Max Data to Receive per Second" values such as "10 MB",
> SocketTimeoutExceptions are harder to reproduce but I can still make it
> happen by heavily loading ListenHTTP.
>
> As an alternative to this property, if a ListenHTTP is followed by
> ControlRate with tight back pressure in between, then a limited data rate
> appears to work much better.
> When InvokeHTTP is the client, and ListenHTTP experiences backpressure
> and does things like send a 503 response or deny connections or close
> connections,
> InvokeHTTP experiences various errors.  This results in files going to
> either Retry or Failure relationships.
> When those are retried, then the number of files that InvokeHTTP wants to
> send exactly equal the number of files that ListenHTTP receives.  Also
> great!
>
> So, I propose removing the ListenHTTP "Max Data to Receive per Second"
> property, with proper migration of properties.  And documenting a UseCase
> on ListenHTTP to explain
> the recommended approach (backpressure) to rate limiting.
>
> Thanks for your thoughts,
> -- Mike
>