[Discuss] Let the client perceive the rate limit

[linlin <li...@apache.org>]

Hi all:
Now, Broker limits the rate of producing messages by setting
`channel.setAutoRead(false)`. But no exception is returned to the client,
or any log is printed, which makes it very difficult for us to troubleshoot
the problem.

The client timeout when sending messages and the connection will be closed
(the client's heartbeat not be processed by Broker). This has brought a lot
of obstacles to our troubleshooting. We don't know if there is a problem
with the Broker or the rate is limited.

I suggest that before setting AutoRead=false, at least one exception should
be returned to the client, so that the user can perceive that it has been
throttled, or at least print the log on the Broker side. Returning an
exception will change the existing behavior

I look forward to your better solution

Re: [Discuss] Let the client perceive the rate limit

[PengHui Li <pe...@apache.org>]

If the producer has been throttled, we should return the exception to the
user after exceeding the publish timeout.
Otherwise, if a producer has been throttled for 1ms, the producer failed
all the pending messages(ordering guarantee),
this might introduce confusion and might introduce more workload after
users retry the messages(the inflight message will be processed).
So I think it's better to provide useful information to users after
publishing message failed with a timeout, such as
"Publish message timeout, the producer has been throttled by the broker",
which will help on the problem investigating.

The throttled producer will also increase the publish latency but have not
reached the publish timeout, for this case, we can
rely on the client-side logs.

-Penghui

On Mon, Nov 1, 2021 at 5:57 PM linlin <li...@apache.org> wrote:

> Hi all:
> Now, Broker limits the rate of producing messages by setting
> `channel.setAutoRead(false)`. But no exception is returned to the client,
> or any log is printed, which makes it very difficult for us to troubleshoot
> the problem.
>
> The client timeout when sending messages and the connection will be closed
> (the client's heartbeat not be processed by Broker). This has brought a lot
> of obstacles to our troubleshooting. We don't know if there is a problem
> with the Broker or the rate is limited.
>
> I suggest that before setting AutoRead=false, at least one exception should
> be returned to the client, so that the user can perceive that it has been
> throttled, or at least print the log on the Broker side. Returning an
> exception will change the existing behavior
>
> I look forward to your better solution
>