connectionTTL causes connection error when consumer takes too long to process message

[Thai Le <ln...@gmail.com>]

Hello,

I have been running consumer with connectionTTL=10000 for some time.
Recently I noticed that when the consumer takes longer than 10 sec to
process the message I got connection error. This makes sense to me because
the server will clear the session after 10 sec (maybe 2 second more due to
ttl-check-interval) and thus making the consumer fail to use the same
session to ack.
Since I am running a cluster of 2 brokers, I think I have to use
connectionTTL to make message redistribution work since the server need to
clear the consumer connection to be ready for redistribution when the
consumer restarts and connects to the other broker. Thus, connectionTTL has
to be shorter than the time kubernetes restart the consumer (about 15 sec),
however this has to be more than the max time a consumer needs to process a
message (around 2 mins). I now think my use of connectionTTL as a way to
notify the broker that the consumer has disconnected is not right. Is there
another config that I need to use?

Regards

Thai Le

Re: connectionTTL causes connection error when consumer takes too long to process message

[Justin Bertram <jb...@apache.org>]

The connection's TTL should not elapse simply because your consumer is busy
processing a message. The core client implementation will still communicate
with the broker behind the scenes to keep the connection alive. If the
connection is failing it's likely due to some kind of network failure.

Can you share the exception stack-trace or error message you receive when
the connection fails? Is this something you can reproduce reliably?


Justin

On Wed, Aug 25, 2021 at 11:02 AM Thai Le <ln...@gmail.com> wrote:

> Hello,
>
> I have been running consumer with connectionTTL=10000 for some time.
> Recently I noticed that when the consumer takes longer than 10 sec to
> process the message I got connection error. This makes sense to me because
> the server will clear the session after 10 sec (maybe 2 second more due to
> ttl-check-interval) and thus making the consumer fail to use the same
> session to ack.
> Since I am running a cluster of 2 brokers, I think I have to use
> connectionTTL to make message redistribution work since the server need to
> clear the consumer connection to be ready for redistribution when the
> consumer restarts and connects to the other broker. Thus, connectionTTL has
> to be shorter than the time kubernetes restart the consumer (about 15 sec),
> however this has to be more than the max time a consumer needs to process a
> message (around 2 mins). I now think my use of connectionTTL as a way to
> notify the broker that the consumer has disconnected is not right. Is there
> another config that I need to use?
>
> Regards
>
> Thai Le
>