[DISCUSS] Optimize the Acktimeout Mechanism in Pulsar Client

[Xiangying Meng <xi...@apache.org>]

Dear Pulsar Community,

I would like to initiate a discussion regarding the optimization of
the acktimeout mechanism on the client side. As we all know, the
Pulsar consumer has a configuration for ack timeout that automatically
redelivers unacknowledged messages after a certain period. The
workflow is approximately as follows:

1. Record a message when it is received.
2. Remove the message from the record when the consumer begins to
acknowledge it.
3. A timed task checks whether the messages from the record have
reached the ack timeout and triggers redelivery for those messages.

This workflow has a potential shortcoming: it does not wait for the
ack response before removing the message from the record. If an ack
request is lost during transmission to the broker - due to issues with
proxy processing or buffer overflow after reaching the broker, for
instance - then the ack timeout will not be resent, leading to an ack
hole. While this situation is quite extreme and the likelihood of
occurrence is extremely low, it is nonetheless a possibility. Another,
more common scenario is when the broker fails to process or persist
the ack request.

In such cases, users who are highly sensitive to ack holes may prefer
the message to be removed only after receiving the ack response.
Perhaps we could add a parameter to the acktimeout to determine
whether to wait for the ack response before removing the message.

I am interested in hearing your thoughts on this issue and look
forward to your responses and valuable suggestions.

Best Regards,
Xiangying

Re: [DISCUSS] Optimize the Acktimeout Mechanism in Pulsar Client

[Xiangying Meng <xi...@apache.org>]

I have considered this issue, but it may increase the user's consumption delay.
However, it is not appropriate to make modifications in the ack timeout.
Yubiao and I have just discussed this issue, and we should handle it
in the AcknowledgementsGroupingTracker to automatically retry failed
ack requests.

Thanks,

On Wed, Mar 27, 2024 at 5:37 PM ZhangJian He <sh...@gmail.com> wrote:
>
> Hi, Xiangying. Have you ever considered the `isAckReceiptEnabled` param?
>
> Thanks
> ZhangJian He
>
>
> On Wed, Mar 27, 2024 at 3:33 PM Xiangying Meng <xi...@apache.org> wrote:
>
> > Dear Pulsar Community,
> >
> > I would like to initiate a discussion regarding the optimization of
> > the acktimeout mechanism on the client side. As we all know, the
> > Pulsar consumer has a configuration for ack timeout that automatically
> > redelivers unacknowledged messages after a certain period. The
> > workflow is approximately as follows:
> >
> > 1. Record a message when it is received.
> > 2. Remove the message from the record when the consumer begins to
> > acknowledge it.
> > 3. A timed task checks whether the messages from the record have
> > reached the ack timeout and triggers redelivery for those messages.
> >
> > This workflow has a potential shortcoming: it does not wait for the
> > ack response before removing the message from the record. If an ack
> > request is lost during transmission to the broker - due to issues with
> > proxy processing or buffer overflow after reaching the broker, for
> > instance - then the ack timeout will not be resent, leading to an ack
> > hole. While this situation is quite extreme and the likelihood of
> > occurrence is extremely low, it is nonetheless a possibility. Another,
> > more common scenario is when the broker fails to process or persist
> > the ack request.
> >
> > In such cases, users who are highly sensitive to ack holes may prefer
> > the message to be removed only after receiving the ack response.
> > Perhaps we could add a parameter to the acktimeout to determine
> > whether to wait for the ack response before removing the message.
> >
> > I am interested in hearing your thoughts on this issue and look
> > forward to your responses and valuable suggestions.
> >
> > Best Regards,
> > Xiangying
> >

Re: [DISCUSS] Optimize the Acktimeout Mechanism in Pulsar Client

[ZhangJian He <sh...@gmail.com>]

Hi, Xiangying. Have you ever considered the `isAckReceiptEnabled` param?

Thanks
ZhangJian He


On Wed, Mar 27, 2024 at 3:33 PM Xiangying Meng <xi...@apache.org> wrote:

> Dear Pulsar Community,
>
> I would like to initiate a discussion regarding the optimization of
> the acktimeout mechanism on the client side. As we all know, the
> Pulsar consumer has a configuration for ack timeout that automatically
> redelivers unacknowledged messages after a certain period. The
> workflow is approximately as follows:
>
> 1. Record a message when it is received.
> 2. Remove the message from the record when the consumer begins to
> acknowledge it.
> 3. A timed task checks whether the messages from the record have
> reached the ack timeout and triggers redelivery for those messages.
>
> This workflow has a potential shortcoming: it does not wait for the
> ack response before removing the message from the record. If an ack
> request is lost during transmission to the broker - due to issues with
> proxy processing or buffer overflow after reaching the broker, for
> instance - then the ack timeout will not be resent, leading to an ack
> hole. While this situation is quite extreme and the likelihood of
> occurrence is extremely low, it is nonetheless a possibility. Another,
> more common scenario is when the broker fails to process or persist
> the ack request.
>
> In such cases, users who are highly sensitive to ack holes may prefer
> the message to be removed only after receiving the ack response.
> Perhaps we could add a parameter to the acktimeout to determine
> whether to wait for the ack response before removing the message.
>
> I am interested in hearing your thoughts on this issue and look
> forward to your responses and valuable suggestions.
>
> Best Regards,
> Xiangying
>