You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Asaf Mesika <as...@gmail.com> on 2022/09/04 14:37:17 UTC
Re: [DISCUSS] User-friendly acknowledgeCumulative API on a partitioned topic or multi-topics
What eventually happened with this idea?
On Fri, Jul 29, 2022 at 8:02 AM PengHui Li <co...@gmail.com> wrote:
> +1
>
> Penghui
> On Jul 28, 2022, 20:14 +0800, lordcheng10 <15...@qq.com.invalid>,
> wrote:
> > Nice feature!
> >
> >
> >
> >
> > ------------------ Original ------------------
> > From: "Yunze Xu"<yzxu@streamnative.io.INVALID>;
> > Date: 2022Äê7ÔÂ15ÈÕ(ÐÇÆÚÎå) ÍíÉÏ6:04
> > To: "dev"<dev@pulsar.apache.org>;
> > Subject: [DISCUSS] User-friendly acknowledgeCumulative API on a
> partitioned topic or multi-topics
> >
> >
> >
> > Hi all,
> >
> > Long days ago I opened a PR to support cumulative acknowledgement
> > for C++ client, but it's controversial about whether should a
> > partitioned consumer acknowledge a message ID cumulatively.
> >
> > See https://github.com/apache/pulsar/pull/6796 for discussion.
> >
> > Currently, the Java client acknowledges the specific partition of the
> > message ID, while the C++ client just fails when calling
> > `acknowledgeCumulative` on a partitioned topic. However, even if the
> > Java client doesn't fail, it's not user friendly.
> >
> > Assuming users called `acknowledgeCumulative` periodically, there is a
> > chance that some messages of the specific partition has never been
> > passed to the method.
> >
> > For example, a consumer received:
> >
> > P0-M0, P1-M0, P0-M1, P1-M1, P0-M2, P1-M2...
> >
> > And the user acknowledged every two messages, i.e.
> >
> > P0-M0, P0-M1, P0-M2
> >
> > Eventually, partition 1 has never been acknowledged.
> >
> > User must maintain its own `Map<String, MessageId>` cache for a
> > partitioned topic or multi-topics consumer with the existing
> > `acknowledgeCumulative` API.
> >
> > Should we make it more friendly for users? For example, we can make
> > `acknowledgeCumulative` accept the map to remind users to maintain
> > the map from topic name to message ID:
> >
> > ```java
> > // the key is the partitioned topic name like my-topic-partition-0
> > void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
> > ```
> >
> > For those who don't want to maintain the map by themselves, maybe we
> > can provide a simpler API like:
> >
> > ```java
> > // acknowlegde all latest received messages
> > void acknowledgeCumulative();
> > ```
> >
> > and provide an option to enable this behavior.
> >
> > Do you have any suggestion on this idea? I will prepare a proposal if
> > there is no disagreement.
> >
> > Thanks,
> > Yunze
>
Re: [DISCUSS] User-friendly acknowledgeCumulative API on a partitioned topic or multi-topics
Posted by Yunze Xu <yz...@streamnative.io.INVALID>.
Yes, I agree. BTW, we can continue the discussion in your proposal.
https://lists.apache.org/thread/3vd136jfhh7g0mhgwxrnqk7mx69qy17m
Thanks,
Yunze
> On Sep 14, 2022, at 14:17, Tarun Annapareddy <ta...@gmail.com> wrote:
>
> Hi,
> There is a concern about the validation of the topic string key in the
> map we are using. In this case is it good to have Map<String, Message>
> topicToMessage? It will help us to validate that the message truly belongs
> to the topic/partition.
>
> Thank you,
> Tarun.
>
> On Fri, 9 Sept 2022 at 19:55, Xiangying Meng <xi...@apache.org> wrote:
>
>> +1
>> This suggestion is really awesome.
>> The new solution can reduce the learning cost of Pulsar users and avoid
>> them taking many detours.
>> Yours,
>> Xiangying
>>
>> On Fri, Sep 9, 2022 at 8:05 PM Shivji Kumar Jha <sh...@gmail.com>
>> wrote:
>>
>>> Created https://github.com/apache/pulsar/issues/17574 for Yunze's
>>> suggestion on exposing the following in java client
>>>
>>> void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
>>>
>>>
>>> Regards,
>>> Shivji Kumar Jha
>>> http://www.shivjijha.com/
>>> +91 8884075512
>>>
>>>
>>> On Fri, 9 Sept 2022 at 11:44, Shivji Kumar Jha <sh...@gmail.com>
>> wrote:
>>>
>>>> Tarun is a colleague at Nutanix who is very eager to do his first patch
>>> in
>>>> an apache Project :-)
>>>>
>>>> Regards,
>>>> Shivji Kumar Jha
>>>> http://www.shivjijha.com/
>>>> +91 8884075512
>>>>
>>>>
>>>> On Wed, 7 Sept 2022 at 17:05, Yunze Xu <yz...@streamnative.io.invalid>
>>>> wrote:
>>>>
>>>>> Sure. I’m glad to see that. Just a little confused about who is Tarun?
>>>>>
>>>>> Thanks,
>>>>> Yunze
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>> On Sep 6, 2022, at 17:40, Shivji Kumar Jha <sh...@gmail.com>
>>> wrote:
>>>>>>
>>>>>> ++ Tarun
>>>>>>
>>>>>> Hi Yunze,
>>>>>>
>>>>>> We would love to have this.
>>>>>>
>>>>>> ```java
>>>>>> // the key is the partitioned topic name like my-topic-partition-0
>>>>>> void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
>>>>>> ```
>>>>>>
>>>>>> If you are busy with other things, do you mind Tarun taking this up
>> ?
>>>>> Happy
>>>>>> to have you as a reviewer.
>>>>>>
>>>>>> Regards,
>>>>>> Shivji Kumar Jha
>>>>>> http://www.shivjijha.com/
>>>>>> +91 8884075512
>>>>>>
>>>>>>
>>>>>> On Sun, 4 Sept 2022 at 21:25, Yunze Xu <yzxu@streamnative.io.invalid
>>>
>>>>> wrote:
>>>>>>
>>>>>>> I am busy on other things recently so there is no further update.
>> But
>>>>>>> I found there is already two methods to acknowledge multiple
>> messages
>>>>>>> in Java client.
>>>>>>>
>>>>>>> ```java
>>>>>>> void acknowledge(Messages<?> messages) throws
>>> PulsarClientException;
>>>>>>>
>>>>>>> void acknowledge(List<MessageId> messageIdList) throws
>>>>>>> PulsarClientException;
>>>>>>> ```
>>>>>>>
>>>>>>> And here is the issue to track the catch up:
>>>>>>>
>>>>>>> https://github.com/apache/pulsar/issues/17428
>>>>>>>
>>>>>>> Yunze
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Sep 4, 2022, at 22:37, Asaf Mesika <as...@gmail.com>
>>> wrote:
>>>>>>>>
>>>>>>>> What eventually happened with this idea?
>>>>>>>>
>>>>>>>> On Fri, Jul 29, 2022 at 8:02 AM PengHui Li <
>> codelipenghui@gmail.com
>>>>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> +1
>>>>>>>>>
>>>>>>>>> Penghui
>>>>>>>>> On Jul 28, 2022, 20:14 +0800, lordcheng10
>>> <1572139390@qq.com.invalid
>>>>>> ,
>>>>>>>>> wrote:
>>>>>>>>>> Nice feature!
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> ------------------ Original ------------------
>>>>>>>>>> From: "Yunze Xu"<yzxu@streamnative.io.INVALID>;
>>>>>>>>>> Date: 2022Äê7ÔÂ15ÈÕ(ÐÇÆÚÎå) ÍíÉÏ6:04
>>>>>>>>>> To: "dev"<dev@pulsar.apache.org>;
>>>>>>>>>> Subject: [DISCUSS] User-friendly acknowledgeCumulative API on a
>>>>>>>>> partitioned topic or multi-topics
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Long days ago I opened a PR to support cumulative
>> acknowledgement
>>>>>>>>>> for C++ client, but it's controversial about whether should a
>>>>>>>>>> partitioned consumer acknowledge a message ID cumulatively.
>>>>>>>>>>
>>>>>>>>>> See https://github.com/apache/pulsar/pull/6796 for discussion.
>>>>>>>>>>
>>>>>>>>>> Currently, the Java client acknowledges the specific partition
>> of
>>>>> the
>>>>>>>>>> message ID, while the C++ client just fails when calling
>>>>>>>>>> `acknowledgeCumulative` on a partitioned topic. However, even if
>>> the
>>>>>>>>>> Java client doesn't fail, it's not user friendly.
>>>>>>>>>>
>>>>>>>>>> Assuming users called `acknowledgeCumulative` periodically,
>> there
>>>>> is a
>>>>>>>>>> chance that some messages of the specific partition has never
>> been
>>>>>>>>>> passed to the method.
>>>>>>>>>>
>>>>>>>>>> For example, a consumer received:
>>>>>>>>>>
>>>>>>>>>> P0-M0, P1-M0, P0-M1, P1-M1, P0-M2, P1-M2...
>>>>>>>>>>
>>>>>>>>>> And the user acknowledged every two messages, i.e.
>>>>>>>>>>
>>>>>>>>>> P0-M0, P0-M1, P0-M2
>>>>>>>>>>
>>>>>>>>>> Eventually, partition 1 has never been acknowledged.
>>>>>>>>>>
>>>>>>>>>> User must maintain its own `Map<String, MessageId>` cache
>> for a
>>>>>>>>>> partitioned topic or multi-topics consumer with the existing
>>>>>>>>>> `acknowledgeCumulative` API.
>>>>>>>>>>
>>>>>>>>>> Should we make it more friendly for users? For example, we can
>>> make
>>>>>>>>>> `acknowledgeCumulative` accept the map to remind users to
>> maintain
>>>>>>>>>> the map from topic name to message ID:
>>>>>>>>>>
>>>>>>>>>> ```java
>>>>>>>>>> // the key is the partitioned topic name like
>> my-topic-partition-0
>>>>>>>>>> void acknowledgeCumulative(Map<String, MessageId>
>>>>> topicToMessageId);
>>>>>>>>>> ```
>>>>>>>>>>
>>>>>>>>>> For those who don't want to maintain the map by themselves,
>> maybe
>>> we
>>>>>>>>>> can provide a simpler API like:
>>>>>>>>>>
>>>>>>>>>> ```java
>>>>>>>>>> // acknowlegde all latest received messages
>>>>>>>>>> void acknowledgeCumulative();
>>>>>>>>>> ```
>>>>>>>>>>
>>>>>>>>>> and provide an option to enable this behavior.
>>>>>>>>>>
>>>>>>>>>> Do you have any suggestion on this idea? I will prepare a
>> proposal
>>>>> if
>>>>>>>>>> there is no disagreement.
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>> Yunze
>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>
>>>>>
>>>
>>
Re: [DISCUSS] User-friendly acknowledgeCumulative API on a partitioned topic or multi-topics
Posted by Tarun Annapareddy <ta...@gmail.com>.
Hi,
There is a concern about the validation of the topic string key in the
map we are using. In this case is it good to have Map<String, Message>
topicToMessage? It will help us to validate that the message truly belongs
to the topic/partition.
Thank you,
Tarun.
On Fri, 9 Sept 2022 at 19:55, Xiangying Meng <xi...@apache.org> wrote:
> +1
> This suggestion is really awesome.
> The new solution can reduce the learning cost of Pulsar users and avoid
> them taking many detours.
> Yours,
> Xiangying
>
> On Fri, Sep 9, 2022 at 8:05 PM Shivji Kumar Jha <sh...@gmail.com>
> wrote:
>
> > Created https://github.com/apache/pulsar/issues/17574 for Yunze's
> > suggestion on exposing the following in java client
> >
> > void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
> >
> >
> > Regards,
> > Shivji Kumar Jha
> > http://www.shivjijha.com/
> > +91 8884075512
> >
> >
> > On Fri, 9 Sept 2022 at 11:44, Shivji Kumar Jha <sh...@gmail.com>
> wrote:
> >
> > > Tarun is a colleague at Nutanix who is very eager to do his first patch
> > in
> > > an apache Project :-)
> > >
> > > Regards,
> > > Shivji Kumar Jha
> > > http://www.shivjijha.com/
> > > +91 8884075512
> > >
> > >
> > > On Wed, 7 Sept 2022 at 17:05, Yunze Xu <yz...@streamnative.io.invalid>
> > > wrote:
> > >
> > >> Sure. I’m glad to see that. Just a little confused about who is Tarun?
> > >>
> > >> Thanks,
> > >> Yunze
> > >>
> > >>
> > >>
> > >>
> > >> > On Sep 6, 2022, at 17:40, Shivji Kumar Jha <sh...@gmail.com>
> > wrote:
> > >> >
> > >> > ++ Tarun
> > >> >
> > >> > Hi Yunze,
> > >> >
> > >> > We would love to have this.
> > >> >
> > >> > ```java
> > >> > // the key is the partitioned topic name like my-topic-partition-0
> > >> > void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
> > >> > ```
> > >> >
> > >> > If you are busy with other things, do you mind Tarun taking this up
> ?
> > >> Happy
> > >> > to have you as a reviewer.
> > >> >
> > >> > Regards,
> > >> > Shivji Kumar Jha
> > >> > http://www.shivjijha.com/
> > >> > +91 8884075512
> > >> >
> > >> >
> > >> > On Sun, 4 Sept 2022 at 21:25, Yunze Xu <yzxu@streamnative.io.invalid
> >
> > >> wrote:
> > >> >
> > >> >> I am busy on other things recently so there is no further update.
> But
> > >> >> I found there is already two methods to acknowledge multiple
> messages
> > >> >> in Java client.
> > >> >>
> > >> >> ```java
> > >> >> void acknowledge(Messages<?> messages) throws
> > PulsarClientException;
> > >> >>
> > >> >> void acknowledge(List<MessageId> messageIdList) throws
> > >> >> PulsarClientException;
> > >> >> ```
> > >> >>
> > >> >> And here is the issue to track the catch up:
> > >> >>
> > >> >> https://github.com/apache/pulsar/issues/17428
> > >> >>
> > >> >> Yunze
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>> On Sep 4, 2022, at 22:37, Asaf Mesika <as...@gmail.com>
> > wrote:
> > >> >>>
> > >> >>> What eventually happened with this idea?
> > >> >>>
> > >> >>> On Fri, Jul 29, 2022 at 8:02 AM PengHui Li <
> codelipenghui@gmail.com
> > >
> > >> >> wrote:
> > >> >>>
> > >> >>>> +1
> > >> >>>>
> > >> >>>> Penghui
> > >> >>>> On Jul 28, 2022, 20:14 +0800, lordcheng10
> > <1572139390@qq.com.invalid
> > >> >,
> > >> >>>> wrote:
> > >> >>>>> Nice feature!
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> ------------------ Original ------------------
> > >> >>>>> From: "Yunze Xu"<yzxu@streamnative.io.INVALID>;
> > >> >>>>> Date: 2022Äê7ÔÂ15ÈÕ(ÐÇÆÚÎå) ÍíÉÏ6:04
> > >> >>>>> To: "dev"<dev@pulsar.apache.org>;
> > >> >>>>> Subject: [DISCUSS] User-friendly acknowledgeCumulative API on a
> > >> >>>> partitioned topic or multi-topics
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> Hi all,
> > >> >>>>>
> > >> >>>>> Long days ago I opened a PR to support cumulative
> acknowledgement
> > >> >>>>> for C++ client, but it's controversial about whether should a
> > >> >>>>> partitioned consumer acknowledge a message ID cumulatively.
> > >> >>>>>
> > >> >>>>> See https://github.com/apache/pulsar/pull/6796 for discussion.
> > >> >>>>>
> > >> >>>>> Currently, the Java client acknowledges the specific partition
> of
> > >> the
> > >> >>>>> message ID, while the C++ client just fails when calling
> > >> >>>>> `acknowledgeCumulative` on a partitioned topic. However, even if
> > the
> > >> >>>>> Java client doesn't fail, it's not user friendly.
> > >> >>>>>
> > >> >>>>> Assuming users called `acknowledgeCumulative` periodically,
> there
> > >> is a
> > >> >>>>> chance that some messages of the specific partition has never
> been
> > >> >>>>> passed to the method.
> > >> >>>>>
> > >> >>>>> For example, a consumer received:
> > >> >>>>>
> > >> >>>>> P0-M0, P1-M0, P0-M1, P1-M1, P0-M2, P1-M2...
> > >> >>>>>
> > >> >>>>> And the user acknowledged every two messages, i.e.
> > >> >>>>>
> > >> >>>>> P0-M0, P0-M1, P0-M2
> > >> >>>>>
> > >> >>>>> Eventually, partition 1 has never been acknowledged.
> > >> >>>>>
> > >> >>>>> User must maintain its own `Map<String, MessageId>` cache
> for a
> > >> >>>>> partitioned topic or multi-topics consumer with the existing
> > >> >>>>> `acknowledgeCumulative` API.
> > >> >>>>>
> > >> >>>>> Should we make it more friendly for users? For example, we can
> > make
> > >> >>>>> `acknowledgeCumulative` accept the map to remind users to
> maintain
> > >> >>>>> the map from topic name to message ID:
> > >> >>>>>
> > >> >>>>> ```java
> > >> >>>>> // the key is the partitioned topic name like
> my-topic-partition-0
> > >> >>>>> void acknowledgeCumulative(Map<String, MessageId>
> > >> topicToMessageId);
> > >> >>>>> ```
> > >> >>>>>
> > >> >>>>> For those who don't want to maintain the map by themselves,
> maybe
> > we
> > >> >>>>> can provide a simpler API like:
> > >> >>>>>
> > >> >>>>> ```java
> > >> >>>>> // acknowlegde all latest received messages
> > >> >>>>> void acknowledgeCumulative();
> > >> >>>>> ```
> > >> >>>>>
> > >> >>>>> and provide an option to enable this behavior.
> > >> >>>>>
> > >> >>>>> Do you have any suggestion on this idea? I will prepare a
> proposal
> > >> if
> > >> >>>>> there is no disagreement.
> > >> >>>>>
> > >> >>>>> Thanks,
> > >> >>>>> Yunze
> > >> >>>>
> > >> >>
> > >> >>
> > >>
> > >>
> >
>
Re: [DISCUSS] User-friendly acknowledgeCumulative API on a partitioned topic or multi-topics
Posted by Tarun Annapareddy <ta...@gmail.com>.
Hi,
I have created PR to support the below functionality, Please provide
feedback.
*Method:* void acknowledgeCumulative(Map<String, MessageId>
topicToMessageId);
*Pull Request:* https://github.com/apache/pulsar/pull/17577
Thanks & Regards
Tarun.
On Fri, 9 Sept 2022 at 19:55, Xiangying Meng <xi...@apache.org> wrote:
> +1
> This suggestion is really awesome.
> The new solution can reduce the learning cost of Pulsar users and avoid
> them taking many detours.
> Yours,
> Xiangying
>
> On Fri, Sep 9, 2022 at 8:05 PM Shivji Kumar Jha <sh...@gmail.com>
> wrote:
>
> > Created https://github.com/apache/pulsar/issues/17574 for Yunze's
> > suggestion on exposing the following in java client
> >
> > void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
> >
> >
> > Regards,
> > Shivji Kumar Jha
> > http://www.shivjijha.com/
> > +91 8884075512
> >
> >
> > On Fri, 9 Sept 2022 at 11:44, Shivji Kumar Jha <sh...@gmail.com>
> wrote:
> >
> > > Tarun is a colleague at Nutanix who is very eager to do his first patch
> > in
> > > an apache Project :-)
> > >
> > > Regards,
> > > Shivji Kumar Jha
> > > http://www.shivjijha.com/
> > > +91 8884075512
> > >
> > >
> > > On Wed, 7 Sept 2022 at 17:05, Yunze Xu <yz...@streamnative.io.invalid>
> > > wrote:
> > >
> > >> Sure. I’m glad to see that. Just a little confused about who is Tarun?
> > >>
> > >> Thanks,
> > >> Yunze
> > >>
> > >>
> > >>
> > >>
> > >> > On Sep 6, 2022, at 17:40, Shivji Kumar Jha <sh...@gmail.com>
> > wrote:
> > >> >
> > >> > ++ Tarun
> > >> >
> > >> > Hi Yunze,
> > >> >
> > >> > We would love to have this.
> > >> >
> > >> > ```java
> > >> > // the key is the partitioned topic name like my-topic-partition-0
> > >> > void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
> > >> > ```
> > >> >
> > >> > If you are busy with other things, do you mind Tarun taking this up
> ?
> > >> Happy
> > >> > to have you as a reviewer.
> > >> >
> > >> > Regards,
> > >> > Shivji Kumar Jha
> > >> > http://www.shivjijha.com/
> > >> > +91 8884075512
> > >> >
> > >> >
> > >> > On Sun, 4 Sept 2022 at 21:25, Yunze Xu <yzxu@streamnative.io.invalid
> >
> > >> wrote:
> > >> >
> > >> >> I am busy on other things recently so there is no further update.
> But
> > >> >> I found there is already two methods to acknowledge multiple
> messages
> > >> >> in Java client.
> > >> >>
> > >> >> ```java
> > >> >> void acknowledge(Messages<?> messages) throws
> > PulsarClientException;
> > >> >>
> > >> >> void acknowledge(List<MessageId> messageIdList) throws
> > >> >> PulsarClientException;
> > >> >> ```
> > >> >>
> > >> >> And here is the issue to track the catch up:
> > >> >>
> > >> >> https://github.com/apache/pulsar/issues/17428
> > >> >>
> > >> >> Yunze
> > >> >>
> > >> >>
> > >> >>
> > >> >>
> > >> >>> On Sep 4, 2022, at 22:37, Asaf Mesika <as...@gmail.com>
> > wrote:
> > >> >>>
> > >> >>> What eventually happened with this idea?
> > >> >>>
> > >> >>> On Fri, Jul 29, 2022 at 8:02 AM PengHui Li <
> codelipenghui@gmail.com
> > >
> > >> >> wrote:
> > >> >>>
> > >> >>>> +1
> > >> >>>>
> > >> >>>> Penghui
> > >> >>>> On Jul 28, 2022, 20:14 +0800, lordcheng10
> > <1572139390@qq.com.invalid
> > >> >,
> > >> >>>> wrote:
> > >> >>>>> Nice feature!
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> ------------------ Original ------------------
> > >> >>>>> From: "Yunze Xu"<yzxu@streamnative.io.INVALID>;
> > >> >>>>> Date: 2022Äê7ÔÂ15ÈÕ(ÐÇÆÚÎå) ÍíÉÏ6:04
> > >> >>>>> To: "dev"<dev@pulsar.apache.org>;
> > >> >>>>> Subject: [DISCUSS] User-friendly acknowledgeCumulative API on a
> > >> >>>> partitioned topic or multi-topics
> > >> >>>>>
> > >> >>>>>
> > >> >>>>>
> > >> >>>>> Hi all,
> > >> >>>>>
> > >> >>>>> Long days ago I opened a PR to support cumulative
> acknowledgement
> > >> >>>>> for C++ client, but it's controversial about whether should a
> > >> >>>>> partitioned consumer acknowledge a message ID cumulatively.
> > >> >>>>>
> > >> >>>>> See https://github.com/apache/pulsar/pull/6796 for discussion.
> > >> >>>>>
> > >> >>>>> Currently, the Java client acknowledges the specific partition
> of
> > >> the
> > >> >>>>> message ID, while the C++ client just fails when calling
> > >> >>>>> `acknowledgeCumulative` on a partitioned topic. However, even if
> > the
> > >> >>>>> Java client doesn't fail, it's not user friendly.
> > >> >>>>>
> > >> >>>>> Assuming users called `acknowledgeCumulative` periodically,
> there
> > >> is a
> > >> >>>>> chance that some messages of the specific partition has never
> been
> > >> >>>>> passed to the method.
> > >> >>>>>
> > >> >>>>> For example, a consumer received:
> > >> >>>>>
> > >> >>>>> P0-M0, P1-M0, P0-M1, P1-M1, P0-M2, P1-M2...
> > >> >>>>>
> > >> >>>>> And the user acknowledged every two messages, i.e.
> > >> >>>>>
> > >> >>>>> P0-M0, P0-M1, P0-M2
> > >> >>>>>
> > >> >>>>> Eventually, partition 1 has never been acknowledged.
> > >> >>>>>
> > >> >>>>> User must maintain its own `Map<String, MessageId>` cache
> for a
> > >> >>>>> partitioned topic or multi-topics consumer with the existing
> > >> >>>>> `acknowledgeCumulative` API.
> > >> >>>>>
> > >> >>>>> Should we make it more friendly for users? For example, we can
> > make
> > >> >>>>> `acknowledgeCumulative` accept the map to remind users to
> maintain
> > >> >>>>> the map from topic name to message ID:
> > >> >>>>>
> > >> >>>>> ```java
> > >> >>>>> // the key is the partitioned topic name like
> my-topic-partition-0
> > >> >>>>> void acknowledgeCumulative(Map<String, MessageId>
> > >> topicToMessageId);
> > >> >>>>> ```
> > >> >>>>>
> > >> >>>>> For those who don't want to maintain the map by themselves,
> maybe
> > we
> > >> >>>>> can provide a simpler API like:
> > >> >>>>>
> > >> >>>>> ```java
> > >> >>>>> // acknowlegde all latest received messages
> > >> >>>>> void acknowledgeCumulative();
> > >> >>>>> ```
> > >> >>>>>
> > >> >>>>> and provide an option to enable this behavior.
> > >> >>>>>
> > >> >>>>> Do you have any suggestion on this idea? I will prepare a
> proposal
> > >> if
> > >> >>>>> there is no disagreement.
> > >> >>>>>
> > >> >>>>> Thanks,
> > >> >>>>> Yunze
> > >> >>>>
> > >> >>
> > >> >>
> > >>
> > >>
> >
>
Re: [DISCUSS] User-friendly acknowledgeCumulative API on a partitioned topic or multi-topics
Posted by Xiangying Meng <xi...@apache.org>.
+1
This suggestion is really awesome.
The new solution can reduce the learning cost of Pulsar users and avoid
them taking many detours.
Yours,
Xiangying
On Fri, Sep 9, 2022 at 8:05 PM Shivji Kumar Jha <sh...@gmail.com> wrote:
> Created https://github.com/apache/pulsar/issues/17574 for Yunze's
> suggestion on exposing the following in java client
>
> void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
>
>
> Regards,
> Shivji Kumar Jha
> http://www.shivjijha.com/
> +91 8884075512
>
>
> On Fri, 9 Sept 2022 at 11:44, Shivji Kumar Jha <sh...@gmail.com> wrote:
>
> > Tarun is a colleague at Nutanix who is very eager to do his first patch
> in
> > an apache Project :-)
> >
> > Regards,
> > Shivji Kumar Jha
> > http://www.shivjijha.com/
> > +91 8884075512
> >
> >
> > On Wed, 7 Sept 2022 at 17:05, Yunze Xu <yz...@streamnative.io.invalid>
> > wrote:
> >
> >> Sure. I’m glad to see that. Just a little confused about who is Tarun?
> >>
> >> Thanks,
> >> Yunze
> >>
> >>
> >>
> >>
> >> > On Sep 6, 2022, at 17:40, Shivji Kumar Jha <sh...@gmail.com>
> wrote:
> >> >
> >> > ++ Tarun
> >> >
> >> > Hi Yunze,
> >> >
> >> > We would love to have this.
> >> >
> >> > ```java
> >> > // the key is the partitioned topic name like my-topic-partition-0
> >> > void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
> >> > ```
> >> >
> >> > If you are busy with other things, do you mind Tarun taking this up ?
> >> Happy
> >> > to have you as a reviewer.
> >> >
> >> > Regards,
> >> > Shivji Kumar Jha
> >> > http://www.shivjijha.com/
> >> > +91 8884075512
> >> >
> >> >
> >> > On Sun, 4 Sept 2022 at 21:25, Yunze Xu <yz...@streamnative.io.invalid>
> >> wrote:
> >> >
> >> >> I am busy on other things recently so there is no further update. But
> >> >> I found there is already two methods to acknowledge multiple messages
> >> >> in Java client.
> >> >>
> >> >> ```java
> >> >> void acknowledge(Messages<?> messages) throws
> PulsarClientException;
> >> >>
> >> >> void acknowledge(List<MessageId> messageIdList) throws
> >> >> PulsarClientException;
> >> >> ```
> >> >>
> >> >> And here is the issue to track the catch up:
> >> >>
> >> >> https://github.com/apache/pulsar/issues/17428
> >> >>
> >> >> Yunze
> >> >>
> >> >>
> >> >>
> >> >>
> >> >>> On Sep 4, 2022, at 22:37, Asaf Mesika <as...@gmail.com>
> wrote:
> >> >>>
> >> >>> What eventually happened with this idea?
> >> >>>
> >> >>> On Fri, Jul 29, 2022 at 8:02 AM PengHui Li <codelipenghui@gmail.com
> >
> >> >> wrote:
> >> >>>
> >> >>>> +1
> >> >>>>
> >> >>>> Penghui
> >> >>>> On Jul 28, 2022, 20:14 +0800, lordcheng10
> <1572139390@qq.com.invalid
> >> >,
> >> >>>> wrote:
> >> >>>>> Nice feature!
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------ Original ------------------
> >> >>>>> From: "Yunze Xu"<yzxu@streamnative.io.INVALID>;
> >> >>>>> Date: 2022Äê7ÔÂ15ÈÕ(ÐÇÆÚÎå) ÍíÉÏ6:04
> >> >>>>> To: "dev"<dev@pulsar.apache.org>;
> >> >>>>> Subject: [DISCUSS] User-friendly acknowledgeCumulative API on a
> >> >>>> partitioned topic or multi-topics
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Hi all,
> >> >>>>>
> >> >>>>> Long days ago I opened a PR to support cumulative acknowledgement
> >> >>>>> for C++ client, but it's controversial about whether should a
> >> >>>>> partitioned consumer acknowledge a message ID cumulatively.
> >> >>>>>
> >> >>>>> See https://github.com/apache/pulsar/pull/6796 for discussion.
> >> >>>>>
> >> >>>>> Currently, the Java client acknowledges the specific partition of
> >> the
> >> >>>>> message ID, while the C++ client just fails when calling
> >> >>>>> `acknowledgeCumulative` on a partitioned topic. However, even if
> the
> >> >>>>> Java client doesn't fail, it's not user friendly.
> >> >>>>>
> >> >>>>> Assuming users called `acknowledgeCumulative` periodically, there
> >> is a
> >> >>>>> chance that some messages of the specific partition has never been
> >> >>>>> passed to the method.
> >> >>>>>
> >> >>>>> For example, a consumer received:
> >> >>>>>
> >> >>>>> P0-M0, P1-M0, P0-M1, P1-M1, P0-M2, P1-M2...
> >> >>>>>
> >> >>>>> And the user acknowledged every two messages, i.e.
> >> >>>>>
> >> >>>>> P0-M0, P0-M1, P0-M2
> >> >>>>>
> >> >>>>> Eventually, partition 1 has never been acknowledged.
> >> >>>>>
> >> >>>>> User must maintain its own `Map<String, MessageId>` cache for a
> >> >>>>> partitioned topic or multi-topics consumer with the existing
> >> >>>>> `acknowledgeCumulative` API.
> >> >>>>>
> >> >>>>> Should we make it more friendly for users? For example, we can
> make
> >> >>>>> `acknowledgeCumulative` accept the map to remind users to maintain
> >> >>>>> the map from topic name to message ID:
> >> >>>>>
> >> >>>>> ```java
> >> >>>>> // the key is the partitioned topic name like my-topic-partition-0
> >> >>>>> void acknowledgeCumulative(Map<String, MessageId>
> >> topicToMessageId);
> >> >>>>> ```
> >> >>>>>
> >> >>>>> For those who don't want to maintain the map by themselves, maybe
> we
> >> >>>>> can provide a simpler API like:
> >> >>>>>
> >> >>>>> ```java
> >> >>>>> // acknowlegde all latest received messages
> >> >>>>> void acknowledgeCumulative();
> >> >>>>> ```
> >> >>>>>
> >> >>>>> and provide an option to enable this behavior.
> >> >>>>>
> >> >>>>> Do you have any suggestion on this idea? I will prepare a proposal
> >> if
> >> >>>>> there is no disagreement.
> >> >>>>>
> >> >>>>> Thanks,
> >> >>>>> Yunze
> >> >>>>
> >> >>
> >> >>
> >>
> >>
>
Re: [DISCUSS] User-friendly acknowledgeCumulative API on a partitioned topic or multi-topics
Posted by Shivji Kumar Jha <sh...@gmail.com>.
Created https://github.com/apache/pulsar/issues/17574 for Yunze's
suggestion on exposing the following in java client
void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
Regards,
Shivji Kumar Jha
http://www.shivjijha.com/
+91 8884075512
On Fri, 9 Sept 2022 at 11:44, Shivji Kumar Jha <sh...@gmail.com> wrote:
> Tarun is a colleague at Nutanix who is very eager to do his first patch in
> an apache Project :-)
>
> Regards,
> Shivji Kumar Jha
> http://www.shivjijha.com/
> +91 8884075512
>
>
> On Wed, 7 Sept 2022 at 17:05, Yunze Xu <yz...@streamnative.io.invalid>
> wrote:
>
>> Sure. I’m glad to see that. Just a little confused about who is Tarun?
>>
>> Thanks,
>> Yunze
>>
>>
>>
>>
>> > On Sep 6, 2022, at 17:40, Shivji Kumar Jha <sh...@gmail.com> wrote:
>> >
>> > ++ Tarun
>> >
>> > Hi Yunze,
>> >
>> > We would love to have this.
>> >
>> > ```java
>> > // the key is the partitioned topic name like my-topic-partition-0
>> > void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
>> > ```
>> >
>> > If you are busy with other things, do you mind Tarun taking this up ?
>> Happy
>> > to have you as a reviewer.
>> >
>> > Regards,
>> > Shivji Kumar Jha
>> > http://www.shivjijha.com/
>> > +91 8884075512
>> >
>> >
>> > On Sun, 4 Sept 2022 at 21:25, Yunze Xu <yz...@streamnative.io.invalid>
>> wrote:
>> >
>> >> I am busy on other things recently so there is no further update. But
>> >> I found there is already two methods to acknowledge multiple messages
>> >> in Java client.
>> >>
>> >> ```java
>> >> void acknowledge(Messages<?> messages) throws PulsarClientException;
>> >>
>> >> void acknowledge(List<MessageId> messageIdList) throws
>> >> PulsarClientException;
>> >> ```
>> >>
>> >> And here is the issue to track the catch up:
>> >>
>> >> https://github.com/apache/pulsar/issues/17428
>> >>
>> >> Yunze
>> >>
>> >>
>> >>
>> >>
>> >>> On Sep 4, 2022, at 22:37, Asaf Mesika <as...@gmail.com> wrote:
>> >>>
>> >>> What eventually happened with this idea?
>> >>>
>> >>> On Fri, Jul 29, 2022 at 8:02 AM PengHui Li <co...@gmail.com>
>> >> wrote:
>> >>>
>> >>>> +1
>> >>>>
>> >>>> Penghui
>> >>>> On Jul 28, 2022, 20:14 +0800, lordcheng10 <1572139390@qq.com.invalid
>> >,
>> >>>> wrote:
>> >>>>> Nice feature!
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> ------------------ Original ------------------
>> >>>>> From: "Yunze Xu"<yzxu@streamnative.io.INVALID>;
>> >>>>> Date: 2022Äê7ÔÂ15ÈÕ(ÐÇÆÚÎå) ÍíÉÏ6:04
>> >>>>> To: "dev"<dev@pulsar.apache.org>;
>> >>>>> Subject: [DISCUSS] User-friendly acknowledgeCumulative API on a
>> >>>> partitioned topic or multi-topics
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> Hi all,
>> >>>>>
>> >>>>> Long days ago I opened a PR to support cumulative acknowledgement
>> >>>>> for C++ client, but it's controversial about whether should a
>> >>>>> partitioned consumer acknowledge a message ID cumulatively.
>> >>>>>
>> >>>>> See https://github.com/apache/pulsar/pull/6796 for discussion.
>> >>>>>
>> >>>>> Currently, the Java client acknowledges the specific partition of
>> the
>> >>>>> message ID, while the C++ client just fails when calling
>> >>>>> `acknowledgeCumulative` on a partitioned topic. However, even if the
>> >>>>> Java client doesn't fail, it's not user friendly.
>> >>>>>
>> >>>>> Assuming users called `acknowledgeCumulative` periodically, there
>> is a
>> >>>>> chance that some messages of the specific partition has never been
>> >>>>> passed to the method.
>> >>>>>
>> >>>>> For example, a consumer received:
>> >>>>>
>> >>>>> P0-M0, P1-M0, P0-M1, P1-M1, P0-M2, P1-M2...
>> >>>>>
>> >>>>> And the user acknowledged every two messages, i.e.
>> >>>>>
>> >>>>> P0-M0, P0-M1, P0-M2
>> >>>>>
>> >>>>> Eventually, partition 1 has never been acknowledged.
>> >>>>>
>> >>>>> User must maintain its own `Map<String, MessageId>` cache for a
>> >>>>> partitioned topic or multi-topics consumer with the existing
>> >>>>> `acknowledgeCumulative` API.
>> >>>>>
>> >>>>> Should we make it more friendly for users? For example, we can make
>> >>>>> `acknowledgeCumulative` accept the map to remind users to maintain
>> >>>>> the map from topic name to message ID:
>> >>>>>
>> >>>>> ```java
>> >>>>> // the key is the partitioned topic name like my-topic-partition-0
>> >>>>> void acknowledgeCumulative(Map<String, MessageId>
>> topicToMessageId);
>> >>>>> ```
>> >>>>>
>> >>>>> For those who don't want to maintain the map by themselves, maybe we
>> >>>>> can provide a simpler API like:
>> >>>>>
>> >>>>> ```java
>> >>>>> // acknowlegde all latest received messages
>> >>>>> void acknowledgeCumulative();
>> >>>>> ```
>> >>>>>
>> >>>>> and provide an option to enable this behavior.
>> >>>>>
>> >>>>> Do you have any suggestion on this idea? I will prepare a proposal
>> if
>> >>>>> there is no disagreement.
>> >>>>>
>> >>>>> Thanks,
>> >>>>> Yunze
>> >>>>
>> >>
>> >>
>>
>>
Re: [DISCUSS] User-friendly acknowledgeCumulative API on a partitioned topic or multi-topics
Posted by Shivji Kumar Jha <sh...@gmail.com>.
Tarun is a colleague at Nutanix who is very eager to do his first patch in
an apache Project :-)
Regards,
Shivji Kumar Jha
http://www.shivjijha.com/
+91 8884075512
On Wed, 7 Sept 2022 at 17:05, Yunze Xu <yz...@streamnative.io.invalid> wrote:
> Sure. I’m glad to see that. Just a little confused about who is Tarun?
>
> Thanks,
> Yunze
>
>
>
>
> > On Sep 6, 2022, at 17:40, Shivji Kumar Jha <sh...@gmail.com> wrote:
> >
> > ++ Tarun
> >
> > Hi Yunze,
> >
> > We would love to have this.
> >
> > ```java
> > // the key is the partitioned topic name like my-topic-partition-0
> > void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
> > ```
> >
> > If you are busy with other things, do you mind Tarun taking this up ?
> Happy
> > to have you as a reviewer.
> >
> > Regards,
> > Shivji Kumar Jha
> > http://www.shivjijha.com/
> > +91 8884075512
> >
> >
> > On Sun, 4 Sept 2022 at 21:25, Yunze Xu <yz...@streamnative.io.invalid>
> wrote:
> >
> >> I am busy on other things recently so there is no further update. But
> >> I found there is already two methods to acknowledge multiple messages
> >> in Java client.
> >>
> >> ```java
> >> void acknowledge(Messages<?> messages) throws PulsarClientException;
> >>
> >> void acknowledge(List<MessageId> messageIdList) throws
> >> PulsarClientException;
> >> ```
> >>
> >> And here is the issue to track the catch up:
> >>
> >> https://github.com/apache/pulsar/issues/17428
> >>
> >> Yunze
> >>
> >>
> >>
> >>
> >>> On Sep 4, 2022, at 22:37, Asaf Mesika <as...@gmail.com> wrote:
> >>>
> >>> What eventually happened with this idea?
> >>>
> >>> On Fri, Jul 29, 2022 at 8:02 AM PengHui Li <co...@gmail.com>
> >> wrote:
> >>>
> >>>> +1
> >>>>
> >>>> Penghui
> >>>> On Jul 28, 2022, 20:14 +0800, lordcheng10 <1572139390@qq.com.invalid
> >,
> >>>> wrote:
> >>>>> Nice feature!
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> ------------------ Original ------------------
> >>>>> From: "Yunze Xu"<yzxu@streamnative.io.INVALID>;
> >>>>> Date: 2022Äê7ÔÂ15ÈÕ(ÐÇÆÚÎå) ÍíÉÏ6:04
> >>>>> To: "dev"<dev@pulsar.apache.org>;
> >>>>> Subject: [DISCUSS] User-friendly acknowledgeCumulative API on a
> >>>> partitioned topic or multi-topics
> >>>>>
> >>>>>
> >>>>>
> >>>>> Hi all,
> >>>>>
> >>>>> Long days ago I opened a PR to support cumulative acknowledgement
> >>>>> for C++ client, but it's controversial about whether should a
> >>>>> partitioned consumer acknowledge a message ID cumulatively.
> >>>>>
> >>>>> See https://github.com/apache/pulsar/pull/6796 for discussion.
> >>>>>
> >>>>> Currently, the Java client acknowledges the specific partition of the
> >>>>> message ID, while the C++ client just fails when calling
> >>>>> `acknowledgeCumulative` on a partitioned topic. However, even if the
> >>>>> Java client doesn't fail, it's not user friendly.
> >>>>>
> >>>>> Assuming users called `acknowledgeCumulative` periodically, there is
> a
> >>>>> chance that some messages of the specific partition has never been
> >>>>> passed to the method.
> >>>>>
> >>>>> For example, a consumer received:
> >>>>>
> >>>>> P0-M0, P1-M0, P0-M1, P1-M1, P0-M2, P1-M2...
> >>>>>
> >>>>> And the user acknowledged every two messages, i.e.
> >>>>>
> >>>>> P0-M0, P0-M1, P0-M2
> >>>>>
> >>>>> Eventually, partition 1 has never been acknowledged.
> >>>>>
> >>>>> User must maintain its own `Map<String, MessageId>` cache for a
> >>>>> partitioned topic or multi-topics consumer with the existing
> >>>>> `acknowledgeCumulative` API.
> >>>>>
> >>>>> Should we make it more friendly for users? For example, we can make
> >>>>> `acknowledgeCumulative` accept the map to remind users to maintain
> >>>>> the map from topic name to message ID:
> >>>>>
> >>>>> ```java
> >>>>> // the key is the partitioned topic name like my-topic-partition-0
> >>>>> void acknowledgeCumulative(Map<String, MessageId>
> topicToMessageId);
> >>>>> ```
> >>>>>
> >>>>> For those who don't want to maintain the map by themselves, maybe we
> >>>>> can provide a simpler API like:
> >>>>>
> >>>>> ```java
> >>>>> // acknowlegde all latest received messages
> >>>>> void acknowledgeCumulative();
> >>>>> ```
> >>>>>
> >>>>> and provide an option to enable this behavior.
> >>>>>
> >>>>> Do you have any suggestion on this idea? I will prepare a proposal if
> >>>>> there is no disagreement.
> >>>>>
> >>>>> Thanks,
> >>>>> Yunze
> >>>>
> >>
> >>
>
>
Re: [DISCUSS] User-friendly acknowledgeCumulative API on a partitioned topic or multi-topics
Posted by Yunze Xu <yz...@streamnative.io.INVALID>.
Sure. I’m glad to see that. Just a little confused about who is Tarun?
Thanks,
Yunze
> On Sep 6, 2022, at 17:40, Shivji Kumar Jha <sh...@gmail.com> wrote:
>
> ++ Tarun
>
> Hi Yunze,
>
> We would love to have this.
>
> ```java
> // the key is the partitioned topic name like my-topic-partition-0
> void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
> ```
>
> If you are busy with other things, do you mind Tarun taking this up ? Happy
> to have you as a reviewer.
>
> Regards,
> Shivji Kumar Jha
> http://www.shivjijha.com/
> +91 8884075512
>
>
> On Sun, 4 Sept 2022 at 21:25, Yunze Xu <yz...@streamnative.io.invalid> wrote:
>
>> I am busy on other things recently so there is no further update. But
>> I found there is already two methods to acknowledge multiple messages
>> in Java client.
>>
>> ```java
>> void acknowledge(Messages<?> messages) throws PulsarClientException;
>>
>> void acknowledge(List<MessageId> messageIdList) throws
>> PulsarClientException;
>> ```
>>
>> And here is the issue to track the catch up:
>>
>> https://github.com/apache/pulsar/issues/17428
>>
>> Yunze
>>
>>
>>
>>
>>> On Sep 4, 2022, at 22:37, Asaf Mesika <as...@gmail.com> wrote:
>>>
>>> What eventually happened with this idea?
>>>
>>> On Fri, Jul 29, 2022 at 8:02 AM PengHui Li <co...@gmail.com>
>> wrote:
>>>
>>>> +1
>>>>
>>>> Penghui
>>>> On Jul 28, 2022, 20:14 +0800, lordcheng10 <15...@qq.com.invalid>,
>>>> wrote:
>>>>> Nice feature!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ------------------ Original ------------------
>>>>> From: "Yunze Xu"<yzxu@streamnative.io.INVALID>;
>>>>> Date: 2022Äê7ÔÂ15ÈÕ(ÐÇÆÚÎå) ÍíÉÏ6:04
>>>>> To: "dev"<dev@pulsar.apache.org>;
>>>>> Subject: [DISCUSS] User-friendly acknowledgeCumulative API on a
>>>> partitioned topic or multi-topics
>>>>>
>>>>>
>>>>>
>>>>> Hi all,
>>>>>
>>>>> Long days ago I opened a PR to support cumulative acknowledgement
>>>>> for C++ client, but it's controversial about whether should a
>>>>> partitioned consumer acknowledge a message ID cumulatively.
>>>>>
>>>>> See https://github.com/apache/pulsar/pull/6796 for discussion.
>>>>>
>>>>> Currently, the Java client acknowledges the specific partition of the
>>>>> message ID, while the C++ client just fails when calling
>>>>> `acknowledgeCumulative` on a partitioned topic. However, even if the
>>>>> Java client doesn't fail, it's not user friendly.
>>>>>
>>>>> Assuming users called `acknowledgeCumulative` periodically, there is a
>>>>> chance that some messages of the specific partition has never been
>>>>> passed to the method.
>>>>>
>>>>> For example, a consumer received:
>>>>>
>>>>> P0-M0, P1-M0, P0-M1, P1-M1, P0-M2, P1-M2...
>>>>>
>>>>> And the user acknowledged every two messages, i.e.
>>>>>
>>>>> P0-M0, P0-M1, P0-M2
>>>>>
>>>>> Eventually, partition 1 has never been acknowledged.
>>>>>
>>>>> User must maintain its own `Map<String, MessageId>` cache for a
>>>>> partitioned topic or multi-topics consumer with the existing
>>>>> `acknowledgeCumulative` API.
>>>>>
>>>>> Should we make it more friendly for users? For example, we can make
>>>>> `acknowledgeCumulative` accept the map to remind users to maintain
>>>>> the map from topic name to message ID:
>>>>>
>>>>> ```java
>>>>> // the key is the partitioned topic name like my-topic-partition-0
>>>>> void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
>>>>> ```
>>>>>
>>>>> For those who don't want to maintain the map by themselves, maybe we
>>>>> can provide a simpler API like:
>>>>>
>>>>> ```java
>>>>> // acknowlegde all latest received messages
>>>>> void acknowledgeCumulative();
>>>>> ```
>>>>>
>>>>> and provide an option to enable this behavior.
>>>>>
>>>>> Do you have any suggestion on this idea? I will prepare a proposal if
>>>>> there is no disagreement.
>>>>>
>>>>> Thanks,
>>>>> Yunze
>>>>
>>
>>
Re: [DISCUSS] User-friendly acknowledgeCumulative API on a partitioned topic or multi-topics
Posted by Shivji Kumar Jha <sh...@gmail.com>.
++ Tarun
Hi Yunze,
We would love to have this.
```java
// the key is the partitioned topic name like my-topic-partition-0
void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
```
If you are busy with other things, do you mind Tarun taking this up ? Happy
to have you as a reviewer.
Regards,
Shivji Kumar Jha
http://www.shivjijha.com/
+91 8884075512
On Sun, 4 Sept 2022 at 21:25, Yunze Xu <yz...@streamnative.io.invalid> wrote:
> I am busy on other things recently so there is no further update. But
> I found there is already two methods to acknowledge multiple messages
> in Java client.
>
> ```java
> void acknowledge(Messages<?> messages) throws PulsarClientException;
>
> void acknowledge(List<MessageId> messageIdList) throws
> PulsarClientException;
> ```
>
> And here is the issue to track the catch up:
>
> https://github.com/apache/pulsar/issues/17428
>
> Yunze
>
>
>
>
> > On Sep 4, 2022, at 22:37, Asaf Mesika <as...@gmail.com> wrote:
> >
> > What eventually happened with this idea?
> >
> > On Fri, Jul 29, 2022 at 8:02 AM PengHui Li <co...@gmail.com>
> wrote:
> >
> >> +1
> >>
> >> Penghui
> >> On Jul 28, 2022, 20:14 +0800, lordcheng10 <15...@qq.com.invalid>,
> >> wrote:
> >>> Nice feature!
> >>>
> >>>
> >>>
> >>>
> >>> ------------------ Original ------------------
> >>> From: "Yunze Xu"<yzxu@streamnative.io.INVALID>;
> >>> Date: 2022Äê7ÔÂ15ÈÕ(ÐÇÆÚÎå) ÍíÉÏ6:04
> >>> To: "dev"<dev@pulsar.apache.org>;
> >>> Subject: [DISCUSS] User-friendly acknowledgeCumulative API on a
> >> partitioned topic or multi-topics
> >>>
> >>>
> >>>
> >>> Hi all,
> >>>
> >>> Long days ago I opened a PR to support cumulative acknowledgement
> >>> for C++ client, but it's controversial about whether should a
> >>> partitioned consumer acknowledge a message ID cumulatively.
> >>>
> >>> See https://github.com/apache/pulsar/pull/6796 for discussion.
> >>>
> >>> Currently, the Java client acknowledges the specific partition of the
> >>> message ID, while the C++ client just fails when calling
> >>> `acknowledgeCumulative` on a partitioned topic. However, even if the
> >>> Java client doesn't fail, it's not user friendly.
> >>>
> >>> Assuming users called `acknowledgeCumulative` periodically, there is a
> >>> chance that some messages of the specific partition has never been
> >>> passed to the method.
> >>>
> >>> For example, a consumer received:
> >>>
> >>> P0-M0, P1-M0, P0-M1, P1-M1, P0-M2, P1-M2...
> >>>
> >>> And the user acknowledged every two messages, i.e.
> >>>
> >>> P0-M0, P0-M1, P0-M2
> >>>
> >>> Eventually, partition 1 has never been acknowledged.
> >>>
> >>> User must maintain its own `Map<String, MessageId>` cache for a
> >>> partitioned topic or multi-topics consumer with the existing
> >>> `acknowledgeCumulative` API.
> >>>
> >>> Should we make it more friendly for users? For example, we can make
> >>> `acknowledgeCumulative` accept the map to remind users to maintain
> >>> the map from topic name to message ID:
> >>>
> >>> ```java
> >>> // the key is the partitioned topic name like my-topic-partition-0
> >>> void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
> >>> ```
> >>>
> >>> For those who don't want to maintain the map by themselves, maybe we
> >>> can provide a simpler API like:
> >>>
> >>> ```java
> >>> // acknowlegde all latest received messages
> >>> void acknowledgeCumulative();
> >>> ```
> >>>
> >>> and provide an option to enable this behavior.
> >>>
> >>> Do you have any suggestion on this idea? I will prepare a proposal if
> >>> there is no disagreement.
> >>>
> >>> Thanks,
> >>> Yunze
> >>
>
>
Re: [DISCUSS] User-friendly acknowledgeCumulative API on a partitioned topic or multi-topics
Posted by Yunze Xu <yz...@streamnative.io.INVALID>.
I am busy on other things recently so there is no further update. But
I found there is already two methods to acknowledge multiple messages
in Java client.
```java
void acknowledge(Messages<?> messages) throws PulsarClientException;
void acknowledge(List<MessageId> messageIdList) throws PulsarClientException;
```
And here is the issue to track the catch up:
https://github.com/apache/pulsar/issues/17428
Yunze
> On Sep 4, 2022, at 22:37, Asaf Mesika <as...@gmail.com> wrote:
>
> What eventually happened with this idea?
>
> On Fri, Jul 29, 2022 at 8:02 AM PengHui Li <co...@gmail.com> wrote:
>
>> +1
>>
>> Penghui
>> On Jul 28, 2022, 20:14 +0800, lordcheng10 <15...@qq.com.invalid>,
>> wrote:
>>> Nice feature!
>>>
>>>
>>>
>>>
>>> ------------------ Original ------------------
>>> From: "Yunze Xu"<yzxu@streamnative.io.INVALID>;
>>> Date: 2022Äê7ÔÂ15ÈÕ(ÐÇÆÚÎå) ÍíÉÏ6:04
>>> To: "dev"<dev@pulsar.apache.org>;
>>> Subject: [DISCUSS] User-friendly acknowledgeCumulative API on a
>> partitioned topic or multi-topics
>>>
>>>
>>>
>>> Hi all,
>>>
>>> Long days ago I opened a PR to support cumulative acknowledgement
>>> for C++ client, but it's controversial about whether should a
>>> partitioned consumer acknowledge a message ID cumulatively.
>>>
>>> See https://github.com/apache/pulsar/pull/6796 for discussion.
>>>
>>> Currently, the Java client acknowledges the specific partition of the
>>> message ID, while the C++ client just fails when calling
>>> `acknowledgeCumulative` on a partitioned topic. However, even if the
>>> Java client doesn't fail, it's not user friendly.
>>>
>>> Assuming users called `acknowledgeCumulative` periodically, there is a
>>> chance that some messages of the specific partition has never been
>>> passed to the method.
>>>
>>> For example, a consumer received:
>>>
>>> P0-M0, P1-M0, P0-M1, P1-M1, P0-M2, P1-M2...
>>>
>>> And the user acknowledged every two messages, i.e.
>>>
>>> P0-M0, P0-M1, P0-M2
>>>
>>> Eventually, partition 1 has never been acknowledged.
>>>
>>> User must maintain its own `Map<String, MessageId>` cache for a
>>> partitioned topic or multi-topics consumer with the existing
>>> `acknowledgeCumulative` API.
>>>
>>> Should we make it more friendly for users? For example, we can make
>>> `acknowledgeCumulative` accept the map to remind users to maintain
>>> the map from topic name to message ID:
>>>
>>> ```java
>>> // the key is the partitioned topic name like my-topic-partition-0
>>> void acknowledgeCumulative(Map<String, MessageId> topicToMessageId);
>>> ```
>>>
>>> For those who don't want to maintain the map by themselves, maybe we
>>> can provide a simpler API like:
>>>
>>> ```java
>>> // acknowlegde all latest received messages
>>> void acknowledgeCumulative();
>>> ```
>>>
>>> and provide an option to enable this behavior.
>>>
>>> Do you have any suggestion on this idea? I will prepare a proposal if
>>> there is no disagreement.
>>>
>>> Thanks,
>>> Yunze
>>