You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pulsar.apache.org by Tarun Annapareddy <ta...@gmail.com> on 2022/09/13 10:14:46 UTC
PIP-Java Client support to Cumulative Acknowledge messages for multiple partitions or topics
Hi devs,
This is the official thread to vote for Java Client support to
Cumulative Acknowledge messages for multiple partitions or topics
PIP Issue: https://github.com/apache/pulsar/issues/17574
Discussion thread:
https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
PR: https://github.com/apache/pulsar/pull/17577
Thank you,
Tarun.
Re: PIP-Java Client support to Cumulative Acknowledge messages for multiple partitions or topics
Posted by Enrico Olivelli <eo...@gmail.com>.
Penghui,
Il giorno ven 21 ott 2022 alle ore 02:04 PengHui Li
<pe...@apache.org> ha scritto:
>
> > while (true) {
> Messages<String> messages = consumer.batchReceive();
> Message<String> finalMessage = null;
> for (Message<String> message : messages) {
> process(message);
> finalMessage = message;
> }
> if (finalMessage != null) {
> consumer.acknowledgeCumulativeAsync(finalMessage);
> }
> }
> ```
>
> I think this could be a good idea for improving the user's experience of
> cumulative ack.
> For one batch receive operation, users can only receive messages from one
> partition/topic.
> The cumulative ack becomes more assured and understandable.
I can agree on this idea, but we must state it cleanly in the javadocs
and we cannot change the behaviour in the future.
This proposal is backward compatible so we could do it but if in the
future we want to go back to
2.10-like behaviour it won't be possible anymore.
It is an important choice, because we are going to change the meaning
of an existing
API.
The very best way is to have a new method, this way no one will have
surprises, no and in the future.
>
> For the performance issue. The worst case is degenerate into the single
> message receiving.
> Users are mostly using the single message receiving for now.
>
> Regarding the issue of batch receive itself, I think we should discuss it
> separately since currently, we are
> split the batch first and then create a new list for users with the split
> messages. This can be improved
> if we do some optimization for this part.
Sure
Enrico
>
> Thanks,
> Penghui
>
> On Thu, Oct 20, 2022 at 7:14 PM 丛搏 <bo...@apache.org> wrote:
>
> > Hi, Enrico
> > I think we temporarily don't add the new API, it will make the users
> > very hard to use. we can try to improve the performance first.
> >
> > the performance can improve:
> > 1. create multi consumer clients to consume the multi-partition topic.
> > 2. change the MultiTopicsConsumerImpl incoming queue, using
> > ConsumerImpl.batchReceive() directly when use
> > MultiTopicsConsumerImpl.batchReceive()
> >
> > Thanks,
> > bo
> >
> > Enrico Olivelli <eo...@gmail.com> 于2022年10月20日周四 17:37写道：
> > >
> > > Il giorno gio 20 ott 2022 alle ore 11:09 丛搏 <bo...@apache.org> ha
> > scritto:
> > > >
> > > > Hi,
> > > >
> > > > I have another solution, we only guarantee the invoke batch receive
> > > > returns the messages are the same topic message.
> > >
> > > This is going to reduce performances of applications that consume from
> > > a partitioned topic or using a pattern or multiple topics, the
> > > application will receive smaller batches.
> > > Also, in the current API you don't have this assumption.
> > >
> > > I think that if we want to go this way we have to add a new
> > > batchReceive API that guarantees that all the messages are from the
> > > same message
> > >
> > > batchReceiveSingleTopic()
> > >
> > > Enrico
> > >
> > >
> > > In this way, we don't
> > > > need to add the new API for ack. use example is:
> > > > ```
> > > > // consumer the multi-partition messages
> > > > while (true) {
> > > > Messages<String> messages = consumer.batchReceive();
> > > > Message<String> finalMessage = null;
> > > > for (Message<String> message : messages) {
> > > > process(message);
> > > > finalMessage = message;
> > > > }
> > > > if (finalMessage != null) {
> > > > consumer.acknowledgeCumulativeAsync(finalMessage);
> > > > }
> > > > }
> > > > ```
> > > > implementation pr: https://github.com/congbobo184/pulsar/pull/5
> > > >
> > > > Thanks,
> > > > bo
> > > >
> > > > Tarun Annapareddy <ta...@gmail.com> 于2022年9月14日周三
> > 17:46写道：
> > > > >
> > > > > Hi Enrico,
> > > > > I can understand the concern about the validation of the topic
> > > > > string key in the map we are using. In this case is it good to have
> > > > > Map<String, Message> topicToMessage? It will help us to validate
> > that the
> > > > > message truly belongs to the topic/partition.
> > > > >
> > > > > Thank you,
> > > > > Tarun.
> > > > >
> > > > > On Wed, 14 Sept 2022 at 07:50, Yunze Xu <yzxu@streamnative.io.invalid
> > >
> > > > > wrote:
> > > > >
> > > > > > Hi Enrico,
> > > > > >
> > > > > > > There is no way to confirm that the MessageId really belongs to
> > the
> > > > > > > partition and you can pass whatever you want
> > > > > >
> > > > > > True. But still, the existing `acknowledgeCumulative` API is not
> > > > > > convenient to use. The original purpose for this new overload is
> > > > > > allowing users to maintain each partition's latest message ID to
> > avoid
> > > > > > frequent ACK commands, then they can flush all these message IDs
> > once.
> > > > > >
> > > > > > Though the keys are actually not used, a natural way is to
> > maintain a
> > > > > > map, not a list or set. For a multi-topics consumer, we can also
> > > > > > add a constraint that the key must exist in an internal consumer.
> > For
> > > > > > a single topic consumer, the key must be the topic name.
> > > > > >
> > > > > > Without the new API, users have to configure
> > `acknowledgmentGroupTime`
> > > > > > and call `acknowledgeCumulative` for each message received.
> > However,
> > > > > > the ACK grouping tracker is very simple that it can only configure
> > the
> > > > > > grouping timeout. I think users need a way to control precisely
> > for when
> > > > > > to send the ACK commands.
> > > > > >
> > > > > > In my design, `acknowledgeCumulative` that accepts a single
> > message ID
> > > > > > will go through the ACK grouping tracker, while
> > > > > > `acknowledgeCumulative` that accepts a map should send ACK commands
> > > > > > immediately because it represents users cache the pending message
> > IDs
> > > > > > by themselves.
> > > > > >
> > > > > > Thanks,
> > > > > > Yunze
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > > On Sep 13, 2022, at 18:36, Enrico Olivelli <eo...@gmail.com>
> > wrote:
> > > > > > >
> > > > > > > -1
> > > > > > > I am not sure this is much useful in this form.
> > > > > > > I had also commented on the PR: We pass a Map<String, MessageId>
> > but
> > > > > > > actually the keys of the map will not be used.
> > > > > > >
> > > > > > > I understand that the current API is not user-friendly, but I
> > don't
> > > > > > > think that adding a dummy parameter to the API will really
> > > > > > > help.
> > > > > > > There is no way to confirm that the MessageId really belongs to
> > the
> > > > > > > partition and you can pass whatever you want
> > > > > > >
> > > > > > > Enrico
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > Il giorno mar 13 set 2022 alle ore 12:24 Xiangying Meng
> > > > > > > <xi...@apache.org> ha scritto:
> > > > > > >>
> > > > > > >> +1 (non-binding)
> > > > > > >> Good work!
> > > > > > >>
> > > > > > >> Sincerely,
> > > > > > >> Xiangying
> > > > > > >>
> > > > > > >> On Tue, Sep 13, 2022 at 6:15 PM Tarun Annapareddy <
> > > > > > >> tarunannapareddy1997@gmail.com> wrote:
> > > > > > >>
> > > > > > >>> Hi devs,
> > > > > > >>> This is the official thread to vote for Java Client support
> > to
> > > > > > >>> Cumulative Acknowledge messages for multiple partitions or
> > topics
> > > > > > >>>
> > > > > > >>> PIP Issue: https://github.com/apache/pulsar/issues/17574
> > > > > > >>> Discussion thread:
> > > > > > >>>
> > https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > > > > > >>> PR: https://github.com/apache/pulsar/pull/17577
> > > > > > >>>
> > > > > > >>> Thank you,
> > > > > > >>> Tarun.
> > > > > > >>>
> > > > > >
> > > > > >
> >
Re: PIP-Java Client support to Cumulative Acknowledge messages for multiple partitions or topics
Posted by PengHui Li <pe...@apache.org>.
> while (true) {
Messages<String> messages = consumer.batchReceive();
Message<String> finalMessage = null;
for (Message<String> message : messages) {
process(message);
finalMessage = message;
}
if (finalMessage != null) {
consumer.acknowledgeCumulativeAsync(finalMessage);
}
}
```
I think this could be a good idea for improving the user's experience of
cumulative ack.
For one batch receive operation, users can only receive messages from one
partition/topic.
The cumulative ack becomes more assured and understandable.
For the performance issue. The worst case is degenerate into the single
message receiving.
Users are mostly using the single message receiving for now.
Regarding the issue of batch receive itself, I think we should discuss it
separately since currently, we are
split the batch first and then create a new list for users with the split
messages. This can be improved
if we do some optimization for this part.
Thanks,
Penghui
On Thu, Oct 20, 2022 at 7:14 PM 丛搏 <bo...@apache.org> wrote:
> Hi, Enrico
> I think we temporarily don't add the new API, it will make the users
> very hard to use. we can try to improve the performance first.
>
> the performance can improve:
> 1. create multi consumer clients to consume the multi-partition topic.
> 2. change the MultiTopicsConsumerImpl incoming queue, using
> ConsumerImpl.batchReceive() directly when use
> MultiTopicsConsumerImpl.batchReceive()
>
> Thanks,
> bo
>
> Enrico Olivelli <eo...@gmail.com> 于2022年10月20日周四 17:37写道：
> >
> > Il giorno gio 20 ott 2022 alle ore 11:09 丛搏 <bo...@apache.org> ha
> scritto:
> > >
> > > Hi,
> > >
> > > I have another solution, we only guarantee the invoke batch receive
> > > returns the messages are the same topic message.
> >
> > This is going to reduce performances of applications that consume from
> > a partitioned topic or using a pattern or multiple topics, the
> > application will receive smaller batches.
> > Also, in the current API you don't have this assumption.
> >
> > I think that if we want to go this way we have to add a new
> > batchReceive API that guarantees that all the messages are from the
> > same message
> >
> > batchReceiveSingleTopic()
> >
> > Enrico
> >
> >
> > In this way, we don't
> > > need to add the new API for ack. use example is:
> > > ```
> > > // consumer the multi-partition messages
> > > while (true) {
> > > Messages<String> messages = consumer.batchReceive();
> > > Message<String> finalMessage = null;
> > > for (Message<String> message : messages) {
> > > process(message);
> > > finalMessage = message;
> > > }
> > > if (finalMessage != null) {
> > > consumer.acknowledgeCumulativeAsync(finalMessage);
> > > }
> > > }
> > > ```
> > > implementation pr: https://github.com/congbobo184/pulsar/pull/5
> > >
> > > Thanks,
> > > bo
> > >
> > > Tarun Annapareddy <ta...@gmail.com> 于2022年9月14日周三
> 17:46写道：
> > > >
> > > > Hi Enrico,
> > > > I can understand the concern about the validation of the topic
> > > > string key in the map we are using. In this case is it good to have
> > > > Map<String, Message> topicToMessage? It will help us to validate
> that the
> > > > message truly belongs to the topic/partition.
> > > >
> > > > Thank you,
> > > > Tarun.
> > > >
> > > > On Wed, 14 Sept 2022 at 07:50, Yunze Xu <yzxu@streamnative.io.invalid
> >
> > > > wrote:
> > > >
> > > > > Hi Enrico,
> > > > >
> > > > > > There is no way to confirm that the MessageId really belongs to
> the
> > > > > > partition and you can pass whatever you want
> > > > >
> > > > > True. But still, the existing `acknowledgeCumulative` API is not
> > > > > convenient to use. The original purpose for this new overload is
> > > > > allowing users to maintain each partition's latest message ID to
> avoid
> > > > > frequent ACK commands, then they can flush all these message IDs
> once.
> > > > >
> > > > > Though the keys are actually not used, a natural way is to
> maintain a
> > > > > map, not a list or set. For a multi-topics consumer, we can also
> > > > > add a constraint that the key must exist in an internal consumer.
> For
> > > > > a single topic consumer, the key must be the topic name.
> > > > >
> > > > > Without the new API, users have to configure
> `acknowledgmentGroupTime`
> > > > > and call `acknowledgeCumulative` for each message received.
> However,
> > > > > the ACK grouping tracker is very simple that it can only configure
> the
> > > > > grouping timeout. I think users need a way to control precisely
> for when
> > > > > to send the ACK commands.
> > > > >
> > > > > In my design, `acknowledgeCumulative` that accepts a single
> message ID
> > > > > will go through the ACK grouping tracker, while
> > > > > `acknowledgeCumulative` that accepts a map should send ACK commands
> > > > > immediately because it represents users cache the pending message
> IDs
> > > > > by themselves.
> > > > >
> > > > > Thanks,
> > > > > Yunze
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > > On Sep 13, 2022, at 18:36, Enrico Olivelli <eo...@gmail.com>
> wrote:
> > > > > >
> > > > > > -1
> > > > > > I am not sure this is much useful in this form.
> > > > > > I had also commented on the PR: We pass a Map<String, MessageId>
> but
> > > > > > actually the keys of the map will not be used.
> > > > > >
> > > > > > I understand that the current API is not user-friendly, but I
> don't
> > > > > > think that adding a dummy parameter to the API will really
> > > > > > help.
> > > > > > There is no way to confirm that the MessageId really belongs to
> the
> > > > > > partition and you can pass whatever you want
> > > > > >
> > > > > > Enrico
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Il giorno mar 13 set 2022 alle ore 12:24 Xiangying Meng
> > > > > > <xi...@apache.org> ha scritto:
> > > > > >>
> > > > > >> +1 (non-binding)
> > > > > >> Good work!
> > > > > >>
> > > > > >> Sincerely,
> > > > > >> Xiangying
> > > > > >>
> > > > > >> On Tue, Sep 13, 2022 at 6:15 PM Tarun Annapareddy <
> > > > > >> tarunannapareddy1997@gmail.com> wrote:
> > > > > >>
> > > > > >>> Hi devs,
> > > > > >>> This is the official thread to vote for Java Client support
> to
> > > > > >>> Cumulative Acknowledge messages for multiple partitions or
> topics
> > > > > >>>
> > > > > >>> PIP Issue: https://github.com/apache/pulsar/issues/17574
> > > > > >>> Discussion thread:
> > > > > >>>
> https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > > > > >>> PR: https://github.com/apache/pulsar/pull/17577
> > > > > >>>
> > > > > >>> Thank you,
> > > > > >>> Tarun.
> > > > > >>>
> > > > >
> > > > >
>
Re: PIP-Java Client support to Cumulative Acknowledge messages for multiple partitions or topics
Posted by 丛搏 <bo...@apache.org>.
Hi, Enrico
I think we temporarily don't add the new API, it will make the users
very hard to use. we can try to improve the performance first.
the performance can improve:
1. create multi consumer clients to consume the multi-partition topic.
2. change the MultiTopicsConsumerImpl incoming queue, using
ConsumerImpl.batchReceive() directly when use
MultiTopicsConsumerImpl.batchReceive()
Thanks,
bo
Enrico Olivelli <eo...@gmail.com> 于2022年10月20日周四 17:37写道：
>
> Il giorno gio 20 ott 2022 alle ore 11:09 丛搏 <bo...@apache.org> ha scritto:
> >
> > Hi,
> >
> > I have another solution, we only guarantee the invoke batch receive
> > returns the messages are the same topic message.
>
> This is going to reduce performances of applications that consume from
> a partitioned topic or using a pattern or multiple topics, the
> application will receive smaller batches.
> Also, in the current API you don't have this assumption.
>
> I think that if we want to go this way we have to add a new
> batchReceive API that guarantees that all the messages are from the
> same message
>
> batchReceiveSingleTopic()
>
> Enrico
>
>
> In this way, we don't
> > need to add the new API for ack. use example is:
> > ```
> > // consumer the multi-partition messages
> > while (true) {
> > Messages<String> messages = consumer.batchReceive();
> > Message<String> finalMessage = null;
> > for (Message<String> message : messages) {
> > process(message);
> > finalMessage = message;
> > }
> > if (finalMessage != null) {
> > consumer.acknowledgeCumulativeAsync(finalMessage);
> > }
> > }
> > ```
> > implementation pr: https://github.com/congbobo184/pulsar/pull/5
> >
> > Thanks,
> > bo
> >
> > Tarun Annapareddy <ta...@gmail.com> 于2022年9月14日周三 17:46写道：
> > >
> > > Hi Enrico,
> > > I can understand the concern about the validation of the topic
> > > string key in the map we are using. In this case is it good to have
> > > Map<String, Message> topicToMessage? It will help us to validate that the
> > > message truly belongs to the topic/partition.
> > >
> > > Thank you,
> > > Tarun.
> > >
> > > On Wed, 14 Sept 2022 at 07:50, Yunze Xu <yz...@streamnative.io.invalid>
> > > wrote:
> > >
> > > > Hi Enrico,
> > > >
> > > > > There is no way to confirm that the MessageId really belongs to the
> > > > > partition and you can pass whatever you want
> > > >
> > > > True. But still, the existing `acknowledgeCumulative` API is not
> > > > convenient to use. The original purpose for this new overload is
> > > > allowing users to maintain each partition's latest message ID to avoid
> > > > frequent ACK commands, then they can flush all these message IDs once.
> > > >
> > > > Though the keys are actually not used, a natural way is to maintain a
> > > > map, not a list or set. For a multi-topics consumer, we can also
> > > > add a constraint that the key must exist in an internal consumer. For
> > > > a single topic consumer, the key must be the topic name.
> > > >
> > > > Without the new API, users have to configure `acknowledgmentGroupTime`
> > > > and call `acknowledgeCumulative` for each message received. However,
> > > > the ACK grouping tracker is very simple that it can only configure the
> > > > grouping timeout. I think users need a way to control precisely for when
> > > > to send the ACK commands.
> > > >
> > > > In my design, `acknowledgeCumulative` that accepts a single message ID
> > > > will go through the ACK grouping tracker, while
> > > > `acknowledgeCumulative` that accepts a map should send ACK commands
> > > > immediately because it represents users cache the pending message IDs
> > > > by themselves.
> > > >
> > > > Thanks,
> > > > Yunze
> > > >
> > > >
> > > >
> > > >
> > > > > On Sep 13, 2022, at 18:36, Enrico Olivelli <eo...@gmail.com> wrote:
> > > > >
> > > > > -1
> > > > > I am not sure this is much useful in this form.
> > > > > I had also commented on the PR: We pass a Map<String, MessageId> but
> > > > > actually the keys of the map will not be used.
> > > > >
> > > > > I understand that the current API is not user-friendly, but I don't
> > > > > think that adding a dummy parameter to the API will really
> > > > > help.
> > > > > There is no way to confirm that the MessageId really belongs to the
> > > > > partition and you can pass whatever you want
> > > > >
> > > > > Enrico
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > Il giorno mar 13 set 2022 alle ore 12:24 Xiangying Meng
> > > > > <xi...@apache.org> ha scritto:
> > > > >>
> > > > >> +1 (non-binding)
> > > > >> Good work!
> > > > >>
> > > > >> Sincerely,
> > > > >> Xiangying
> > > > >>
> > > > >> On Tue, Sep 13, 2022 at 6:15 PM Tarun Annapareddy <
> > > > >> tarunannapareddy1997@gmail.com> wrote:
> > > > >>
> > > > >>> Hi devs,
> > > > >>> This is the official thread to vote for Java Client support to
> > > > >>> Cumulative Acknowledge messages for multiple partitions or topics
> > > > >>>
> > > > >>> PIP Issue: https://github.com/apache/pulsar/issues/17574
> > > > >>> Discussion thread:
> > > > >>> https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > > > >>> PR: https://github.com/apache/pulsar/pull/17577
> > > > >>>
> > > > >>> Thank you,
> > > > >>> Tarun.
> > > > >>>
> > > >
> > > >
Re: PIP-Java Client support to Cumulative Acknowledge messages for multiple partitions or topics
Posted by Enrico Olivelli <eo...@gmail.com>.
Il giorno gio 20 ott 2022 alle ore 11:09 丛搏 <bo...@apache.org> ha scritto:
>
> Hi,
>
> I have another solution, we only guarantee the invoke batch receive
> returns the messages are the same topic message.
This is going to reduce performances of applications that consume from
a partitioned topic or using a pattern or multiple topics, the
application will receive smaller batches.
Also, in the current API you don't have this assumption.
I think that if we want to go this way we have to add a new
batchReceive API that guarantees that all the messages are from the
same message
batchReceiveSingleTopic()
Enrico
In this way, we don't
> need to add the new API for ack. use example is:
> ```
> // consumer the multi-partition messages
> while (true) {
> Messages<String> messages = consumer.batchReceive();
> Message<String> finalMessage = null;
> for (Message<String> message : messages) {
> process(message);
> finalMessage = message;
> }
> if (finalMessage != null) {
> consumer.acknowledgeCumulativeAsync(finalMessage);
> }
> }
> ```
> implementation pr: https://github.com/congbobo184/pulsar/pull/5
>
> Thanks,
> bo
>
> Tarun Annapareddy <ta...@gmail.com> 于2022年9月14日周三 17:46写道：
> >
> > Hi Enrico,
> > I can understand the concern about the validation of the topic
> > string key in the map we are using. In this case is it good to have
> > Map<String, Message> topicToMessage? It will help us to validate that the
> > message truly belongs to the topic/partition.
> >
> > Thank you,
> > Tarun.
> >
> > On Wed, 14 Sept 2022 at 07:50, Yunze Xu <yz...@streamnative.io.invalid>
> > wrote:
> >
> > > Hi Enrico,
> > >
> > > > There is no way to confirm that the MessageId really belongs to the
> > > > partition and you can pass whatever you want
> > >
> > > True. But still, the existing `acknowledgeCumulative` API is not
> > > convenient to use. The original purpose for this new overload is
> > > allowing users to maintain each partition's latest message ID to avoid
> > > frequent ACK commands, then they can flush all these message IDs once.
> > >
> > > Though the keys are actually not used, a natural way is to maintain a
> > > map, not a list or set. For a multi-topics consumer, we can also
> > > add a constraint that the key must exist in an internal consumer. For
> > > a single topic consumer, the key must be the topic name.
> > >
> > > Without the new API, users have to configure `acknowledgmentGroupTime`
> > > and call `acknowledgeCumulative` for each message received. However,
> > > the ACK grouping tracker is very simple that it can only configure the
> > > grouping timeout. I think users need a way to control precisely for when
> > > to send the ACK commands.
> > >
> > > In my design, `acknowledgeCumulative` that accepts a single message ID
> > > will go through the ACK grouping tracker, while
> > > `acknowledgeCumulative` that accepts a map should send ACK commands
> > > immediately because it represents users cache the pending message IDs
> > > by themselves.
> > >
> > > Thanks,
> > > Yunze
> > >
> > >
> > >
> > >
> > > > On Sep 13, 2022, at 18:36, Enrico Olivelli <eo...@gmail.com> wrote:
> > > >
> > > > -1
> > > > I am not sure this is much useful in this form.
> > > > I had also commented on the PR: We pass a Map<String, MessageId> but
> > > > actually the keys of the map will not be used.
> > > >
> > > > I understand that the current API is not user-friendly, but I don't
> > > > think that adding a dummy parameter to the API will really
> > > > help.
> > > > There is no way to confirm that the MessageId really belongs to the
> > > > partition and you can pass whatever you want
> > > >
> > > > Enrico
> > > >
> > > >
> > > >
> > > >
> > > > Il giorno mar 13 set 2022 alle ore 12:24 Xiangying Meng
> > > > <xi...@apache.org> ha scritto:
> > > >>
> > > >> +1 (non-binding)
> > > >> Good work!
> > > >>
> > > >> Sincerely,
> > > >> Xiangying
> > > >>
> > > >> On Tue, Sep 13, 2022 at 6:15 PM Tarun Annapareddy <
> > > >> tarunannapareddy1997@gmail.com> wrote:
> > > >>
> > > >>> Hi devs,
> > > >>> This is the official thread to vote for Java Client support to
> > > >>> Cumulative Acknowledge messages for multiple partitions or topics
> > > >>>
> > > >>> PIP Issue: https://github.com/apache/pulsar/issues/17574
> > > >>> Discussion thread:
> > > >>> https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > > >>> PR: https://github.com/apache/pulsar/pull/17577
> > > >>>
> > > >>> Thank you,
> > > >>> Tarun.
> > > >>>
> > >
> > >
Re: PIP-Java Client support to Cumulative Acknowledge messages for multiple partitions or topics
Posted by 丛搏 <bo...@apache.org>.
Hi,
I have another solution, we only guarantee the invoke batch receive
returns the messages are the same topic message. In this way, we don't
need to add the new API for ack. use example is:
```
// consumer the multi-partition messages
while (true) {
Messages<String> messages = consumer.batchReceive();
Message<String> finalMessage = null;
for (Message<String> message : messages) {
process(message);
finalMessage = message;
}
if (finalMessage != null) {
consumer.acknowledgeCumulativeAsync(finalMessage);
}
}
```
implementation pr: https://github.com/congbobo184/pulsar/pull/5
Thanks,
bo
Tarun Annapareddy <ta...@gmail.com> 于2022年9月14日周三 17:46写道：
>
> Hi Enrico,
> I can understand the concern about the validation of the topic
> string key in the map we are using. In this case is it good to have
> Map<String, Message> topicToMessage? It will help us to validate that the
> message truly belongs to the topic/partition.
>
> Thank you,
> Tarun.
>
> On Wed, 14 Sept 2022 at 07:50, Yunze Xu <yz...@streamnative.io.invalid>
> wrote:
>
> > Hi Enrico,
> >
> > > There is no way to confirm that the MessageId really belongs to the
> > > partition and you can pass whatever you want
> >
> > True. But still, the existing `acknowledgeCumulative` API is not
> > convenient to use. The original purpose for this new overload is
> > allowing users to maintain each partition's latest message ID to avoid
> > frequent ACK commands, then they can flush all these message IDs once.
> >
> > Though the keys are actually not used, a natural way is to maintain a
> > map, not a list or set. For a multi-topics consumer, we can also
> > add a constraint that the key must exist in an internal consumer. For
> > a single topic consumer, the key must be the topic name.
> >
> > Without the new API, users have to configure `acknowledgmentGroupTime`
> > and call `acknowledgeCumulative` for each message received. However,
> > the ACK grouping tracker is very simple that it can only configure the
> > grouping timeout. I think users need a way to control precisely for when
> > to send the ACK commands.
> >
> > In my design, `acknowledgeCumulative` that accepts a single message ID
> > will go through the ACK grouping tracker, while
> > `acknowledgeCumulative` that accepts a map should send ACK commands
> > immediately because it represents users cache the pending message IDs
> > by themselves.
> >
> > Thanks,
> > Yunze
> >
> >
> >
> >
> > > On Sep 13, 2022, at 18:36, Enrico Olivelli <eo...@gmail.com> wrote:
> > >
> > > -1
> > > I am not sure this is much useful in this form.
> > > I had also commented on the PR: We pass a Map<String, MessageId> but
> > > actually the keys of the map will not be used.
> > >
> > > I understand that the current API is not user-friendly, but I don't
> > > think that adding a dummy parameter to the API will really
> > > help.
> > > There is no way to confirm that the MessageId really belongs to the
> > > partition and you can pass whatever you want
> > >
> > > Enrico
> > >
> > >
> > >
> > >
> > > Il giorno mar 13 set 2022 alle ore 12:24 Xiangying Meng
> > > <xi...@apache.org> ha scritto:
> > >>
> > >> +1 (non-binding)
> > >> Good work!
> > >>
> > >> Sincerely,
> > >> Xiangying
> > >>
> > >> On Tue, Sep 13, 2022 at 6:15 PM Tarun Annapareddy <
> > >> tarunannapareddy1997@gmail.com> wrote:
> > >>
> > >>> Hi devs,
> > >>> This is the official thread to vote for Java Client support to
> > >>> Cumulative Acknowledge messages for multiple partitions or topics
> > >>>
> > >>> PIP Issue: https://github.com/apache/pulsar/issues/17574
> > >>> Discussion thread:
> > >>> https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > >>> PR: https://github.com/apache/pulsar/pull/17577
> > >>>
> > >>> Thank you,
> > >>> Tarun.
> > >>>
> >
> >
Re: PIP-Java Client support to Cumulative Acknowledge messages for multiple partitions or topics
Posted by Tarun Annapareddy <ta...@gmail.com>.
Hi Enrico,
I can understand the concern about the validation of the topic
string key in the map we are using. In this case is it good to have
Map<String, Message> topicToMessage? It will help us to validate that the
message truly belongs to the topic/partition.
Thank you,
Tarun.
On Wed, 14 Sept 2022 at 07:50, Yunze Xu <yz...@streamnative.io.invalid>
wrote:
> Hi Enrico,
>
> > There is no way to confirm that the MessageId really belongs to the
> > partition and you can pass whatever you want
>
> True. But still, the existing `acknowledgeCumulative` API is not
> convenient to use. The original purpose for this new overload is
> allowing users to maintain each partition's latest message ID to avoid
> frequent ACK commands, then they can flush all these message IDs once.
>
> Though the keys are actually not used, a natural way is to maintain a
> map, not a list or set. For a multi-topics consumer, we can also
> add a constraint that the key must exist in an internal consumer. For
> a single topic consumer, the key must be the topic name.
>
> Without the new API, users have to configure `acknowledgmentGroupTime`
> and call `acknowledgeCumulative` for each message received. However,
> the ACK grouping tracker is very simple that it can only configure the
> grouping timeout. I think users need a way to control precisely for when
> to send the ACK commands.
>
> In my design, `acknowledgeCumulative` that accepts a single message ID
> will go through the ACK grouping tracker, while
> `acknowledgeCumulative` that accepts a map should send ACK commands
> immediately because it represents users cache the pending message IDs
> by themselves.
>
> Thanks,
> Yunze
>
>
>
>
> > On Sep 13, 2022, at 18:36, Enrico Olivelli <eo...@gmail.com> wrote:
> >
> > -1
> > I am not sure this is much useful in this form.
> > I had also commented on the PR: We pass a Map<String, MessageId> but
> > actually the keys of the map will not be used.
> >
> > I understand that the current API is not user-friendly, but I don't
> > think that adding a dummy parameter to the API will really
> > help.
> > There is no way to confirm that the MessageId really belongs to the
> > partition and you can pass whatever you want
> >
> > Enrico
> >
> >
> >
> >
> > Il giorno mar 13 set 2022 alle ore 12:24 Xiangying Meng
> > <xi...@apache.org> ha scritto:
> >>
> >> +1 (non-binding)
> >> Good work!
> >>
> >> Sincerely,
> >> Xiangying
> >>
> >> On Tue, Sep 13, 2022 at 6:15 PM Tarun Annapareddy <
> >> tarunannapareddy1997@gmail.com> wrote:
> >>
> >>> Hi devs,
> >>> This is the official thread to vote for Java Client support to
> >>> Cumulative Acknowledge messages for multiple partitions or topics
> >>>
> >>> PIP Issue: https://github.com/apache/pulsar/issues/17574
> >>> Discussion thread:
> >>> https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> >>> PR: https://github.com/apache/pulsar/pull/17577
> >>>
> >>> Thank you,
> >>> Tarun.
> >>>
>
>
Re: PIP-Java Client support to Cumulative Acknowledge messages for multiple partitions or topics
Posted by Yunze Xu <yz...@streamnative.io.INVALID>.
Hi Enrico,
> There is no way to confirm that the MessageId really belongs to the
> partition and you can pass whatever you want
True. But still, the existing `acknowledgeCumulative` API is not
convenient to use. The original purpose for this new overload is
allowing users to maintain each partition's latest message ID to avoid
frequent ACK commands, then they can flush all these message IDs once.
Though the keys are actually not used, a natural way is to maintain a
map, not a list or set. For a multi-topics consumer, we can also
add a constraint that the key must exist in an internal consumer. For
a single topic consumer, the key must be the topic name.
Without the new API, users have to configure `acknowledgmentGroupTime`
and call `acknowledgeCumulative` for each message received. However,
the ACK grouping tracker is very simple that it can only configure the
grouping timeout. I think users need a way to control precisely for when
to send the ACK commands.
In my design, `acknowledgeCumulative` that accepts a single message ID
will go through the ACK grouping tracker, while
`acknowledgeCumulative` that accepts a map should send ACK commands
immediately because it represents users cache the pending message IDs
by themselves.
Thanks,
Yunze
> On Sep 13, 2022, at 18:36, Enrico Olivelli <eo...@gmail.com> wrote:
>
> -1
> I am not sure this is much useful in this form.
> I had also commented on the PR: We pass a Map<String, MessageId> but
> actually the keys of the map will not be used.
>
> I understand that the current API is not user-friendly, but I don't
> think that adding a dummy parameter to the API will really
> help.
> There is no way to confirm that the MessageId really belongs to the
> partition and you can pass whatever you want
>
> Enrico
>
>
>
>
> Il giorno mar 13 set 2022 alle ore 12:24 Xiangying Meng
> <xi...@apache.org> ha scritto:
>>
>> +1 (non-binding)
>> Good work!
>>
>> Sincerely,
>> Xiangying
>>
>> On Tue, Sep 13, 2022 at 6:15 PM Tarun Annapareddy <
>> tarunannapareddy1997@gmail.com> wrote:
>>
>>> Hi devs,
>>> This is the official thread to vote for Java Client support to
>>> Cumulative Acknowledge messages for multiple partitions or topics
>>>
>>> PIP Issue: https://github.com/apache/pulsar/issues/17574
>>> Discussion thread:
>>> https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
>>> PR: https://github.com/apache/pulsar/pull/17577
>>>
>>> Thank you,
>>> Tarun.
>>>
Re: PIP-Java Client support to Cumulative Acknowledge messages for multiple partitions or topics
Posted by Enrico Olivelli <eo...@gmail.com>.
-1
I am not sure this is much useful in this form.
I had also commented on the PR: We pass a Map<String, MessageId> but
actually the keys of the map will not be used.
I understand that the current API is not user-friendly, but I don't
think that adding a dummy parameter to the API will really
help.
There is no way to confirm that the MessageId really belongs to the
partition and you can pass whatever you want
Enrico
Il giorno mar 13 set 2022 alle ore 12:24 Xiangying Meng
<xi...@apache.org> ha scritto:
>
> +1 (non-binding)
> Good work!
>
> Sincerely,
> Xiangying
>
> On Tue, Sep 13, 2022 at 6:15 PM Tarun Annapareddy <
> tarunannapareddy1997@gmail.com> wrote:
>
> > Hi devs,
> > This is the official thread to vote for Java Client support to
> > Cumulative Acknowledge messages for multiple partitions or topics
> >
> > PIP Issue: https://github.com/apache/pulsar/issues/17574
> > Discussion thread:
> > https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> > PR: https://github.com/apache/pulsar/pull/17577
> >
> > Thank you,
> > Tarun.
> >
Re: PIP-Java Client support to Cumulative Acknowledge messages for multiple partitions or topics
Posted by Xiangying Meng <xi...@apache.org>.
+1 (non-binding)
Good work!
Sincerely,
Xiangying
On Tue, Sep 13, 2022 at 6:15 PM Tarun Annapareddy <
tarunannapareddy1997@gmail.com> wrote:
> Hi devs,
> This is the official thread to vote for Java Client support to
> Cumulative Acknowledge messages for multiple partitions or topics
>
> PIP Issue: https://github.com/apache/pulsar/issues/17574
> Discussion thread:
> https://lists.apache.org/thread/k090ftlqc149yr1cnprxb29vxg160131
> PR: https://github.com/apache/pulsar/pull/17577
>
> Thank you,
> Tarun.
>