You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "BewareMyPower (via GitHub)" <gi...@apache.org> on 2023/02/16 03:38:45 UTC
[GitHub] [pulsar-client-go] BewareMyPower opened a new pull request, #957: Support grouping ACK requests by time and size
BewareMyPower opened a new pull request, #957:
URL: https://github.com/apache/pulsar-client-go/pull/957
Fixes #949
### Motivation
Currently the Go client does not support grouping ACK requests, so each time `Ack` (or similar APIs) is called, a ACK request will be sent, which could downgrade the performance. We need to support configuring the time and size to cache `MessageID` before sending ACK requests.
### Modifications
- Add an `AckGroupingOptions` field to `ConsumerOptions`, when it's nil, use 100ms as the max time and 1000 as the max size.
- Add an `ackGroupingTracker` interface to support grouping ACK requests.
- When `AckWithResponse` is false, adding the `MessageID` instance to the tracker instead of sending the requests to `eventsCh`.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
- Added `ack_grouping_tracker_test.go` to verify `ackGroupingTracker` in various cases
- The consumer side change can be covered by existing tests because the default `AckGroupingOptions` config is `{ MaxSize: 1000, MaxTime: 100*time.Millisecond }`.
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no)
- The public API: (**yes**)
- The schema: (yes / no / don't know)
- The default values of configurations: (yes / no)
- The wire protocol: (yes / no)
### Documentation
- Does this pull request introduce a new feature? (yes)
- If yes, how is the feature documented? (GoDocs)
- If a feature is not applicable for documentation, explain why?
- If a feature is not documented yet in this PR, please create a followup issue for adding the documentation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] panszobe commented on pull request #957: Support grouping ACK requests by time and size
Posted by "panszobe (via GitHub)" <gi...@apache.org>.
panszobe commented on PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#issuecomment-1439623863
Does Community has some Performance Test Data?
I used the latest version(v0.9.1-0.20230222003822-7d257b01b121) to test today, still found that which Test Case by EnableBatchIndexAcknowledgment=true consumed messages much more slower than EnableBatchIndexAcknowledgment=false, and could not catch up with the producer‘s production.
As consuming from the same topic, consumers of JAVA Client by EnableBatchIndexAcknowledgment=true consumed messages smoothly, and backlog was not enlarged.
I think that _Support grouping ACK_ feature is OK, but maybe _EnableBatchIndexAcknowledgment_ feature have much room for improvement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on a diff in pull request #957: Support grouping ACK requests by time and size
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#discussion_r1112112794
##########
pulsar/ack_grouping_tracker.go:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package pulsar
+
+import (
+ "time"
+
+ "github.com/bits-and-blooms/bitset"
+)
+
+type ackGroupingTracker interface {
+ add(id MessageID)
+
+ addCumulative(id MessageID)
+
+ isDuplicate(id MessageID) bool
+
+ flush()
+
+ flushAndClean()
+
+ close()
+}
+
+type ackFlushType int
+
+const (
+ flushOnly ackFlushType = iota
+ flushAndClean
+ flushAndClose
+)
+
+func newAckGroupingTracker(options *AckGroupingOptions,
+ ackIndividual func(id MessageID),
+ ackCumulative func(id MessageID)) ackGroupingTracker {
+ if options == nil {
+ options = &AckGroupingOptions{
+ MaxSize: 1000,
+ MaxTime: 100 * time.Millisecond,
+ }
+ }
+
+ if options.MaxSize <= 1 {
+ return &immediateAckGroupingTracker{
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ }
+ }
+
+ c := &cachedAcks{
+ singleAcks: make([]MessageID, options.MaxSize),
+ pendingAcks: make(map[int64]*bitset.BitSet),
+ lastCumulativeAck: EarliestMessageID(),
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ ackList: func(ids []MessageID) {
+ // TODO: support ack a list of MessageIDs
+ for _, id := range ids {
+ ackIndividual(id)
+ }
+ },
+ }
+
+ timeout := time.NewTicker(time.Hour)
+ if options.MaxTime > 0 {
+ timeout = time.NewTicker(options.MaxTime)
+ } else {
+ timeout.Stop()
+ }
+ t := &timedAckGroupingTracker{
+ ackIndividualCh: make(chan MessageID),
+ ackCumulativeCh: make(chan MessageID),
+ duplicateIDCh: make(chan MessageID),
+ duplicateResultCh: make(chan bool),
+ flushCh: make(chan ackFlushType, 10),
Review Comment:
Since all flush operations are synchronous, I changed it to a unbuffered channel now. Let's see if any test would fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] panszobe commented on pull request #957: Support grouping ACK requests by time and size
Posted by "panszobe (via GitHub)" <gi...@apache.org>.
panszobe commented on PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#issuecomment-1441151660
> @panszobe From the flamegraph, I think the root case is the inefficiency process of sending ACKs. The most time of `internalAck` method is `internal.(*rpcClient).RequestOnCnxNoWait`. In this method, `SendRequestNoWait` takes 3/4 of the total time, while the `internal.baseCommand` function also takes about 1/4 of the total time.
>
> While I'm looking forward to your test results of other `AckGroupingOption` configs, I'm going to test the performance locally. Could you give the information of your test data? Including:
>
> * The batch configs of the producer
> * The message size
> * The ACK type (individual or cumulative?)
The situations of producer: about 20MB/s, 200k messages/s, batch = 10. Through JAVA pressure test procedure.
ACK type is individual, we use `AckID` method to commit. SubscriptionType is Shared.
I adjusted the `AckGroupingOption` configs at 09:49, increasing MaxSize to 1000 and MaxTime to 1000, metrics monitor as below:

It seems that the adjustment does not have any improvement.
Looking forward to your local test. @BewareMyPower
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on a diff in pull request #957: Support grouping ACK requests by time and size
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#discussion_r1112109432
##########
pulsar/ack_grouping_tracker.go:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package pulsar
+
+import (
+ "time"
+
+ "github.com/bits-and-blooms/bitset"
+)
+
+type ackGroupingTracker interface {
+ add(id MessageID)
+
+ addCumulative(id MessageID)
+
+ isDuplicate(id MessageID) bool
+
+ flush()
+
+ flushAndClean()
+
+ close()
+}
+
+type ackFlushType int
+
+const (
+ flushOnly ackFlushType = iota
+ flushAndClean
+ flushAndClose
+)
+
+func newAckGroupingTracker(options *AckGroupingOptions,
+ ackIndividual func(id MessageID),
+ ackCumulative func(id MessageID)) ackGroupingTracker {
+ if options == nil {
+ options = &AckGroupingOptions{
+ MaxSize: 1000,
+ MaxTime: 100 * time.Millisecond,
+ }
+ }
+
+ if options.MaxSize <= 1 {
+ return &immediateAckGroupingTracker{
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ }
+ }
+
+ c := &cachedAcks{
+ singleAcks: make([]MessageID, options.MaxSize),
+ pendingAcks: make(map[int64]*bitset.BitSet),
+ lastCumulativeAck: EarliestMessageID(),
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ ackList: func(ids []MessageID) {
+ // TODO: support ack a list of MessageIDs
+ for _, id := range ids {
+ ackIndividual(id)
+ }
+ },
+ }
+
+ timeout := time.NewTicker(time.Hour)
+ if options.MaxTime > 0 {
+ timeout = time.NewTicker(options.MaxTime)
+ } else {
+ timeout.Stop()
+ }
+ t := &timedAckGroupingTracker{
+ ackIndividualCh: make(chan MessageID),
+ ackCumulativeCh: make(chan MessageID),
+ duplicateIDCh: make(chan MessageID),
+ duplicateResultCh: make(chan bool),
+ flushCh: make(chan ackFlushType, 10),
+ waitFlushCh: make(chan bool),
+ }
+ go func() {
+ for {
+ select {
+ case id := <-t.ackIndividualCh:
+ if c.addAndCheckIfFull(id) {
+ c.flushIndividualAcks()
+ if options.MaxTime > 0 {
+ timeout.Reset(options.MaxTime)
+ }
+ }
+ case id := <-t.ackCumulativeCh:
+ c.tryUpdateLastCumulativeAck(id)
+ if options.MaxTime <= 0 {
+ c.flushCumulativeAck()
+ }
+ case id := <-t.duplicateIDCh:
+ t.duplicateResultCh <- c.isDuplicate(id)
+ case <-timeout.C:
+ c.flush()
+ case ackFlushType := <-t.flushCh:
+ timeout.Stop()
+ c.flush()
+ if ackFlushType == flushAndClean {
+ c.clean()
+ }
+ t.waitFlushCh <- true
+ if ackFlushType == flushAndClose {
+ return
+ }
+ }
+ }
+ }()
+ return t
+}
+
+type immediateAckGroupingTracker struct {
+ ackIndividual func(id MessageID)
+ ackCumulative func(id MessageID)
+}
+
+func (i *immediateAckGroupingTracker) add(id MessageID) {
+ i.ackIndividual(id)
+}
+
+func (i *immediateAckGroupingTracker) addCumulative(id MessageID) {
+ i.ackCumulative(id)
+}
+
+func (i *immediateAckGroupingTracker) isDuplicate(id MessageID) bool {
+ return false
+}
+
+func (i *immediateAckGroupingTracker) flush() {
+}
+
+func (i *immediateAckGroupingTracker) flushAndClean() {
+}
+
+func (i *immediateAckGroupingTracker) close() {
+}
+
+type cachedAcks struct {
+ singleAcks []MessageID
+ index int
+
+ // Key is the hash code of the ledger id and the netry id,
+ // Value is the bit set that represents which messages are acknowledged if the entry stores a batch.
+ // The bit 1 represents the message has been acknowledged, i.e. the bits "111" represents all messages
+ // in the batch whose batch size is 3 are not acknowledged.
+ // After the 1st message (i.e. batch index is 0) is acknowledged, the bits will become "011".
+ // Value is nil if the entry represents a single message.
+ pendingAcks map[int64]*bitset.BitSet
Review Comment:
Yes. It records the pending ACKs (`MessageID` instances) that have not been sent, so the messages in these `MessageID`s could be received by redelivering.
Here I use different data structures with Java client:
```java
// MessageId -> null
private final ConcurrentSkipListSet<MessageIdImpl> pendingIndividualAcks;
// MessageId -> BitSet
private final ConcurrentHashMap<MessageIdImpl, ConcurrentBitSetRecyclable> pendingIndividualBatchIndexAcks;
```
I merged these two maps into a single one in the Golang client. In addition, when Java client check if the cache is full, the `pendingIndividualBatchIndexAcks`, which represents batch index ACKs, is not counted. IMO it should be counted and I will push a fix to the Java client soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] panszobe commented on pull request #957: Support grouping ACK requests by time and size
Posted by "panszobe (via GitHub)" <gi...@apache.org>.
panszobe commented on PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#issuecomment-1451991455
> @panszobe I'm working on this issue currently. But since I'm busy on something else this week, the PR was not opened. If you mean #968, then it's not related to this issue. It's just a problem I found when I wrote the code.
OK. Looking forward to the new PR. Thanks~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] panszobe commented on pull request #957: Support grouping ACK requests by time and size
Posted by "panszobe (via GitHub)" <gi...@apache.org>.
panszobe commented on PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#issuecomment-1451654688
@BewareMyPower hi~
Seeing the new PR, I used latest verision(v0.9.1-0.20230301160414-42ded0d59c46) to test today, but it seems that it has no improvements.
<img width="1332" alt="image" src="https://user-images.githubusercontent.com/75996911/222393346-b0ed76a0-a22f-49d0-9aa0-7b6a0ca6eb30.png">
<img width="1326" alt="image" src="https://user-images.githubusercontent.com/75996911/222393116-59cf877b-d272-4709-8ccc-f70f7bdc11cb.png">
<img width="1355" alt="image" src="https://user-images.githubusercontent.com/75996911/222400096-0324b02d-15d3-415c-81a4-f2ea8fc980ba.png">
<img width="1303" alt="image" src="https://user-images.githubusercontent.com/75996911/222401301-e8759ec1-fb77-4457-b150-2479dced3eac.png">
According to the flamegraphs as above, SendRequest and MessageReceive took much time.
<img width="1324" alt="image" src="https://user-images.githubusercontent.com/75996911/222404062-7d0bddf7-9b95-44b1-81c3-22b3147892e5.png">
As above, consuming rates and message receving rates also show the test result.
The subscription is very simple, just receiving and acking the Message, has no business processing logic, as below:

But it still could not catch up with the producer's production while the other one which disables `BatchIndexAcknowledgment` has no large backlog and keeps stable.
Do you have any other idea? Or there is any optimizations doing now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on a diff in pull request #957: Support grouping ACK requests by time and size
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#discussion_r1112101120
##########
pulsar/ack_grouping_tracker.go:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package pulsar
+
+import (
+ "time"
+
+ "github.com/bits-and-blooms/bitset"
+)
+
+type ackGroupingTracker interface {
+ add(id MessageID)
+
+ addCumulative(id MessageID)
+
+ isDuplicate(id MessageID) bool
+
+ flush()
+
+ flushAndClean()
+
+ close()
+}
+
+type ackFlushType int
+
+const (
+ flushOnly ackFlushType = iota
+ flushAndClean
+ flushAndClose
+)
+
+func newAckGroupingTracker(options *AckGroupingOptions,
+ ackIndividual func(id MessageID),
+ ackCumulative func(id MessageID)) ackGroupingTracker {
+ if options == nil {
+ options = &AckGroupingOptions{
+ MaxSize: 1000,
+ MaxTime: 100 * time.Millisecond,
+ }
+ }
+
+ if options.MaxSize <= 1 {
+ return &immediateAckGroupingTracker{
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ }
+ }
+
+ c := &cachedAcks{
+ singleAcks: make([]MessageID, options.MaxSize),
+ pendingAcks: make(map[int64]*bitset.BitSet),
+ lastCumulativeAck: EarliestMessageID(),
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ ackList: func(ids []MessageID) {
+ // TODO: support ack a list of MessageIDs
+ for _, id := range ids {
+ ackIndividual(id)
+ }
+ },
+ }
+
+ timeout := time.NewTicker(time.Hour)
+ if options.MaxTime > 0 {
+ timeout = time.NewTicker(options.MaxTime)
+ } else {
+ timeout.Stop()
+ }
+ t := &timedAckGroupingTracker{
+ ackIndividualCh: make(chan MessageID),
+ ackCumulativeCh: make(chan MessageID),
+ duplicateIDCh: make(chan MessageID),
+ duplicateResultCh: make(chan bool),
+ flushCh: make(chan ackFlushType, 10),
Review Comment:
I just set it with a large value in case `flush` could be called in different threads and forgot to change it back. I'm going to find more proper value soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on pull request #957: Support grouping ACK requests by time and size
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#issuecomment-1440081904
@panszobe From the flamegraph, I think the root case is the inefficiency process of sending ACKs. The most time of `internalAck` method is `internal.(*rpcClient).RequestOnCnxNoWait`. In this method, `SendRequestNoWait` takes 3/4 of the total time, while the `internal.baseCommand` function also takes about 1/4 of the total time.
While I'm looking forward to your test results of other `AckGroupingOption` configs, I'm going to test the performance locally. Could you give the information of your test data? Including:
- The batch configs of the producer
- The message size
- The ACK type (individual or cumulative?)
BTW, the `flush` method of the ACK grouping tracker only does the following things:
1. Call `partitionConsumer#sendIndividualAck` or `partitionConsumer#sendCumulativeAck`, which just send the ACKs to the channel `eventsCh`
2. Remove some elements from the map.
I didn't see the `flushIndividualAcks` method from the flamegraph so it looks like it's not the bottle neck. /cc @shibd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on pull request #957: Support grouping ACK requests by time and size
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#issuecomment-1439749388
> Maybe we should set ackIndividualCh and ackCumulativeCh size > 0
@shibd What's the benefit of setting this channel size greater than 0? I think we don't need to cache these MessageIDs again in the channel. They are already cached in `cachedAcks`.
> Does Community has some Performance Test Data?
Currently not. Could you provide more information to diagnosis? For example, the prometheues metrics `ProcessingTime` records the processing time between a message is received and acknowledged. You can also adjust the `AckGroupingOption` to see if there are some improvements, e.g. increasing the `MaxTime` or `MaxSize`. @panszobe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on a diff in pull request #957: Support grouping ACK requests by time and size
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#discussion_r1112112794
##########
pulsar/ack_grouping_tracker.go:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package pulsar
+
+import (
+ "time"
+
+ "github.com/bits-and-blooms/bitset"
+)
+
+type ackGroupingTracker interface {
+ add(id MessageID)
+
+ addCumulative(id MessageID)
+
+ isDuplicate(id MessageID) bool
+
+ flush()
+
+ flushAndClean()
+
+ close()
+}
+
+type ackFlushType int
+
+const (
+ flushOnly ackFlushType = iota
+ flushAndClean
+ flushAndClose
+)
+
+func newAckGroupingTracker(options *AckGroupingOptions,
+ ackIndividual func(id MessageID),
+ ackCumulative func(id MessageID)) ackGroupingTracker {
+ if options == nil {
+ options = &AckGroupingOptions{
+ MaxSize: 1000,
+ MaxTime: 100 * time.Millisecond,
+ }
+ }
+
+ if options.MaxSize <= 1 {
+ return &immediateAckGroupingTracker{
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ }
+ }
+
+ c := &cachedAcks{
+ singleAcks: make([]MessageID, options.MaxSize),
+ pendingAcks: make(map[int64]*bitset.BitSet),
+ lastCumulativeAck: EarliestMessageID(),
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ ackList: func(ids []MessageID) {
+ // TODO: support ack a list of MessageIDs
+ for _, id := range ids {
+ ackIndividual(id)
+ }
+ },
+ }
+
+ timeout := time.NewTicker(time.Hour)
+ if options.MaxTime > 0 {
+ timeout = time.NewTicker(options.MaxTime)
+ } else {
+ timeout.Stop()
+ }
+ t := &timedAckGroupingTracker{
+ ackIndividualCh: make(chan MessageID),
+ ackCumulativeCh: make(chan MessageID),
+ duplicateIDCh: make(chan MessageID),
+ duplicateResultCh: make(chan bool),
+ flushCh: make(chan ackFlushType, 10),
Review Comment:
Since all flush operations are synchronous, I changed it to an unbuffered channel now. Let's see if any test would fail.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on pull request #957: Support grouping ACK requests by time and size
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#issuecomment-1441158571
> increasing MaxSize to 1000 and MaxTime to 1000,
The default MaxSize is 1000, which means every time 1000 messages are acknowledged, the ACK request will be sent. Increasing MaxTime to 1000 won't help because the receiving speed is far more than 1000 messages per second.
BTW, I just remembered an important thing that this PR does not support the list ACK. See https://github.com/apache/pulsar-client-go/blob/7d257b01b1216e0009495a4aaf83b1716ba458a9/pulsar/ack_grouping_tracker.go#L72. i.e. each time 1000 message IDs are flushed, 1000 ACK requests will be sent. So it does not make a difference.
I'm going to implement the list ACK soon and see if it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] panszobe commented on pull request #957: Support grouping ACK requests by time and size
Posted by "panszobe (via GitHub)" <gi...@apache.org>.
panszobe commented on PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#issuecomment-1441178404
> BTW, I just remembered an important thing that this PR does not support the list ACK. See
>
> https://github.com/apache/pulsar-client-go/blob/7d257b01b1216e0009495a4aaf83b1716ba458a9/pulsar/ack_grouping_tracker.go#L72
>
Emmm, I made a mistake while adjusting the options. As your diagnosis, I adjusted the options by increasing the two settings to 10000, which still didn't make a difference.
Maybe the break point is your new PR for supporting the list ACK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] shibd commented on pull request #957: Support grouping ACK requests by time and size
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#issuecomment-1439631259
> Does Community has some Performance Test Data?
>
> I used the latest version(v0.9.1-0.20230222003822-7d257b01b121) to test today, still found that which Test Case by EnableBatchIndexAcknowledgment=true consumed messages much more slower than EnableBatchIndexAcknowledgment=false, and could not catch up with the producer‘s production. As consuming from the same topic, consumers of JAVA Client by EnableBatchIndexAcknowledgment=true consumed messages smoothly, and backlog was not enlarged.
>
> I think that _Support grouping ACK_ feature is OK, but maybe _EnableBatchIndexAcknowledgment_ feature have much room for improvement.
Maybe we should set ackIndividualCh and ackCumulativeCh size > 0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] shibd commented on pull request #957: Support grouping ACK requests by time and size
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#issuecomment-1439790432
> @shibd What's the benefit of setting this channel size greater than 0? I think we don't need to cache these MessageIDs again in the channel. They are already cached in cachedAcks.
Just guessing. For example, if the channel has buffers while the thread is flushing data, then the add operation will not be blocked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] shibd commented on a diff in pull request #957: Support grouping ACK requests by time and size
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#discussion_r1112465551
##########
pulsar/ack_grouping_tracker.go:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package pulsar
+
+import (
+ "time"
+
+ "github.com/bits-and-blooms/bitset"
+)
+
+type ackGroupingTracker interface {
+ add(id MessageID)
+
+ addCumulative(id MessageID)
+
+ isDuplicate(id MessageID) bool
+
+ flush()
+
+ flushAndClean()
+
+ close()
+}
+
+type ackFlushType int
+
+const (
+ flushOnly ackFlushType = iota
+ flushAndClean
+ flushAndClose
+)
+
+func newAckGroupingTracker(options *AckGroupingOptions,
+ ackIndividual func(id MessageID),
+ ackCumulative func(id MessageID)) ackGroupingTracker {
+ if options == nil {
+ options = &AckGroupingOptions{
+ MaxSize: 1000,
+ MaxTime: 100 * time.Millisecond,
+ }
+ }
+
+ if options.MaxSize <= 1 {
+ return &immediateAckGroupingTracker{
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ }
+ }
+
+ c := &cachedAcks{
+ singleAcks: make([]MessageID, options.MaxSize),
+ pendingAcks: make(map[int64]*bitset.BitSet),
+ lastCumulativeAck: EarliestMessageID(),
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ ackList: func(ids []MessageID) {
+ // TODO: support ack a list of MessageIDs
+ for _, id := range ids {
+ ackIndividual(id)
+ }
+ },
+ }
+
+ timeout := time.NewTicker(time.Hour)
+ if options.MaxTime > 0 {
+ timeout = time.NewTicker(options.MaxTime)
+ } else {
+ timeout.Stop()
+ }
+ t := &timedAckGroupingTracker{
+ ackIndividualCh: make(chan MessageID),
+ ackCumulativeCh: make(chan MessageID),
+ duplicateIDCh: make(chan MessageID),
+ duplicateResultCh: make(chan bool),
+ flushCh: make(chan ackFlushType, 10),
+ waitFlushCh: make(chan bool),
+ }
+ go func() {
+ for {
+ select {
+ case id := <-t.ackIndividualCh:
+ if c.addAndCheckIfFull(id) {
+ c.flushIndividualAcks()
+ if options.MaxTime > 0 {
+ timeout.Reset(options.MaxTime)
+ }
+ }
+ case id := <-t.ackCumulativeCh:
+ c.tryUpdateLastCumulativeAck(id)
+ if options.MaxTime <= 0 {
+ c.flushCumulativeAck()
+ }
+ case id := <-t.duplicateIDCh:
+ t.duplicateResultCh <- c.isDuplicate(id)
+ case <-timeout.C:
+ c.flush()
+ case ackFlushType := <-t.flushCh:
+ timeout.Stop()
+ c.flush()
+ if ackFlushType == flushAndClean {
+ c.clean()
+ }
+ t.waitFlushCh <- true
+ if ackFlushType == flushAndClose {
+ return
+ }
+ }
+ }
+ }()
+ return t
+}
+
+type immediateAckGroupingTracker struct {
+ ackIndividual func(id MessageID)
+ ackCumulative func(id MessageID)
+}
+
+func (i *immediateAckGroupingTracker) add(id MessageID) {
+ i.ackIndividual(id)
+}
+
+func (i *immediateAckGroupingTracker) addCumulative(id MessageID) {
+ i.ackCumulative(id)
+}
+
+func (i *immediateAckGroupingTracker) isDuplicate(id MessageID) bool {
+ return false
+}
+
+func (i *immediateAckGroupingTracker) flush() {
+}
+
+func (i *immediateAckGroupingTracker) flushAndClean() {
+}
+
+func (i *immediateAckGroupingTracker) close() {
+}
+
+type cachedAcks struct {
+ singleAcks []MessageID
+ index int
+
+ // Key is the hash code of the ledger id and the netry id,
+ // Value is the bit set that represents which messages are acknowledged if the entry stores a batch.
+ // The bit 1 represents the message has been acknowledged, i.e. the bits "111" represents all messages
+ // in the batch whose batch size is 3 are not acknowledged.
+ // After the 1st message (i.e. batch index is 0) is acknowledged, the bits will become "011".
+ // Value is nil if the entry represents a single message.
+ pendingAcks map[int64]*bitset.BitSet
Review Comment:
I got it. thanks for your explain.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on a diff in pull request #957: Support grouping ACK requests by time and size
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#discussion_r1112651643
##########
pulsar/consumer_partition_test.go:
##########
@@ -37,6 +37,8 @@ func TestSingleMessageIDNoAckTracker(t *testing.T) {
metrics: newTestMetrics(),
decryptor: crypto.NewNoopDecryptor(),
}
+ pc.ackGroupingTracker = newAckGroupingTracker(&AckGroupingOptions{MaxSize: 0},
Review Comment:
This line is to fix the error that `pc.ackGroupingTracker` is `nil` in this test without the explicit initialization of `pc.ackGroupingTracker` because the `partitionConsumer` is created manually, not by the `Consumer`. The tests for different `AckGroupingOptions` are added into `ack_grouping_tracker_test.go`.
> MaxSize > 0
The default config (`MaxSize: 1000, MaxTime: 10ms`) has been verified in all existing tests that call acknowledge related methods.
> A Individual ack in a batch message.
TestBatchIndexAck
This PR has already changed 600+ lines. Adding these tests could make it very huge. What do you think of adding the different options into a single test like `TestBatchIndexAck`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on pull request #957: Support grouping ACK requests by time and size
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#issuecomment-1441234325
Okay. When I open a new PR, I will also perform a simple test in my local env to see if it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] shibd commented on a diff in pull request #957: Support grouping ACK requests by time and size
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#discussion_r1111720182
##########
pulsar/ack_grouping_tracker.go:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package pulsar
+
+import (
+ "time"
+
+ "github.com/bits-and-blooms/bitset"
+)
+
+type ackGroupingTracker interface {
+ add(id MessageID)
+
+ addCumulative(id MessageID)
+
+ isDuplicate(id MessageID) bool
+
+ flush()
+
+ flushAndClean()
+
+ close()
+}
+
+type ackFlushType int
+
+const (
+ flushOnly ackFlushType = iota
+ flushAndClean
+ flushAndClose
+)
+
+func newAckGroupingTracker(options *AckGroupingOptions,
+ ackIndividual func(id MessageID),
+ ackCumulative func(id MessageID)) ackGroupingTracker {
+ if options == nil {
+ options = &AckGroupingOptions{
+ MaxSize: 1000,
+ MaxTime: 100 * time.Millisecond,
+ }
+ }
+
+ if options.MaxSize <= 1 {
+ return &immediateAckGroupingTracker{
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ }
+ }
+
+ c := &cachedAcks{
+ singleAcks: make([]MessageID, options.MaxSize),
+ pendingAcks: make(map[int64]*bitset.BitSet),
+ lastCumulativeAck: EarliestMessageID(),
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ ackList: func(ids []MessageID) {
+ // TODO: support ack a list of MessageIDs
+ for _, id := range ids {
+ ackIndividual(id)
+ }
+ },
+ }
+
+ timeout := time.NewTicker(time.Hour)
+ if options.MaxTime > 0 {
+ timeout = time.NewTicker(options.MaxTime)
+ } else {
+ timeout.Stop()
+ }
+ t := &timedAckGroupingTracker{
+ ackIndividualCh: make(chan MessageID),
+ ackCumulativeCh: make(chan MessageID),
+ duplicateIDCh: make(chan MessageID),
+ duplicateResultCh: make(chan bool),
+ flushCh: make(chan ackFlushType, 10),
Review Comment:
Why only set `flushCh` is 10?
##########
pulsar/ack_grouping_tracker.go:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package pulsar
+
+import (
+ "time"
+
+ "github.com/bits-and-blooms/bitset"
+)
+
+type ackGroupingTracker interface {
+ add(id MessageID)
+
+ addCumulative(id MessageID)
+
+ isDuplicate(id MessageID) bool
+
+ flush()
+
+ flushAndClean()
+
+ close()
+}
+
+type ackFlushType int
+
+const (
+ flushOnly ackFlushType = iota
+ flushAndClean
+ flushAndClose
+)
+
+func newAckGroupingTracker(options *AckGroupingOptions,
+ ackIndividual func(id MessageID),
+ ackCumulative func(id MessageID)) ackGroupingTracker {
+ if options == nil {
+ options = &AckGroupingOptions{
+ MaxSize: 1000,
+ MaxTime: 100 * time.Millisecond,
+ }
+ }
+
+ if options.MaxSize <= 1 {
+ return &immediateAckGroupingTracker{
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ }
+ }
+
+ c := &cachedAcks{
+ singleAcks: make([]MessageID, options.MaxSize),
+ pendingAcks: make(map[int64]*bitset.BitSet),
+ lastCumulativeAck: EarliestMessageID(),
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ ackList: func(ids []MessageID) {
+ // TODO: support ack a list of MessageIDs
+ for _, id := range ids {
+ ackIndividual(id)
+ }
+ },
+ }
+
+ timeout := time.NewTicker(time.Hour)
+ if options.MaxTime > 0 {
+ timeout = time.NewTicker(options.MaxTime)
+ } else {
+ timeout.Stop()
+ }
+ t := &timedAckGroupingTracker{
+ ackIndividualCh: make(chan MessageID),
+ ackCumulativeCh: make(chan MessageID),
+ duplicateIDCh: make(chan MessageID),
+ duplicateResultCh: make(chan bool),
+ flushCh: make(chan ackFlushType, 10),
+ waitFlushCh: make(chan bool),
+ }
+ go func() {
+ for {
+ select {
+ case id := <-t.ackIndividualCh:
+ if c.addAndCheckIfFull(id) {
+ c.flushIndividualAcks()
+ if options.MaxTime > 0 {
+ timeout.Reset(options.MaxTime)
+ }
+ }
+ case id := <-t.ackCumulativeCh:
+ c.tryUpdateLastCumulativeAck(id)
+ if options.MaxTime <= 0 {
+ c.flushCumulativeAck()
+ }
+ case id := <-t.duplicateIDCh:
+ t.duplicateResultCh <- c.isDuplicate(id)
+ case <-timeout.C:
+ c.flush()
+ case ackFlushType := <-t.flushCh:
+ timeout.Stop()
+ c.flush()
+ if ackFlushType == flushAndClean {
+ c.clean()
+ }
+ t.waitFlushCh <- true
+ if ackFlushType == flushAndClose {
+ return
+ }
+ }
+ }
+ }()
+ return t
+}
+
+type immediateAckGroupingTracker struct {
+ ackIndividual func(id MessageID)
+ ackCumulative func(id MessageID)
+}
+
+func (i *immediateAckGroupingTracker) add(id MessageID) {
+ i.ackIndividual(id)
+}
+
+func (i *immediateAckGroupingTracker) addCumulative(id MessageID) {
+ i.ackCumulative(id)
+}
+
+func (i *immediateAckGroupingTracker) isDuplicate(id MessageID) bool {
+ return false
+}
+
+func (i *immediateAckGroupingTracker) flush() {
+}
+
+func (i *immediateAckGroupingTracker) flushAndClean() {
+}
+
+func (i *immediateAckGroupingTracker) close() {
+}
+
+type cachedAcks struct {
+ singleAcks []MessageID
+ index int
+
+ // Key is the hash code of the ledger id and the netry id,
+ // Value is the bit set that represents which messages are acknowledged if the entry stores a batch.
+ // The bit 1 represents the message has been acknowledged, i.e. the bits "111" represents all messages
+ // in the batch whose batch size is 3 are not acknowledged.
+ // After the 1st message (i.e. batch index is 0) is acknowledged, the bits will become "011".
+ // Value is nil if the entry represents a single message.
+ pendingAcks map[int64]*bitset.BitSet
Review Comment:
Why define `pendingAcks` here? Does it look like it's to judge `isDuplicate`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on a diff in pull request #957: Support grouping ACK requests by time and size
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#discussion_r1112739430
##########
pulsar/consumer_partition_test.go:
##########
@@ -37,6 +37,8 @@ func TestSingleMessageIDNoAckTracker(t *testing.T) {
metrics: newTestMetrics(),
decryptor: crypto.NewNoopDecryptor(),
}
+ pc.ackGroupingTracker = newAckGroupingTracker(&AckGroupingOptions{MaxSize: 0},
Review Comment:
I've added the test cases to `TestBatchIndexAck` and fix the bug that `AckGroupingOptions` is not exposed, PTAL again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] panszobe commented on pull request #957: Support grouping ACK requests by time and size
Posted by "panszobe (via GitHub)" <gi...@apache.org>.
panszobe commented on PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#issuecomment-1439903123
>
> > Does Community has some Performance Test Data?
>
> Currently not. Could you provide more information to diagnosis? For example, the prometheues metrics `ProcessingTime` records the processing time between a message is received and acknowledged. You can also adjust the `AckGroupingOption` to see if there are some improvements, e.g. increasing the `MaxTime` or `MaxSize`. A flamegraph pprof is also good for analysis. @panszobe
There are some prometheues metrics in Grafana Graph:



Version label stands for
enable_batch_master: the latest version and `EnableBatchIndexAcknowledgment` set true
disable_batch_master: the latest version and `EnableBatchIndexAcknowledgment` set false
disable_batch_v0.6.0: version 0.6.0 and `EnableBatchIndexAcknowledgment` not provided
As the metrics shows, receiving and acking messages for **enable_batch_master** is slower than other two versions, processing time metrics looks like it does't have a very large difference.
There are three flamegraph screenshot for **enable_batch_master**:
<img width="702" alt="image" src="https://user-images.githubusercontent.com/75996911/220611352-6b6ef805-f0c1-40df-baa5-b530fe988f88.png">
<img width="1116" alt="image" src="https://user-images.githubusercontent.com/75996911/220612260-fa312ba3-525e-4104-aa63-b159ce763013.png">
<img width="988" alt="image" src="https://user-images.githubusercontent.com/75996911/220611440-cde21f03-1895-40c9-99a6-73dde7efc1f4.png">
For *disable_batch_master**:
<img width="646" alt="image" src="https://user-images.githubusercontent.com/75996911/220612791-288e284f-98ef-418f-95cc-6a6134a851cc.png">
<img width="704" alt="image" src="https://user-images.githubusercontent.com/75996911/220612702-08c8d535-8d06-456f-91c1-9236f6f3a918.png">
As above, compare to **disable_batch_master**, `runEventsLoop` and `internalSendRequest` takes much more time.
Next I will try to adjust `AckGroupingOption` to see if there are some improvements.
Please have a look, if you can diagnose what's wrong from something provided above is better.
@BewareMyPower @shibd
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on pull request #957: Support grouping ACK requests by time and size
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#issuecomment-1451854264
@panszobe I'm working on this issue currently. But since I'm busy on something else this week, the PR was not opened. If you mean https://github.com/apache/pulsar-client-go/pull/968, then it's not related to this issue. It's just a problem I found when I wrote the code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] BewareMyPower commented on a diff in pull request #957: Support grouping ACK requests by time and size
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on code in PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#discussion_r1112101120
##########
pulsar/ack_grouping_tracker.go:
##########
@@ -0,0 +1,298 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package pulsar
+
+import (
+ "time"
+
+ "github.com/bits-and-blooms/bitset"
+)
+
+type ackGroupingTracker interface {
+ add(id MessageID)
+
+ addCumulative(id MessageID)
+
+ isDuplicate(id MessageID) bool
+
+ flush()
+
+ flushAndClean()
+
+ close()
+}
+
+type ackFlushType int
+
+const (
+ flushOnly ackFlushType = iota
+ flushAndClean
+ flushAndClose
+)
+
+func newAckGroupingTracker(options *AckGroupingOptions,
+ ackIndividual func(id MessageID),
+ ackCumulative func(id MessageID)) ackGroupingTracker {
+ if options == nil {
+ options = &AckGroupingOptions{
+ MaxSize: 1000,
+ MaxTime: 100 * time.Millisecond,
+ }
+ }
+
+ if options.MaxSize <= 1 {
+ return &immediateAckGroupingTracker{
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ }
+ }
+
+ c := &cachedAcks{
+ singleAcks: make([]MessageID, options.MaxSize),
+ pendingAcks: make(map[int64]*bitset.BitSet),
+ lastCumulativeAck: EarliestMessageID(),
+ ackIndividual: ackIndividual,
+ ackCumulative: ackCumulative,
+ ackList: func(ids []MessageID) {
+ // TODO: support ack a list of MessageIDs
+ for _, id := range ids {
+ ackIndividual(id)
+ }
+ },
+ }
+
+ timeout := time.NewTicker(time.Hour)
+ if options.MaxTime > 0 {
+ timeout = time.NewTicker(options.MaxTime)
+ } else {
+ timeout.Stop()
+ }
+ t := &timedAckGroupingTracker{
+ ackIndividualCh: make(chan MessageID),
+ ackCumulativeCh: make(chan MessageID),
+ duplicateIDCh: make(chan MessageID),
+ duplicateResultCh: make(chan bool),
+ flushCh: make(chan ackFlushType, 10),
Review Comment:
I just set it with a large value in case `flush` could be called in different threads. I will try figuring out a more proper value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] shibd merged pull request #957: Support grouping ACK requests by time and size
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd merged PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar-client-go] shibd commented on a diff in pull request #957: Support grouping ACK requests by time and size
Posted by "shibd (via GitHub)" <gi...@apache.org>.
shibd commented on code in PR #957:
URL: https://github.com/apache/pulsar-client-go/pull/957#discussion_r1112469747
##########
pulsar/consumer_partition_test.go:
##########
@@ -37,6 +37,8 @@ func TestSingleMessageIDNoAckTracker(t *testing.T) {
metrics: newTestMetrics(),
decryptor: crypto.NewNoopDecryptor(),
}
+ pc.ackGroupingTracker = newAckGroupingTracker(&AckGroupingOptions{MaxSize: 0},
Review Comment:
Can you add more unit tests? Such as:
- MaxSize > 0
- A Individual ack in a batch message.
- Other Similar to: [CPP test](https://github.com/apache/pulsar-client-cpp/blob/06eab69f4caedc1962a21ac52e5c19de868609cd/tests/AcknowledgeTest.cc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org