You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/02/28 13:55:40 UTC

[GitHub] [pulsar] Jason918 opened a new pull request #14494: [PIP-74] Support auto scaled consumer receiver queue

Jason918 opened a new pull request #14494:
URL: https://github.com/apache/pulsar/pull/14494


   <!--
   ### Contribution Checklist
     
     - Name the pull request in the form "[Issue XYZ][component] Title of the pull request", where *XYZ* should be replaced by the actual issue number.
       Skip *Issue XYZ* if there is no associated github issue for this pull request.
       Skip *component* if you are unsure about which is the best component. E.g. `[docs] Fix typo in produce method`.
   
     - Fill out the template below to describe the changes contributed by the pull request. That will give reviewers the context they need to do the review.
     
     - Each pull request should address only one issue, not mix up code from multiple issues.
     
     - Each commit in the pull request has a meaningful commit message
   
     - Once all items of the checklist are addressed, remove the above text and this checklist, leaving only the filled out template below.
   
   **(The sections below can be removed for hotfixes of typos)**
   -->
   
   
   
   ### Motivation
   
   
   This is part of the work for [PIP 74](https://github.com/apache/pulsar/wiki/PIP-74%3A-Pulsar-client-memory-limits)
   We need auto scale `currentReceiverQueue` to control client memory.
   
   ### Modifications
   
   Add optional  autoScaledReceiverQueueSizeEnabled for consumer client.
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   This change added tests and can be verified as follows:
   
   *(example:)*
     - org.apache.pulsar.client.impl.AutoScaledReceiverQueueSizeTest
     
   ### Does this pull request potentially affect one of the following parts:
   
   *If `yes` was chosen, please highlight the changes*
   
     - Dependencies (does it add or upgrade a dependency): (no)
     - The public API: (no)
     - The schema: (no)
     - The default values of configurations: (no)
     - The wire protocol: (no)
     - The rest endpoints: (no)
     - The admin cli options: (no)
     - Anything that affects deployment: (no)
   
   ### Documentation
   
   Check the box below and label this PR (if you have committer privilege).
   
   Need to update docs? 
     
   - [x] `no-need-doc` 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] eolivelli commented on a change in pull request #14494: [PIP-74] Support auto scaled consumer receiver queue

Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #14494:
URL: https://github.com/apache/pulsar/pull/14494#discussion_r834993977



##########
File path: pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceConsumer.java
##########
@@ -591,6 +605,28 @@ public static void main(String[] args) throws Exception {
                     reportHistogram.getValueAtPercentile(99), reportHistogram.getValueAtPercentile(99.9),
                     reportHistogram.getValueAtPercentile(99.99), reportHistogram.getMaxValue());
 
+            if (arguments.autoScaledReceiverQueueSize && log.isDebugEnabled() && qRecorder != null) {
+                qHistogram = qRecorder.getIntervalHistogram(qHistogram);
+                log.debug("ReceiverQueueUsage: cnt={},mean={}, min={},max={},25pct={},50pct={},75pct={}",

Review comment:
       Why 'debug'?
   It looks like this is a very useful information 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Jason918 edited a comment on pull request #14494: [PIP-74] Support auto scaled consumer receiver queue

Posted by GitBox <gi...@apache.org>.
Jason918 edited a comment on pull request #14494:
URL: https://github.com/apache/pulsar/pull/14494#issuecomment-1072991108


   Here is a test with `bin/pulsar-perf` on local standalone server to confirm the effect of this PR.
   
   ## Single topic consumer
   ```
   bin/pulsar-perf consume -aq -q 1000000 persistent://public/default/test
   bin/pulsar-perf produce -r $RATE -s 128 -bm 0 -time 60 persistent://public/default/test
   ```
   | Rate | Final receiver queue size |
   | - | - |
   | 1 | 2 |
   | 10 | 2 |
   | 100 | 4 |
   | 1000 | 32 |
   | 10000 | 512 |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Jason918 closed pull request #14494: [PIP-74] Support auto scaled consumer receiver queue

Posted by GitBox <gi...@apache.org>.
Jason918 closed pull request #14494:
URL: https://github.com/apache/pulsar/pull/14494


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Jason918 commented on pull request #14494: [PIP-74] Support auto scaled consumer receiver queue

Posted by GitBox <gi...@apache.org>.
Jason918 commented on pull request #14494:
URL: https://github.com/apache/pulsar/pull/14494#issuecomment-1085342645


   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Jason918 commented on pull request #14494: [PIP-74] Support auto scaled consumer receiver queue

Posted by GitBox <gi...@apache.org>.
Jason918 commented on pull request #14494:
URL: https://github.com/apache/pulsar/pull/14494#issuecomment-1055080218


   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Jason918 edited a comment on pull request #14494: [PIP-74] Support auto scaled consumer receiver queue

Posted by GitBox <gi...@apache.org>.
Jason918 edited a comment on pull request #14494:
URL: https://github.com/apache/pulsar/pull/14494#issuecomment-1072991108


   Here is a test with `bin/pulsar-perf` on local standalone server to confirm the effect of this PR.
   
   ## Non-partitioned topic consumer
   ```
   bin/pulsar-perf consume -aq -q 1000000 persistent://public/default/test
   bin/pulsar-perf produce -r $RATE -s 128 -bm 0 -time 60 persistent://public/default/test
   ```
   | RATE | Final receiver queue size |
   | - | - |
   | 1 | 2 |
   | 10 | 2 |
   | 100 | 4 |
   | 1000 | 32 |
   | 10000 | 512 |
   
   ## 3-partitioned topic consumer
   ```
   bin/pulsar-perf consume -aq -q 1000000 persistent://public/default/multi-partitions
   bin/pulsar-perf produce -r $RATE -s 128 -bm 0 -time 60 -np 3 persistent://public/default/multi-partitions
   ```
   | RATE | Final receiver queue size (MultiTopicConsumer)| Sub-consumers receiver queue size |
   | - | - | - |
   | 1 | 3 | 1,1,1 |
   | 10 | 3 | 1,1,1 |
   | 100 | 6 | 2,1,1 |
   | 1000 | 12 | 16,32,16 |
   | 5000 | 48 | 128,128,128 |
   | 10000 | 96 | 1024,512,512 |
   | 20000 | 192 | 1024,512,512 |
   | 40000 | 384 | 1024,1024,1024 |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Jason918 commented on a change in pull request #14494: [PIP-74] Support auto scaled consumer receiver queue

Posted by GitBox <gi...@apache.org>.
Jason918 commented on a change in pull request #14494:
URL: https://github.com/apache/pulsar/pull/14494#discussion_r836004198



##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
##########
@@ -411,10 +411,28 @@ public UnAckedMessageTracker getUnAckedMessageTracker() {
         return unsubscribeFuture;
     }
 
+    @Override
+    public void initReceiverQueueSize() {
+        if (conf.isAutoScaledReceiverQueueSizeEnabled()) {
+            // turn on autoScaledReceiverQueueSize
+            int size = Math.min(INITIAL_RECEIVER_QUEUE_SIZE, maxReceiverQueueSize);
+            if (batchReceivePolicy.getMaxNumMessages() > 0) {
+                // consumerImpl may store (half-1) permits locally.
+                size = Math.max(size, 2 * batchReceivePolicy.getMaxNumMessages() - 2);

Review comment:
       > If `batchReceivePolicy.getMaxNumMessages()` is close or equal to `maxReceiverQueueSize`, will `size` exceed the limit of `ConsumerBuilderImpl.receiverQueueSize()`?
   
   @aloyszhang Yes, it should exceed the limit if we expect it to consume normally. And this should be a pre-existing issue that consumer would get stuck if `batchReceivePolicy.getMaxNumMessages()` is equal to `receiverQueueSize`
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Jason918 closed pull request #14494: [PIP-74] Support auto scaled consumer receiver queue

Posted by GitBox <gi...@apache.org>.
Jason918 closed pull request #14494:
URL: https://github.com/apache/pulsar/pull/14494


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Jason918 commented on pull request #14494: [PIP-74] Support auto scaled consumer receiver queue

Posted by GitBox <gi...@apache.org>.
Jason918 commented on pull request #14494:
URL: https://github.com/apache/pulsar/pull/14494#issuecomment-1085618536


   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] aloyszhang commented on a change in pull request #14494: [PIP-74] Support auto scaled consumer receiver queue

Posted by GitBox <gi...@apache.org>.
aloyszhang commented on a change in pull request #14494:
URL: https://github.com/apache/pulsar/pull/14494#discussion_r835703766



##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
##########
@@ -411,10 +411,28 @@ public UnAckedMessageTracker getUnAckedMessageTracker() {
         return unsubscribeFuture;
     }
 
+    @Override
+    public void initReceiverQueueSize() {
+        if (conf.isAutoScaledReceiverQueueSizeEnabled()) {
+            // turn on autoScaledReceiverQueueSize
+            int size = Math.min(INITIAL_RECEIVER_QUEUE_SIZE, maxReceiverQueueSize);
+            if (batchReceivePolicy.getMaxNumMessages() > 0) {
+                // consumerImpl may store (half-1) permits locally.
+                size = Math.max(size, 2 * batchReceivePolicy.getMaxNumMessages() - 2);

Review comment:
       If `batchReceivePolicy.getMaxNumMessages()` is close or equal to `maxReceiverQueueSize`, will `size` exceed the  limit of `ConsumerBuilderImpl.receiverQueueSize()`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Jason918 commented on pull request #14494: [PIP-74] Support auto scaled consumer receiver queue

Posted by GitBox <gi...@apache.org>.
Jason918 commented on pull request #14494:
URL: https://github.com/apache/pulsar/pull/14494#issuecomment-1072991108


   Here is a test with `bin/pulsar-perf` on local standalone server to confirm the effect of this PR.
   
   ## Single partition consumer
   ```
   bin/pulsar-perf consume -aq -q 1000000 persistent://public/default/test
   bin/pulsar-perf produce -r $RATE -s 128 -bm 0 -time 60 persistent://public/default/test
   ```
   | Rate | Final receiver queue size |
   | - | - |
   | 1 | 2 |
   | 10 | 2 |
   | 100 | 4 |
   | 1000 | 32 |
   | 10000 | 512 |


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Jason918 commented on pull request #14494: [PIP-74] Support auto scaled consumer receiver queue

Posted by GitBox <gi...@apache.org>.
Jason918 commented on pull request #14494:
URL: https://github.com/apache/pulsar/pull/14494#issuecomment-1073020762


   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [pulsar] Jason918 commented on a change in pull request #14494: [PIP-74] Support auto scaled consumer receiver queue

Posted by GitBox <gi...@apache.org>.
Jason918 commented on a change in pull request #14494:
URL: https://github.com/apache/pulsar/pull/14494#discussion_r836002989



##########
File path: pulsar-testclient/src/main/java/org/apache/pulsar/testclient/PerformanceConsumer.java
##########
@@ -591,6 +605,28 @@ public static void main(String[] args) throws Exception {
                     reportHistogram.getValueAtPercentile(99), reportHistogram.getValueAtPercentile(99.9),
                     reportHistogram.getValueAtPercentile(99.99), reportHistogram.getMaxValue());
 
+            if (arguments.autoScaledReceiverQueueSize && log.isDebugEnabled() && qRecorder != null) {
+                qHistogram = qRecorder.getIntervalHistogram(qHistogram);
+                log.debug("ReceiverQueueUsage: cnt={},mean={}, min={},max={},25pct={},50pct={},75pct={}",

Review comment:
       > Why 'debug'? It looks like this is a very useful information
   
   I think this histogram info is too detail for performance test.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org