You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/03/05 08:12:25 UTC

[GitHub] [pulsar] Jason918 opened a new pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Jason918 opened a new pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566


   ### Motivation
   
   Currently, if we subscribe new partitions/topics when multiTopicConsumer is paused, the sub consumers will start in non-pause mode and then changed to pause. So there will be some chance that the Flow command still can be sent to brokers.
   
   ### Modifications
   
   Set `startPause` config when creating sub consumers.
   
   ### Verifying this change
   
   - [ ] Make sure that the change passes the CI checks.
   
   This change is already covered by existing tests, such as 
   - org.apache.pulsar.client.api.PartitionedProducerConsumerTest#testPauseAndResume
   - org.apache.pulsar.client.api.SimpleProducerConsumerTest#testPauseAndResume
   - org.apache.pulsar.client.impl.ZeroQueueSizeTest#testPauseAndResume
   ### Does this pull request potentially affect one of the following parts:
   
   *If `yes` was chosen, please highlight the changes*
   
     - Dependencies (does it add or upgrade a dependency): (no)
     - The public API: (no)
     - The schema: (no)
     - The default values of configurations: (no)
     - The wire protocol: (no)
     - The rest endpoints: (no)
     - The admin cli options: (no)
     - Anything that affects deployment: (no)
   
   ### Documentation
   
   Check the box below and label this PR (if you have committer privilege).
   
   Need to update docs? 
     
   - [x] `no-need-doc` 
   
   Internal optimization.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Jason918 commented on pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
Jason918 commented on pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#issuecomment-1072025973


   > After creating a new `ConsumerImpl` for a topic partition, before sending `FLOW` command, client will check
   > 
   > ```java
   > if (!(firstTimeConnect && hasParentConsumer) && getCurrentReceiverQueueSize() != 0) {
   >                     increaseAvailablePermits(cnx, getCurrentReceiverQueueSize());
   >                 }
   > ```
   > 
   > `hasParentConsumer` is always `true` for multiTopicConsumer, so the new created `ConsumerImpl` will not send `FLOW` command to broker.
   
   Great point. I have updated the motivation description.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] aloyszhang commented on a change in pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
aloyszhang commented on a change in pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#discussion_r826691462



##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
##########
@@ -1036,6 +1037,8 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
                         synchronized (pauseMutex) {
                             if (paused) {

Review comment:
       I think maybe we can remove the `synchronized (pauseMutex)` too




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] aloyszhang commented on a change in pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
aloyszhang commented on a change in pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#discussion_r826690622



##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
##########
@@ -1063,6 +1067,8 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
                     synchronized (pauseMutex) {
                         if (paused) {

Review comment:
       Ditto




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] aloyszhang edited a comment on pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#issuecomment-1067689183


   @Jason918 
   After creating a new `ConsumerImpl` for a topic partition, before sending `FLOW` command, client will check 
   ```java
   if (!(firstTimeConnect && hasParentConsumer) && getCurrentReceiverQueueSize() != 0) {
                       increaseAvailablePermits(cnx, getCurrentReceiverQueueSize());
                   }
   ```
   `hasParentConsumer` is always `true` for multiTopicConsumer, so the new created `ConsumerImpl` will not send `FLOW` command to broker.
   
   BTW, [#11974](https://github.com/apache/pulsar/pull/11974) introduce `startPaused` for creating new `ConsumerImpl`, so this pull request is a nice optimize for me. 
   
   Left a comment, PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] aloyszhang commented on a change in pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
aloyszhang commented on a change in pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#discussion_r829779365



##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
##########
@@ -1036,6 +1037,8 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
                         synchronized (pauseMutex) {
                             if (paused) {

Review comment:
       make sense to me




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] aloyszhang commented on a change in pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
aloyszhang commented on a change in pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#discussion_r826690414



##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
##########
@@ -1036,6 +1037,8 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
                         synchronized (pauseMutex) {
                             if (paused) {

Review comment:
       If we create a new consumer with `pause` state, we don't need to pause it again.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Jason918 commented on a change in pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
Jason918 commented on a change in pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#discussion_r829690543



##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
##########
@@ -1036,6 +1037,8 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
                         synchronized (pauseMutex) {
                             if (paused) {

Review comment:
       > I think maybe we can remove the `synchronized (pauseMutex)` too
   
   Not really, if the pause state changes between after 1038 and before line 1043, pause state of this newConsumer will be wrong.
   
   > If we create a new consumer with `pause` state, we don't need to pause it again.
   
   Yes, but current implementation of `newConsumer.pause()` and `newConsumer.resume()` is idempotent and handles the case when it's called repeatedly. I think the code is more clear this way. WDYT? @aloyszhang 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] aloyszhang commented on pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
aloyszhang commented on pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#issuecomment-1067689183


   @Jason918 
   After creating a new `ConsumerImpl` for a topic partition, before sending `FLOW` command, client will check 
   ```java
   if (!(firstTimeConnect && hasParentConsumer) && getCurrentReceiverQueueSize() != 0) {
                       increaseAvailablePermits(cnx, getCurrentReceiverQueueSize());
                   }
   ```
   `hasParentConsumer` is always `true` for multiTopicConsumer, so the new created will not send `FLOW` command to broker.
   
   BTW, [#](https://github.com/apache/pulsar/pull/11974) introduce `startPaused` for creating new `ConsumerImpl`, so this pull request is a nice optimize for me. And left a comment, PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] aloyszhang edited a comment on pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#issuecomment-1067689183


   @Jason918 
   After creating a new `ConsumerImpl` for a topic partition, before sending `FLOW` command, client will check 
   ```java
   if (!(firstTimeConnect && hasParentConsumer) && getCurrentReceiverQueueSize() != 0) {
                       increaseAvailablePermits(cnx, getCurrentReceiverQueueSize());
                   }
   ```
   `hasParentConsumer` is always `true` for multiTopicConsumer, so the new created will not send `FLOW` command to broker.
   
   BTW, [#11974](https://github.com/apache/pulsar/pull/11974) introduce `startPaused` for creating new `ConsumerImpl`, so this pull request is a nice optimize for me. 
   
   Left a comment, PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Jason918 commented on pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
Jason918 commented on pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#issuecomment-1060228741


   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] aloyszhang commented on a change in pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
aloyszhang commented on a change in pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#discussion_r829779365



##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
##########
@@ -1036,6 +1037,8 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
                         synchronized (pauseMutex) {
                             if (paused) {

Review comment:
       make sense to me




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Jason918 commented on a change in pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
Jason918 commented on a change in pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#discussion_r829690543



##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
##########
@@ -1036,6 +1037,8 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
                         synchronized (pauseMutex) {
                             if (paused) {

Review comment:
       > I think maybe we can remove the `synchronized (pauseMutex)` too
   
   Not really, if the pause state changes between after 1038 and before line 1043, pause state of this newConsumer will be wrong.
   
   > If we create a new consumer with `pause` state, we don't need to pause it again.
   
   Yes, but current implementation of `newConsumer.pause()` and `newConsumer.resume()` is idempotent and handles the case when it's called repeatedly. I think the code is more clear this way. WDYT? @aloyszhang 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] aloyszhang edited a comment on pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
aloyszhang edited a comment on pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#issuecomment-1067689183


   @Jason918 
   After creating a new `ConsumerImpl` for a topic partition, before sending `FLOW` command, client will check 
   ```java
   if (!(firstTimeConnect && hasParentConsumer) && getCurrentReceiverQueueSize() != 0) {
                       increaseAvailablePermits(cnx, getCurrentReceiverQueueSize());
                   }
   ```
   `hasParentConsumer` is always `true` for multiTopicConsumer, so the new created will not send `FLOW` command to broker.
   
   BTW, [#11974](https://github.com/apache/pulsar/pull/11974) introduce `startPaused` for creating new `ConsumerImpl`, so this pull request is a nice optimize for me. And left a comment, PTAL.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] aloyszhang commented on a change in pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
aloyszhang commented on a change in pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#discussion_r826691462



##########
File path: pulsar-client/src/main/java/org/apache/pulsar/client/impl/MultiTopicsConsumerImpl.java
##########
@@ -1036,6 +1037,8 @@ private void doSubscribeTopicPartitions(Schema<T> schema,
                         synchronized (pauseMutex) {
                             if (paused) {

Review comment:
       I think maybe we even can remove the `synchronized (pauseMutex)`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Jason918 commented on pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
Jason918 commented on pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#issuecomment-1072025973


   > After creating a new `ConsumerImpl` for a topic partition, before sending `FLOW` command, client will check
   > 
   > ```java
   > if (!(firstTimeConnect && hasParentConsumer) && getCurrentReceiverQueueSize() != 0) {
   >                     increaseAvailablePermits(cnx, getCurrentReceiverQueueSize());
   >                 }
   > ```
   > 
   > `hasParentConsumer` is always `true` for multiTopicConsumer, so the new created `ConsumerImpl` will not send `FLOW` command to broker.
   
   Great point. I have updated the motivation description.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Jason918 commented on pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
Jason918 commented on pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#issuecomment-1059883345


   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Jason918 commented on pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
Jason918 commented on pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566#issuecomment-1060538462


   /pulsarbot run-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Jason918 merged pull request #14566: [client] Optimize pause when creating sub consumers in multi-topic consumer

Posted by GitBox <gi...@apache.org>.
Jason918 merged pull request #14566:
URL: https://github.com/apache/pulsar/pull/14566


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org