You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/02/18 06:00:15 UTC

[GitHub] [pulsar] mattisonchao opened a new pull request #14367: [Issue 14362] Fix Broker HealthCheck Endpoint Exposes Race Conditions

mattisonchao opened a new pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367


   Fixes #14362
   
   Master Issue: #14362
   
   ### Motivation
   
   See #14362 
   
   According to relative fix #7724, I found that the root cause of this part is that we are using some async methods and not waiting for this call to complete.
   For this reason, after #14091, I don't think we need to delete all subscriptions anymore. Then the race condition disappears.
   
   ![image](https://user-images.githubusercontent.com/74767115/154626609-d67d3a58-f99d-4fb4-80f2-8f275a12e42a.png)
   
   ### Modifications
   
   - Remove ``delete all subscriptions logic``.
   - Add multiple thread tests to ensure we don't have a legacy subscription.
   
   ### Verifying this change
   
   - [x] Make sure that the change passes the CI checks.
   
   ### Documentation
   
   - [x] `no-need-doc` 
     
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] eolivelli commented on a change in pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#discussion_r817505653



##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
##########
@@ -338,39 +336,60 @@ public void healthCheck(@Suspended AsyncResponse asyncResponse,
         NamespaceName namespaceName = (topicVersion == TopicVersion.V2)
                 ? NamespaceService.getHeartbeatNamespaceV2(pulsar().getAdvertisedAddress(), pulsar().getConfiguration())
                 : NamespaceService.getHeartbeatNamespace(pulsar().getAdvertisedAddress(), pulsar().getConfiguration());
-        String topicName = String.format("persistent://%s/%s", namespaceName, HEALTH_CHECK_TOPIC_SUFFIX);
+        final String topicName = String.format("persistent://%s/%s", namespaceName, HEALTH_CHECK_TOPIC_SUFFIX);
         LOG.info("[{}] Running healthCheck with topic={}", clientAppId(), topicName);
-        String messageStr = UUID.randomUUID().toString();
+        final String messageStr = UUID.randomUUID().toString();
+        final String subscriptionName = "healthCheck-" + messageStr;
         // create non-partitioned topic manually and close the previous reader if present.
         return pulsar().getBrokerService().getTopic(topicName, true)
-                // check and clean all subscriptions
                 .thenCompose(topicOptional -> {
                     if (!topicOptional.isPresent()) {
                         LOG.error("[{}] Fail to run health check while get topic {}. because get null value.",
                                 clientAppId(), topicName);
-                        throw new RestException(Status.NOT_FOUND, "Topic [{}] not found after create.");
+                        throw new RestException(Status.NOT_FOUND,
+                                String.format("Topic [%s] not found after create.", topicName));
                     }
-                    Topic topic = topicOptional.get();
-                    // clean all subscriptions
-                    return FutureUtil.waitForAll(topic.getSubscriptions().values()
-                            .stream().map(Subscription::deleteForcefully).collect(Collectors.toList()))
-                            .thenApply(__ -> topic);
-                }).thenCompose(topic -> {
                     try {
                         PulsarClient client = pulsar().getClient();
                         return client.newProducer(Schema.STRING).topic(topicName).createAsync()
-                                        .thenCombine(client.newReader(Schema.STRING).topic(topicName)
+                                .thenCombine(client.newReader(Schema.STRING).topic(topicName)
+                                        .subscriptionName(subscriptionName)
                                         .startMessageId(MessageId.latest).createAsync(), (producer, reader) ->
-                                                        producer.sendAsync(messageStr).thenCompose(__ ->
-                                                                healthCheckRecursiveReadNext(reader, messageStr))
-                                                        .thenCompose(__ -> {
-                                                            List<CompletableFuture<Void>> closeFutures =
-                                                                    new ArrayList<>();
-                                                            closeFutures.add(producer.closeAsync());
-                                                            closeFutures.add(reader.closeAsync());
-                                                            return FutureUtil.waitForAll(closeFutures);
-                                                        })
-                                        ).thenAccept(ignore -> {});
+                                        producer.sendAsync(messageStr)
+                                                .thenCompose(__ -> healthCheckRecursiveReadNext(reader, messageStr))

Review comment:
       are you sure that the 'handle' will run even if there is a problem in "sendAsync" ?
   my understanding is that "thenCompose" is triggered only in case of success.

##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
##########
@@ -338,39 +336,60 @@ public void healthCheck(@Suspended AsyncResponse asyncResponse,
         NamespaceName namespaceName = (topicVersion == TopicVersion.V2)
                 ? NamespaceService.getHeartbeatNamespaceV2(pulsar().getAdvertisedAddress(), pulsar().getConfiguration())
                 : NamespaceService.getHeartbeatNamespace(pulsar().getAdvertisedAddress(), pulsar().getConfiguration());
-        String topicName = String.format("persistent://%s/%s", namespaceName, HEALTH_CHECK_TOPIC_SUFFIX);
+        final String topicName = String.format("persistent://%s/%s", namespaceName, HEALTH_CHECK_TOPIC_SUFFIX);
         LOG.info("[{}] Running healthCheck with topic={}", clientAppId(), topicName);
-        String messageStr = UUID.randomUUID().toString();
+        final String messageStr = UUID.randomUUID().toString();
+        final String subscriptionName = "healthCheck-" + messageStr;
         // create non-partitioned topic manually and close the previous reader if present.
         return pulsar().getBrokerService().getTopic(topicName, true)
-                // check and clean all subscriptions
                 .thenCompose(topicOptional -> {
                     if (!topicOptional.isPresent()) {
                         LOG.error("[{}] Fail to run health check while get topic {}. because get null value.",
                                 clientAppId(), topicName);
-                        throw new RestException(Status.NOT_FOUND, "Topic [{}] not found after create.");
+                        throw new RestException(Status.NOT_FOUND,
+                                String.format("Topic [%s] not found after create.", topicName));
                     }
-                    Topic topic = topicOptional.get();
-                    // clean all subscriptions
-                    return FutureUtil.waitForAll(topic.getSubscriptions().values()
-                            .stream().map(Subscription::deleteForcefully).collect(Collectors.toList()))
-                            .thenApply(__ -> topic);
-                }).thenCompose(topic -> {
                     try {
                         PulsarClient client = pulsar().getClient();
                         return client.newProducer(Schema.STRING).topic(topicName).createAsync()
-                                        .thenCombine(client.newReader(Schema.STRING).topic(topicName)
+                                .thenCombine(client.newReader(Schema.STRING).topic(topicName)
+                                        .subscriptionName(subscriptionName)
                                         .startMessageId(MessageId.latest).createAsync(), (producer, reader) ->
-                                                        producer.sendAsync(messageStr).thenCompose(__ ->
-                                                                healthCheckRecursiveReadNext(reader, messageStr))
-                                                        .thenCompose(__ -> {
-                                                            List<CompletableFuture<Void>> closeFutures =
-                                                                    new ArrayList<>();
-                                                            closeFutures.add(producer.closeAsync());
-                                                            closeFutures.add(reader.closeAsync());
-                                                            return FutureUtil.waitForAll(closeFutures);
-                                                        })
-                                        ).thenAccept(ignore -> {});
+                                        producer.sendAsync(messageStr)
+                                                .thenCompose(__ -> healthCheckRecursiveReadNext(reader, messageStr))
+                                                .handle((unused, exception) -> {
+                                                    // no matter exception or success, we still need to
+                                                    // close producer/reader
+                                                    List<CompletableFuture<Void>> closeFutures = new ArrayList<>();
+                                                    closeFutures.add(producer.closeAsync());
+                                                    closeFutures.add(reader.closeAsync());

Review comment:
       if the reader creation fails this 'reader' variable is null
   
   so we are missing the handling of the failure of the Reader creation, in that case we will miss to close the producer




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] michaeljmarshall commented on pull request #14367: [Issue 14362] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1043989288


   Thanks for working on this @mattisonchao!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Technoboy- commented on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
Technoboy- commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1052116565


   Another issue is that :  
   Due to the health check topic being deleted by GC, #13816 was involved.  So health check topic is being system topic. 
   The system topic will do the compaction. but the message sent to the health check topic without a key. So the compact action is never meaningful.  So we'd better optimize the compact part to exclude the health topic.
   What do you think? @codelipenghui @mattisonchao @michaeljmarshall 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on a change in pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#discussion_r815253206



##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
##########
@@ -343,41 +350,81 @@ public void healthCheck(@Suspended AsyncResponse asyncResponse,
         String messageStr = UUID.randomUUID().toString();
         // create non-partitioned topic manually and close the previous reader if present.
         return pulsar().getBrokerService().getTopic(topicName, true)
-                // check and clean all subscriptions
                 .thenCompose(topicOptional -> {
                     if (!topicOptional.isPresent()) {
                         LOG.error("[{}] Fail to run health check while get topic {}. because get null value.",
                                 clientAppId(), topicName);
                         throw new RestException(Status.NOT_FOUND, "Topic [{}] not found after create.");

Review comment:
       Good catch!
   Fixed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] michaeljmarshall commented on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1058488409


   I will review this within the next couple hours, thank you for your work @mattisonchao!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] gaoran10 commented on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
gaoran10 commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1066597354


   The https://github.com/apache/pulsar/pull/14658 cherry-pick this PR to `branch-2.9`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] eolivelli commented on a change in pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#discussion_r815492492



##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiHealthCheckTest.java
##########
@@ -55,16 +61,91 @@ public void cleanup() throws Exception {
 
     @Test
     public void testHealthCheckup() throws Exception {
-        admin.brokers().healthcheck();
+        final int times = 30;
+        new Thread(() -> {

Review comment:
       Can we ensure that this Thread is done in the end of the trst? Otherwise we are leaking resources. 
   I would use the Executor provided by the Pulsar service  to run this activity

##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiHealthCheckTest.java
##########
@@ -55,16 +61,91 @@ public void cleanup() throws Exception {
 
     @Test
     public void testHealthCheckup() throws Exception {
-        admin.brokers().healthcheck();
+        final int times = 30;
+        new Thread(() -> {
+            for (int i = 0; i < times; i++) {
+                try {
+                    admin.brokers().healthcheck();
+                    Thread.sleep(20);
+                } catch (PulsarAdminException | InterruptedException e) {
+                    Assert.fail();

Review comment:
       This won't usually work well as assertion errors should escape from the test method.
   You can use a CompletareFuture and complete it exceptionally

##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiHealthCheckTest.java
##########
@@ -55,16 +61,91 @@ public void cleanup() throws Exception {
 
     @Test
     public void testHealthCheckup() throws Exception {
-        admin.brokers().healthcheck();
+        final int times = 30;
+        new Thread(() -> {
+            for (int i = 0; i < times; i++) {
+                try {
+                    admin.brokers().healthcheck();
+                    Thread.sleep(20);

Review comment:
       This test will take many minutes to complete, can we make it shorter?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1056510784


   @eolivelli 
   Yes, I agree with you.
   
   I think we can avoid making this approach more complicated by removing the "cleanup" process after using ``Completable#handle`` to ensure we always close the reader/producer.
   
   But I think if the subscription still needs to be rechecked, as reader shutdown can fail due to a number of factors.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] michaeljmarshall commented on pull request #14367: [Issue 14362] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1043990623


   @mattisonchao - I should have added in my review that your tests look good.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1057016514


   @Technoboy-  
   I fix some behaviour, PTAL :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao edited a comment on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao edited a comment on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1056510784


   @eolivelli 
   Yes, I agree with you.
   
   I think we can avoid making this approach more complicated by removing the "cleanup" process after using ``Completable#handle`` to ensure we always close the reader/producer.
   
   But I think the subscription still needs to be rechecked, as reader shutdown can fail due to a number of factors.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1046476849


   /pulsarbot rerun-failure-checks


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on a change in pull request #14367: [WIP][Issue 14362] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#discussion_r810720963



##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
##########
@@ -350,12 +347,6 @@ public void healthCheck(@Suspended AsyncResponse asyncResponse,
                                 clientAppId(), topicName);
                         throw new RestException(Status.NOT_FOUND, "Topic [{}] not found after create.");
                     }
-                    Topic topic = topicOptional.get();
-                    // clean all subscriptions
-                    return FutureUtil.waitForAll(topic.getSubscriptions().values()
-                            .stream().map(Subscription::deleteForcefully).collect(Collectors.toList()))

Review comment:
       Hi, @michaeljmarshall 
   If we use the ``delete`` method instead of ``deleteForce``. When the reader call ``closeAsync``method throws an exception and the connection is not closed. They will always be there and cause issues like #7724 mentioned.
   So, I think we need to recheck the consumer last connected time to force deleted after the "delete" method to ensure this.

##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
##########
@@ -350,12 +347,6 @@ public void healthCheck(@Suspended AsyncResponse asyncResponse,
                                 clientAppId(), topicName);
                         throw new RestException(Status.NOT_FOUND, "Topic [{}] not found after create.");
                     }
-                    Topic topic = topicOptional.get();
-                    // clean all subscriptions
-                    return FutureUtil.waitForAll(topic.getSubscriptions().values()
-                            .stream().map(Subscription::deleteForcefully).collect(Collectors.toList()))

Review comment:
       Hi, @michaeljmarshall  @Technoboy- 
   If we use the ``delete`` method instead of ``deleteForce``. When the reader call ``closeAsync``method throws an exception and the connection is not closed. They will always be there and cause issues like #7724 mentioned.
   So, I think we need to recheck the consumer last connected time to force deleted after the "delete" method to ensure this.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on pull request #14367: [Issue 14362] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1043964281


   I need more eyes to help me review this change to avoid regression.
   
   @codelipenghui @michaeljmarshall @Technoboy-  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on a change in pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#discussion_r815252492



##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
##########
@@ -343,41 +350,81 @@ public void healthCheck(@Suspended AsyncResponse asyncResponse,
         String messageStr = UUID.randomUUID().toString();
         // create non-partitioned topic manually and close the previous reader if present.
         return pulsar().getBrokerService().getTopic(topicName, true)
-                // check and clean all subscriptions
                 .thenCompose(topicOptional -> {
                     if (!topicOptional.isPresent()) {
                         LOG.error("[{}] Fail to run health check while get topic {}. because get null value.",
                                 clientAppId(), topicName);
                         throw new RestException(Status.NOT_FOUND, "Topic [{}] not found after create.");
                     }
-                    Topic topic = topicOptional.get();
-                    // clean all subscriptions
-                    return FutureUtil.waitForAll(topic.getSubscriptions().values()
-                            .stream().map(Subscription::deleteForcefully).collect(Collectors.toList()))
-                            .thenApply(__ -> topic);
-                }).thenCompose(topic -> {
+                    return tryCleanPreviousSubscriptions(topicOptional.get());
+                }).thenCompose(unused-> {
                     try {
                         PulsarClient client = pulsar().getClient();
                         return client.newProducer(Schema.STRING).topic(topicName).createAsync()
-                                        .thenCombine(client.newReader(Schema.STRING).topic(topicName)
+                                .thenCombine(client.newReader(Schema.STRING).topic(topicName)
                                         .startMessageId(MessageId.latest).createAsync(), (producer, reader) ->
-                                                        producer.sendAsync(messageStr).thenCompose(__ ->
-                                                                healthCheckRecursiveReadNext(reader, messageStr))
-                                                        .thenCompose(__ -> {
-                                                            List<CompletableFuture<Void>> closeFutures =
-                                                                    new ArrayList<>();
-                                                            closeFutures.add(producer.closeAsync());
-                                                            closeFutures.add(reader.closeAsync());
-                                                            return FutureUtil.waitForAll(closeFutures);
-                                                        })
-                                        ).thenAccept(ignore -> {});
+                                        producer.sendAsync(messageStr).thenCompose(__ ->
+                                                        healthCheckRecursiveReadNext(reader, messageStr))
+                                                .thenCompose(__ -> {
+                                                    List<CompletableFuture<Void>> closeFutures =
+                                                            new ArrayList<>();
+                                                    closeFutures.add(producer.closeAsync());
+                                                    closeFutures.add(reader.closeAsync());
+                                                    return FutureUtil.waitForAll(closeFutures);
+                                                })
+                                ).thenAccept(ignore -> {});
                     } catch (PulsarServerException e) {
                         LOG.error("[{}] Fail to run health check while get client.", clientAppId());
                         throw new RestException(e);
                     }
                 });
     }
 
+    private CompletableFuture<Void> tryCleanPreviousSubscriptions(Topic topic) {
+        // clean all subscriptions
+        return FutureUtil.waitForAll(topic.getSubscriptions().values()
+                .stream().filter(subscription ->
+                // All system topics are using compaction, even though is not explicitly set in the policies.
+                        !subscription.getName().equals(Compactor.COMPACTION_SUBSCRIPTION))
+                .map(Subscription::delete).collect(Collectors.toList()))

Review comment:
       @Technoboy- 
   Because all the system topic has compaction subscription that name is ``__compaction``, every time we call health check, we will create a new reader with a subscription name's ``reader-xxx``. we just need to remove the reader subscriptions and don't need to remove ``__compaction`` subscription.
   I'm not sure if I explain clearly, please let me know what you think, very thanks~
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on a change in pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#discussion_r817575150



##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
##########
@@ -338,39 +336,60 @@ public void healthCheck(@Suspended AsyncResponse asyncResponse,
         NamespaceName namespaceName = (topicVersion == TopicVersion.V2)
                 ? NamespaceService.getHeartbeatNamespaceV2(pulsar().getAdvertisedAddress(), pulsar().getConfiguration())
                 : NamespaceService.getHeartbeatNamespace(pulsar().getAdvertisedAddress(), pulsar().getConfiguration());
-        String topicName = String.format("persistent://%s/%s", namespaceName, HEALTH_CHECK_TOPIC_SUFFIX);
+        final String topicName = String.format("persistent://%s/%s", namespaceName, HEALTH_CHECK_TOPIC_SUFFIX);
         LOG.info("[{}] Running healthCheck with topic={}", clientAppId(), topicName);
-        String messageStr = UUID.randomUUID().toString();
+        final String messageStr = UUID.randomUUID().toString();
+        final String subscriptionName = "healthCheck-" + messageStr;
         // create non-partitioned topic manually and close the previous reader if present.
         return pulsar().getBrokerService().getTopic(topicName, true)
-                // check and clean all subscriptions
                 .thenCompose(topicOptional -> {
                     if (!topicOptional.isPresent()) {
                         LOG.error("[{}] Fail to run health check while get topic {}. because get null value.",
                                 clientAppId(), topicName);
-                        throw new RestException(Status.NOT_FOUND, "Topic [{}] not found after create.");
+                        throw new RestException(Status.NOT_FOUND,
+                                String.format("Topic [%s] not found after create.", topicName));
                     }
-                    Topic topic = topicOptional.get();
-                    // clean all subscriptions
-                    return FutureUtil.waitForAll(topic.getSubscriptions().values()
-                            .stream().map(Subscription::deleteForcefully).collect(Collectors.toList()))
-                            .thenApply(__ -> topic);
-                }).thenCompose(topic -> {
                     try {
                         PulsarClient client = pulsar().getClient();
                         return client.newProducer(Schema.STRING).topic(topicName).createAsync()
-                                        .thenCombine(client.newReader(Schema.STRING).topic(topicName)
+                                .thenCombine(client.newReader(Schema.STRING).topic(topicName)
+                                        .subscriptionName(subscriptionName)
                                         .startMessageId(MessageId.latest).createAsync(), (producer, reader) ->
-                                                        producer.sendAsync(messageStr).thenCompose(__ ->
-                                                                healthCheckRecursiveReadNext(reader, messageStr))
-                                                        .thenCompose(__ -> {
-                                                            List<CompletableFuture<Void>> closeFutures =
-                                                                    new ArrayList<>();
-                                                            closeFutures.add(producer.closeAsync());
-                                                            closeFutures.add(reader.closeAsync());
-                                                            return FutureUtil.waitForAll(closeFutures);
-                                                        })
-                                        ).thenAccept(ignore -> {});
+                                        producer.sendAsync(messageStr)
+                                                .thenCompose(__ -> healthCheckRecursiveReadNext(reader, messageStr))
+                                                .handle((unused, exception) -> {
+                                                    // no matter exception or success, we still need to
+                                                    // close producer/reader
+                                                    List<CompletableFuture<Void>> closeFutures = new ArrayList<>();
+                                                    closeFutures.add(producer.closeAsync());
+                                                    closeFutures.add(reader.closeAsync());

Review comment:
       Good catch!!
   
   I will fix it ASAP




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Technoboy- commented on a change in pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
Technoboy- commented on a change in pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#discussion_r814606780



##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
##########
@@ -343,41 +350,81 @@ public void healthCheck(@Suspended AsyncResponse asyncResponse,
         String messageStr = UUID.randomUUID().toString();
         // create non-partitioned topic manually and close the previous reader if present.
         return pulsar().getBrokerService().getTopic(topicName, true)
-                // check and clean all subscriptions
                 .thenCompose(topicOptional -> {
                     if (!topicOptional.isPresent()) {
                         LOG.error("[{}] Fail to run health check while get topic {}. because get null value.",
                                 clientAppId(), topicName);
                         throw new RestException(Status.NOT_FOUND, "Topic [{}] not found after create.");

Review comment:
       Topic [{}] ?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on a change in pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#discussion_r815253206



##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
##########
@@ -343,41 +350,81 @@ public void healthCheck(@Suspended AsyncResponse asyncResponse,
         String messageStr = UUID.randomUUID().toString();
         // create non-partitioned topic manually and close the previous reader if present.
         return pulsar().getBrokerService().getTopic(topicName, true)
-                // check and clean all subscriptions
                 .thenCompose(topicOptional -> {
                     if (!topicOptional.isPresent()) {
                         LOG.error("[{}] Fail to run health check while get topic {}. because get null value.",
                                 clientAppId(), topicName);
                         throw new RestException(Status.NOT_FOUND, "Topic [{}] not found after create.");

Review comment:
       Good catch!
   FIxed.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1057754030


   @codelipenghui @Technoboy-  @eolivelli @lhotari  @michaeljmarshall @Jason918 
   
   PTAL, when you have time ~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] codelipenghui merged pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
codelipenghui merged pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] eolivelli commented on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1056496329


   Let's see this problem from another point of view:
   
   Readers are mapped to Non-Durable subscriptions that are tied to the Reader, so if you close gracefully the Reader the subscription should be automatically deleted.
   **I wonder if there is really a need to do this subscriptions clean up at all.**
   I mean, is a Pulsar application that uses are Reader, you usually don't have to care about possible remaining dangling subscriptions on the broker, they are automatically cleaned up.
   
   So if we ensure that we close the Reader and the Producer and we do not perform the "clean up", there won't be concurrency issues and everything will work cleanly as expected.
   
   After all this "heartbeat" is like a dummy application that writes a message and uses a reader.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1056482122


   @codelipenghui  @eolivelli @michaeljmarshall @Technoboy- 
   After deep think, I think we just have two-point can make the subscription still exist after the health check:
   
   1. Throw an error before closing the reader.
   2. Delete subscription fails after closing reader.
   
   So we can use another method to fix that issues to avoid using the non-thread-safe cleaner:
   
   1. Use ``Completable#handle`` to fix problem 1, the reader needs to be closed regardless of whether there is an exception.
   2. Recheck the subscription after closing reading, and force deletion if it still exists after closing reading.
   
   Please let me know if you have any suggestions. Thanks~


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1055985617


   > @mattisonchao would you have time to address review comments ? This patch is very important, as k8s users are affects by this problem
   
   I will fix it ASAP.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Technoboy- commented on a change in pull request #14367: [Issue 14362] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
Technoboy- commented on a change in pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#discussion_r809730770



##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
##########
@@ -350,12 +347,6 @@ public void healthCheck(@Suspended AsyncResponse asyncResponse,
                                 clientAppId(), topicName);
                         throw new RestException(Status.NOT_FOUND, "Topic [{}] not found after create.");
                     }
-                    Topic topic = topicOptional.get();
-                    // clean all subscriptions
-                    return FutureUtil.waitForAll(topic.getSubscriptions().values()
-                            .stream().map(Subscription::deleteForcefully).collect(Collectors.toList()))

Review comment:
       Moving this logic to the end of the `produce-consumer`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on pull request #14367: [WIP][Issue 14362] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1046382652


   @michaeljmarshall  @Technoboy- 
   I changed logic of this PR, PTAL :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on a change in pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#discussion_r817403838



##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
##########
@@ -343,41 +349,71 @@ public void healthCheck(@Suspended AsyncResponse asyncResponse,
         String messageStr = UUID.randomUUID().toString();
         // create non-partitioned topic manually and close the previous reader if present.
         return pulsar().getBrokerService().getTopic(topicName, true)
-                // check and clean all subscriptions
                 .thenCompose(topicOptional -> {
                     if (!topicOptional.isPresent()) {
                         LOG.error("[{}] Fail to run health check while get topic {}. because get null value.",
                                 clientAppId(), topicName);
-                        throw new RestException(Status.NOT_FOUND, "Topic [{}] not found after create.");
+                        throw new RestException(Status.NOT_FOUND,
+                                String.format("Topic [%s] not found after create.", topicName));
                     }
-                    Topic topic = topicOptional.get();
-                    // clean all subscriptions
-                    return FutureUtil.waitForAll(topic.getSubscriptions().values()
-                            .stream().map(Subscription::deleteForcefully).collect(Collectors.toList()))
-                            .thenApply(__ -> topic);
-                }).thenCompose(topic -> {
                     try {
                         PulsarClient client = pulsar().getClient();
                         return client.newProducer(Schema.STRING).topic(topicName).createAsync()
-                                        .thenCombine(client.newReader(Schema.STRING).topic(topicName)
+                                .thenCombine(client.newReader(Schema.STRING).topic(topicName)
                                         .startMessageId(MessageId.latest).createAsync(), (producer, reader) ->
-                                                        producer.sendAsync(messageStr).thenCompose(__ ->
-                                                                healthCheckRecursiveReadNext(reader, messageStr))
-                                                        .thenCompose(__ -> {
-                                                            List<CompletableFuture<Void>> closeFutures =
-                                                                    new ArrayList<>();
-                                                            closeFutures.add(producer.closeAsync());
-                                                            closeFutures.add(reader.closeAsync());
-                                                            return FutureUtil.waitForAll(closeFutures);
-                                                        })
-                                        ).thenAccept(ignore -> {});
+                                        producer.sendAsync(messageStr).thenCompose(__ ->
+                                                        healthCheckRecursiveReadNext(reader, messageStr))
+                                                .thenCompose(__ -> {

Review comment:
       @eolivelli 
   Yes, I totally agree with you. I will change here to use  ``whenComplete``.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] eolivelli commented on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1060889216


   @mattisonchao 
   can you please port this patch to branch-2.8 and branch-2.9 ?
   the patch does not apply cleanly


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] eolivelli commented on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1058951499


   @codelipenghui if we cut a new rc for 2.10  is would be great to have this patch.
   It is a big problem for k8s users


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] michaeljmarshall commented on a change in pull request #14367: [Issue 14362] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on a change in pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#discussion_r809708742



##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
##########
@@ -350,12 +347,6 @@ public void healthCheck(@Suspended AsyncResponse asyncResponse,
                                 clientAppId(), topicName);
                         throw new RestException(Status.NOT_FOUND, "Topic [{}] not found after create.");
                     }
-                    Topic topic = topicOptional.get();
-                    // clean all subscriptions
-                    return FutureUtil.waitForAll(topic.getSubscriptions().values()
-                            .stream().map(Subscription::deleteForcefully).collect(Collectors.toList()))

Review comment:
       I am wondering if it is risky to remove this whole block. Instead, what if we simply called `Subscription::delete`? That would leave any subscription with active consumers in places. I think that should prevent the race I saw in the issue.
   
   Additionally, we'd need to ignore the exceptionally completed future that the `delete` method returns.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] eolivelli commented on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
eolivelli commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1055596847


   @mattisonchao would you have time to address review comments ?
   This patch is very important, as k8s users are affects by this problem


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao commented on a change in pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao commented on a change in pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#discussion_r817334580



##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiHealthCheckTest.java
##########
@@ -55,16 +61,91 @@ public void cleanup() throws Exception {
 
     @Test
     public void testHealthCheckup() throws Exception {
-        admin.brokers().healthcheck();
+        final int times = 30;
+        new Thread(() -> {
+            for (int i = 0; i < times; i++) {
+                try {
+                    admin.brokers().healthcheck();
+                    Thread.sleep(20);
+                } catch (PulsarAdminException | InterruptedException e) {
+                    Assert.fail();

Review comment:
       Fixed

##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiHealthCheckTest.java
##########
@@ -55,16 +61,91 @@ public void cleanup() throws Exception {
 
     @Test
     public void testHealthCheckup() throws Exception {
-        admin.brokers().healthcheck();
+        final int times = 30;
+        new Thread(() -> {
+            for (int i = 0; i < times; i++) {
+                try {
+                    admin.brokers().healthcheck();
+                    Thread.sleep(20);

Review comment:
       Fixed

##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/AdminApiHealthCheckTest.java
##########
@@ -55,16 +61,91 @@ public void cleanup() throws Exception {
 
     @Test
     public void testHealthCheckup() throws Exception {
-        admin.brokers().healthcheck();
+        final int times = 30;
+        new Thread(() -> {

Review comment:
       Fixed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] eolivelli commented on a change in pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
eolivelli commented on a change in pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#discussion_r817388570



##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
##########
@@ -343,41 +349,71 @@ public void healthCheck(@Suspended AsyncResponse asyncResponse,
         String messageStr = UUID.randomUUID().toString();
         // create non-partitioned topic manually and close the previous reader if present.
         return pulsar().getBrokerService().getTopic(topicName, true)
-                // check and clean all subscriptions
                 .thenCompose(topicOptional -> {
                     if (!topicOptional.isPresent()) {
                         LOG.error("[{}] Fail to run health check while get topic {}. because get null value.",
                                 clientAppId(), topicName);
-                        throw new RestException(Status.NOT_FOUND, "Topic [{}] not found after create.");
+                        throw new RestException(Status.NOT_FOUND,
+                                String.format("Topic [%s] not found after create.", topicName));
                     }
-                    Topic topic = topicOptional.get();
-                    // clean all subscriptions
-                    return FutureUtil.waitForAll(topic.getSubscriptions().values()
-                            .stream().map(Subscription::deleteForcefully).collect(Collectors.toList()))
-                            .thenApply(__ -> topic);
-                }).thenCompose(topic -> {
                     try {
                         PulsarClient client = pulsar().getClient();
                         return client.newProducer(Schema.STRING).topic(topicName).createAsync()
-                                        .thenCombine(client.newReader(Schema.STRING).topic(topicName)
+                                .thenCombine(client.newReader(Schema.STRING).topic(topicName)
                                         .startMessageId(MessageId.latest).createAsync(), (producer, reader) ->
-                                                        producer.sendAsync(messageStr).thenCompose(__ ->
-                                                                healthCheckRecursiveReadNext(reader, messageStr))
-                                                        .thenCompose(__ -> {
-                                                            List<CompletableFuture<Void>> closeFutures =
-                                                                    new ArrayList<>();
-                                                            closeFutures.add(producer.closeAsync());
-                                                            closeFutures.add(reader.closeAsync());
-                                                            return FutureUtil.waitForAll(closeFutures);
-                                                        })
-                                        ).thenAccept(ignore -> {});
+                                        producer.sendAsync(messageStr).thenCompose(__ ->
+                                                        healthCheckRecursiveReadNext(reader, messageStr))
+                                                .thenCompose(__ -> {

Review comment:
       I have now realised a problem on this code and it was there before.
   In case of exception we are not closing the producer and the reader!
   This is because thenCompose works only on the happy path.
   We have to rework this chain in a way that the producer and the reader are always closed (with a whenComplete for instance)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Technoboy- commented on a change in pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
Technoboy- commented on a change in pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#discussion_r815307660



##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/BrokersBase.java
##########
@@ -343,41 +350,81 @@ public void healthCheck(@Suspended AsyncResponse asyncResponse,
         String messageStr = UUID.randomUUID().toString();
         // create non-partitioned topic manually and close the previous reader if present.
         return pulsar().getBrokerService().getTopic(topicName, true)
-                // check and clean all subscriptions
                 .thenCompose(topicOptional -> {
                     if (!topicOptional.isPresent()) {
                         LOG.error("[{}] Fail to run health check while get topic {}. because get null value.",
                                 clientAppId(), topicName);
                         throw new RestException(Status.NOT_FOUND, "Topic [{}] not found after create.");
                     }
-                    Topic topic = topicOptional.get();
-                    // clean all subscriptions
-                    return FutureUtil.waitForAll(topic.getSubscriptions().values()
-                            .stream().map(Subscription::deleteForcefully).collect(Collectors.toList()))
-                            .thenApply(__ -> topic);
-                }).thenCompose(topic -> {
+                    return tryCleanPreviousSubscriptions(topicOptional.get());
+                }).thenCompose(unused-> {
                     try {
                         PulsarClient client = pulsar().getClient();
                         return client.newProducer(Schema.STRING).topic(topicName).createAsync()
-                                        .thenCombine(client.newReader(Schema.STRING).topic(topicName)
+                                .thenCombine(client.newReader(Schema.STRING).topic(topicName)
                                         .startMessageId(MessageId.latest).createAsync(), (producer, reader) ->
-                                                        producer.sendAsync(messageStr).thenCompose(__ ->
-                                                                healthCheckRecursiveReadNext(reader, messageStr))
-                                                        .thenCompose(__ -> {
-                                                            List<CompletableFuture<Void>> closeFutures =
-                                                                    new ArrayList<>();
-                                                            closeFutures.add(producer.closeAsync());
-                                                            closeFutures.add(reader.closeAsync());
-                                                            return FutureUtil.waitForAll(closeFutures);
-                                                        })
-                                        ).thenAccept(ignore -> {});
+                                        producer.sendAsync(messageStr).thenCompose(__ ->
+                                                        healthCheckRecursiveReadNext(reader, messageStr))
+                                                .thenCompose(__ -> {
+                                                    List<CompletableFuture<Void>> closeFutures =
+                                                            new ArrayList<>();
+                                                    closeFutures.add(producer.closeAsync());
+                                                    closeFutures.add(reader.closeAsync());
+                                                    return FutureUtil.waitForAll(closeFutures);
+                                                })
+                                ).thenAccept(ignore -> {});
                     } catch (PulsarServerException e) {
                         LOG.error("[{}] Fail to run health check while get client.", clientAppId());
                         throw new RestException(e);
                     }
                 });
     }
 
+    private CompletableFuture<Void> tryCleanPreviousSubscriptions(Topic topic) {
+        // clean all subscriptions
+        return FutureUtil.waitForAll(topic.getSubscriptions().values()
+                .stream().filter(subscription ->
+                // All system topics are using compaction, even though is not explicitly set in the policies.
+                        !subscription.getName().equals(Compactor.COMPACTION_SUBSCRIPTION))
+                .map(Subscription::delete).collect(Collectors.toList()))

Review comment:
       Ok, we can add this. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] Technoboy- commented on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
Technoboy- commented on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1057951278


   > @codelipenghui @Technoboy- @eolivelli @lhotari @michaeljmarshall @Jason918
   > 
   > PTAL, when you have time ~
   
   LGTM.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [pulsar] mattisonchao edited a comment on pull request #14367: [Broker] Fix Broker HealthCheck Endpoint Exposes Race Conditions

Posted by GitBox <gi...@apache.org>.
mattisonchao edited a comment on pull request #14367:
URL: https://github.com/apache/pulsar/pull/14367#issuecomment-1057016514


   @eolivelli 
   I fix some behaviour, PTAL :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org