You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "eolivelli (via GitHub)" <gi...@apache.org> on 2024/05/01 11:25:02 UTC

[PR] Ensure namespace deletion doesn't fail [pulsar]

eolivelli opened a new pull request, #22627:
URL: https://github.com/apache/pulsar/pull/22627

   ### Motivation
   
   Deleting a namespace sometimes fails and leaves znodes around, especially the "bundle owner" nodes `/namespace/NAMESPACE/xxxxxx `
   
   ### Modifications
   
   
   - ensure that all the children of /namespace/NAMESPACE are deleted, especially the ephemeral nodes for ownership
   - ensure that on the code path that handles the deletion of the namespace we make progress, the main fact is about not relying on "cache.exists" before using a "delete": the new code is more pessimistic that goes with "deleteIfExists", handling "NotFound" gracefully
   - add more logs about the delete namespace execution path
   
   #### With this patch
   
   - delete namespace seems to consistently work
   - there are no more spammy logs about errors around the topic policies event reader (changelog_events...)
   - in case we see an error we log at which point the procedure was (because we log all the paths that are deleted from zk)
   
   ### Verifying this change
   
   This change is a trivial rework / code cleanup without any test coverage.
   
   ### Does this pull request potentially affect one of the following parts:
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   *If the box was checked, please highlight the changes*
   
   - [ ] Dependencies (add or upgrade a dependency)
   - [ ] The public API
   - [ ] The schema
   - [ ] The default values of configurations
   - [ ] The threading model
   - [ ] The binary protocol
   - [ ] The REST endpoints
   - [ ] The admin CLI options
   - [ ] The metrics
   - [ ] Anything that affects deployment
   
   ### Documentation
   
   <!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
   
   - [ ] `doc` <!-- Your PR contains doc changes. -->
   - [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
   - [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
   - [ ] `doc-complete` <!-- Docs have been already added -->
   
   ### Matching PR in forked repository
   
   PR in forked repository: <!-- ENTER URL HERE -->
   
   <!--
   After opening this PR, the build in apache/pulsar will fail and instructions will
   be provided for opening a PR in the PR author's forked repository.
   
   apache/pulsar pull requests should be first tested in your own fork since the 
   apache/pulsar CI based on GitHub Actions has constrained resources and quota.
   GitHub Actions provides separate quota for pull requests that are executed in 
   a forked repository.
   
   The tests will be run in the forked repository until all PR review comments have
   been handled, the tests pass and the PR is approved by a reviewer.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][broker]Ensure namespace deletion doesn't fail [pulsar]

Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on PR #22627:
URL: https://github.com/apache/pulsar/pull/22627#issuecomment-2089789998

   @lhotari I think that it is worth to port this fix to all the active branches, maybe up to 3.0. Wdyt?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][broker]Ensure namespace deletion doesn't fail [pulsar]

Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #22627:
URL: https://github.com/apache/pulsar/pull/22627#issuecomment-2089800111

   > @lhotari I think that it is worth to port this fix to all the active branches, maybe up to 3.0. Wdyt?
   
   @eolivelli makes sense. Btw. there's a rabbit hole in namespace deletion. Currently concurrency limits are missing and a namespace deletion will attempt to concurrency delete all possible topics in the namespace. This is something that would have to be addressed to improve the reliability of namespace deletion. More details in #22541 comments such as https://github.com/apache/pulsar/pull/22541#issuecomment-2071621213 and https://github.com/apache/pulsar/pull/22541#issuecomment-2071687995 . 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][broker]Ensure namespace deletion doesn't fail [pulsar]

Posted by "dlg99 (via GitHub)" <gi...@apache.org>.
dlg99 commented on code in PR #22627:
URL: https://github.com/apache/pulsar/pull/22627#discussion_r1586479577


##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/BaseResources.java:
##########
@@ -197,22 +197,21 @@ protected CompletableFuture<Void> deleteAsync(String path) {
     }
 
     protected CompletableFuture<Void> deleteIfExistsAsync(String path) {
-        return cache.exists(path).thenCompose(exists -> {
-            if (!exists) {
-                return CompletableFuture.completedFuture(null);
+        log.info("Deleting path: {}", path);
+        CompletableFuture<Void> future = new CompletableFuture<>();
+        cache.delete(path).whenComplete((ignore, ex) -> {
+            if (ex != null && ex.getCause() instanceof MetadataStoreException.NotFoundException) {
+                log.info("Path {} did not exist in metadata store", path);
+                future.complete(null);
+            } else if (ex != null) {
+                log.info("Cannot path from metadata store: {}", path, ex);

Review Comment:
   ```suggestion
                   log.info("Failed to delete path from metadata store: {}", path, ex);
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][broker]Ensure namespace deletion doesn't fail [pulsar]

Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli merged PR #22627:
URL: https://github.com/apache/pulsar/pull/22627


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Ensure namespace deletion doesn't fail [pulsar]

Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on PR #22627:
URL: https://github.com/apache/pulsar/pull/22627#issuecomment-2088317159

   This script reproduces MANY problems with namespace deletions, just run it in a loop on a Pulsar cluster with a few brokers (like 6) and more than one ZK server (I am using 3 nodes)
   
   reproduce.sh
   
   ```
   #! /bin/bash
   export PATH=$PATH:/pulsar/bin
   set -x
   pulsar-admin topics delete-partitioned-topic non-persistent://T1/N1/P1 -f
   bin/pulsar zookeeper-shell -server pulsar-zookeeper-ca:2181 ls -R /T1/N1 | grep namespace
   pulsar-admin namespaces delete T1/N1
   pulsar-admin tenants delete T1
   
   pulsar-admin tenants create T1
   pulsar-admin namespaces create T1/N1 -b 64
   pulsar-admin topics create-partitioned-topic non-persistent://T1/N1/P1 -p 20
   
   exit 0
   ```
   
   
   Execute it in a loop
   
   `while true ; do ./reproduce.sh ; done`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [improve][broker]Ensure namespace deletion doesn't fail [pulsar]

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #22627:
URL: https://github.com/apache/pulsar/pull/22627#issuecomment-2106767000

   ## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22627?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   Attention: Patch coverage is `43.13725%` with `29 lines` in your changes are missing coverage. Please review.
   > Project coverage is 32.47%. Comparing base [(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`23c39c1`)](https://app.codecov.io/gh/apache/pulsar/pull/22627?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
   > Report is 261 commits behind head on master.
   
   <details><summary>Additional details and impacted files</summary>
   
   
   [![Impacted file tree graph](https://app.codecov.io/gh/apache/pulsar/pull/22627/graphs/tree.svg?width=650&height=150&src=pr&token=acYqCpsK9J&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master   #22627       +/-   ##
   =============================================
   - Coverage     73.57%   32.47%   -41.10%     
   + Complexity    32624       87    -32537     
   =============================================
     Files          1877     1745      -132     
     Lines        139502   133886     -5616     
     Branches      15299    14673      -626     
   =============================================
   - Hits         102638    43482    -59156     
   - Misses        28908    84033    +55125     
   + Partials       7956     6371     -1585     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22627/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22627/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.52% <43.13%> (+2.94%)` | :arrow_up: |
   | [systests](https://app.codecov.io/gh/apache/pulsar/pull/22627/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `24.80% <5.88%> (+0.47%)` | :arrow_up: |
   | [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22627/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files](https://app.codecov.io/gh/apache/pulsar/pull/22627?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [.../service/SystemTopicBasedTopicPoliciesService.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FSystemTopicBasedTopicPoliciesService.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL1N5c3RlbVRvcGljQmFzZWRUb3BpY1BvbGljaWVzU2VydmljZS5qYXZh) | `52.26% <100.00%> (-21.93%)` | :arrow_down: |
   | [...he/pulsar/metadata/impl/AbstractMetadataStore.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fmetadata%2Fimpl%2FAbstractMetadataStore.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLW1ldGFkYXRhL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvbWV0YWRhdGEvaW1wbC9BYnN0cmFjdE1ldGFkYXRhU3RvcmUuamF2YQ==) | `55.31% <100.00%> (-29.25%)` | :arrow_down: |
   | [...ulsar/broker/resources/LocalPoliciesResources.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-broker-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fresources%2FLocalPoliciesResources.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9icm9rZXIvcmVzb3VyY2VzL0xvY2FsUG9saWNpZXNSZXNvdXJjZXMuamF2YQ==) | `44.82% <0.00%> (-31.04%)` | :arrow_down: |
   | [...he/pulsar/broker/resources/NamespaceResources.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-broker-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fresources%2FNamespaceResources.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9icm9rZXIvcmVzb3VyY2VzL05hbWVzcGFjZVJlc291cmNlcy5qYXZh) | `53.90% <25.00%> (-26.25%)` | :arrow_down: |
   | [...apache/pulsar/broker/resources/TopicResources.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-broker-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fresources%2FTopicResources.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9icm9rZXIvcmVzb3VyY2VzL1RvcGljUmVzb3VyY2VzLmphdmE=) | `39.58% <33.33%> (-57.03%)` | :arrow_down: |
   | [.../org/apache/pulsar/metadata/api/MetadataStore.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fmetadata%2Fapi%2FMetadataStore.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLW1ldGFkYXRhL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvbWV0YWRhdGEvYXBpL01ldGFkYXRhU3RvcmUuamF2YQ==) | `60.00% <50.00%> (-20.00%)` | :arrow_down: |
   | [.../apache/pulsar/broker/resources/BaseResources.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-broker-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fresources%2FBaseResources.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9icm9rZXIvcmVzb3VyY2VzL0Jhc2VSZXNvdXJjZXMuamF2YQ==) | `55.85% <50.00%> (-16.88%)` | :arrow_down: |
   | [...pache/pulsar/broker/admin/impl/NamespacesBase.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fadmin%2Fimpl%2FNamespacesBase.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi9pbXBsL05hbWVzcGFjZXNCYXNlLmphdmE=) | `15.22% <0.00%> (-57.86%)` | :arrow_down: |
   
   ... and [1517 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/22627/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org