You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "eolivelli (via GitHub)" <gi...@apache.org> on 2024/05/01 11:25:02 UTC
[PR] Ensure namespace deletion doesn't fail [pulsar]
eolivelli opened a new pull request, #22627:
URL: https://github.com/apache/pulsar/pull/22627
### Motivation
Deleting a namespace sometimes fails and leaves znodes around, especially the "bundle owner" nodes `/namespace/NAMESPACE/xxxxxx `
### Modifications
- ensure that all the children of /namespace/NAMESPACE are deleted, especially the ephemeral nodes for ownership
- ensure that on the code path that handles the deletion of the namespace we make progress, the main fact is about not relying on "cache.exists" before using a "delete": the new code is more pessimistic that goes with "deleteIfExists", handling "NotFound" gracefully
- add more logs about the delete namespace execution path
#### With this patch
- delete namespace seems to consistently work
- there are no more spammy logs about errors around the topic policies event reader (changelog_events...)
- in case we see an error we log at which point the procedure was (because we log all the paths that are deleted from zk)
### Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: <!-- ENTER URL HERE -->
<!--
After opening this PR, the build in apache/pulsar will fail and instructions will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed in
a forked repository.
The tests will be run in the forked repository until all PR review comments have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][broker]Ensure namespace deletion doesn't fail [pulsar]
Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on PR #22627:
URL: https://github.com/apache/pulsar/pull/22627#issuecomment-2089789998
@lhotari I think that it is worth to port this fix to all the active branches, maybe up to 3.0. Wdyt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][broker]Ensure namespace deletion doesn't fail [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #22627:
URL: https://github.com/apache/pulsar/pull/22627#issuecomment-2089800111
> @lhotari I think that it is worth to port this fix to all the active branches, maybe up to 3.0. Wdyt?
@eolivelli makes sense. Btw. there's a rabbit hole in namespace deletion. Currently concurrency limits are missing and a namespace deletion will attempt to concurrency delete all possible topics in the namespace. This is something that would have to be addressed to improve the reliability of namespace deletion. More details in #22541 comments such as https://github.com/apache/pulsar/pull/22541#issuecomment-2071621213 and https://github.com/apache/pulsar/pull/22541#issuecomment-2071687995 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][broker]Ensure namespace deletion doesn't fail [pulsar]
Posted by "dlg99 (via GitHub)" <gi...@apache.org>.
dlg99 commented on code in PR #22627:
URL: https://github.com/apache/pulsar/pull/22627#discussion_r1586479577
##########
pulsar-broker-common/src/main/java/org/apache/pulsar/broker/resources/BaseResources.java:
##########
@@ -197,22 +197,21 @@ protected CompletableFuture<Void> deleteAsync(String path) {
}
protected CompletableFuture<Void> deleteIfExistsAsync(String path) {
- return cache.exists(path).thenCompose(exists -> {
- if (!exists) {
- return CompletableFuture.completedFuture(null);
+ log.info("Deleting path: {}", path);
+ CompletableFuture<Void> future = new CompletableFuture<>();
+ cache.delete(path).whenComplete((ignore, ex) -> {
+ if (ex != null && ex.getCause() instanceof MetadataStoreException.NotFoundException) {
+ log.info("Path {} did not exist in metadata store", path);
+ future.complete(null);
+ } else if (ex != null) {
+ log.info("Cannot path from metadata store: {}", path, ex);
Review Comment:
```suggestion
log.info("Failed to delete path from metadata store: {}", path, ex);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][broker]Ensure namespace deletion doesn't fail [pulsar]
Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli merged PR #22627:
URL: https://github.com/apache/pulsar/pull/22627
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Ensure namespace deletion doesn't fail [pulsar]
Posted by "eolivelli (via GitHub)" <gi...@apache.org>.
eolivelli commented on PR #22627:
URL: https://github.com/apache/pulsar/pull/22627#issuecomment-2088317159
This script reproduces MANY problems with namespace deletions, just run it in a loop on a Pulsar cluster with a few brokers (like 6) and more than one ZK server (I am using 3 nodes)
reproduce.sh
```
#! /bin/bash
export PATH=$PATH:/pulsar/bin
set -x
pulsar-admin topics delete-partitioned-topic non-persistent://T1/N1/P1 -f
bin/pulsar zookeeper-shell -server pulsar-zookeeper-ca:2181 ls -R /T1/N1 | grep namespace
pulsar-admin namespaces delete T1/N1
pulsar-admin tenants delete T1
pulsar-admin tenants create T1
pulsar-admin namespaces create T1/N1 -b 64
pulsar-admin topics create-partitioned-topic non-persistent://T1/N1/P1 -p 20
exit 0
```
Execute it in a loop
`while true ; do ./reproduce.sh ; done`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve][broker]Ensure namespace deletion doesn't fail [pulsar]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #22627:
URL: https://github.com/apache/pulsar/pull/22627#issuecomment-2106767000
## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/22627?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `43.13725%` with `29 lines` in your changes are missing coverage. Please review.
> Project coverage is 32.47%. Comparing base [(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`23c39c1`)](https://app.codecov.io/gh/apache/pulsar/pull/22627?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 261 commits behind head on master.
<details><summary>Additional details and impacted files</summary>
[](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
```diff
@@ Coverage Diff @@
## master #22627 +/- ##
=============================================
- Coverage 73.57% 32.47% -41.10%
+ Complexity 32624 87 -32537
=============================================
Files 1877 1745 -132
Lines 139502 133886 -5616
Branches 15299 14673 -626
=============================================
- Hits 102638 43482 -59156
- Misses 28908 84033 +55125
+ Partials 7956 6371 -1585
```
| [Flag](https://app.codecov.io/gh/apache/pulsar/pull/22627/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [inttests](https://app.codecov.io/gh/apache/pulsar/pull/22627/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.52% <43.13%> (+2.94%)` | :arrow_up: |
| [systests](https://app.codecov.io/gh/apache/pulsar/pull/22627/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `24.80% <5.88%> (+0.47%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pulsar/pull/22627/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files](https://app.codecov.io/gh/apache/pulsar/pull/22627?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [.../service/SystemTopicBasedTopicPoliciesService.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FSystemTopicBasedTopicPoliciesService.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL1N5c3RlbVRvcGljQmFzZWRUb3BpY1BvbGljaWVzU2VydmljZS5qYXZh) | `52.26% <100.00%> (-21.93%)` | :arrow_down: |
| [...he/pulsar/metadata/impl/AbstractMetadataStore.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fmetadata%2Fimpl%2FAbstractMetadataStore.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLW1ldGFkYXRhL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvbWV0YWRhdGEvaW1wbC9BYnN0cmFjdE1ldGFkYXRhU3RvcmUuamF2YQ==) | `55.31% <100.00%> (-29.25%)` | :arrow_down: |
| [...ulsar/broker/resources/LocalPoliciesResources.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-broker-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fresources%2FLocalPoliciesResources.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9icm9rZXIvcmVzb3VyY2VzL0xvY2FsUG9saWNpZXNSZXNvdXJjZXMuamF2YQ==) | `44.82% <0.00%> (-31.04%)` | :arrow_down: |
| [...he/pulsar/broker/resources/NamespaceResources.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-broker-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fresources%2FNamespaceResources.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9icm9rZXIvcmVzb3VyY2VzL05hbWVzcGFjZVJlc291cmNlcy5qYXZh) | `53.90% <25.00%> (-26.25%)` | :arrow_down: |
| [...apache/pulsar/broker/resources/TopicResources.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-broker-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fresources%2FTopicResources.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9icm9rZXIvcmVzb3VyY2VzL1RvcGljUmVzb3VyY2VzLmphdmE=) | `39.58% <33.33%> (-57.03%)` | :arrow_down: |
| [.../org/apache/pulsar/metadata/api/MetadataStore.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-metadata%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fmetadata%2Fapi%2FMetadataStore.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLW1ldGFkYXRhL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9wdWxzYXIvbWV0YWRhdGEvYXBpL01ldGFkYXRhU3RvcmUuamF2YQ==) | `60.00% <50.00%> (-20.00%)` | :arrow_down: |
| [.../apache/pulsar/broker/resources/BaseResources.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-broker-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fresources%2FBaseResources.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9icm9rZXIvcmVzb3VyY2VzL0Jhc2VSZXNvdXJjZXMuamF2YQ==) | `55.85% <50.00%> (-16.88%)` | :arrow_down: |
| [...pache/pulsar/broker/admin/impl/NamespacesBase.java](https://app.codecov.io/gh/apache/pulsar/pull/22627?src=pr&el=tree&filepath=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fadmin%2Fimpl%2FNamespacesBase.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi9pbXBsL05hbWVzcGFjZXNCYXNlLmphdmE=) | `15.22% <0.00%> (-57.86%)` | :arrow_down: |
... and [1517 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/22627/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org