You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "BewareMyPower (via GitHub)" <gi...@apache.org> on 2024/02/02 13:43:22 UTC
[PR] [fix][broker] Ignore non-retryable errors when loading topic-level policies before loading topics [pulsar]
BewareMyPower opened a new pull request, #22014:
URL: https://github.com/apache/pulsar/pull/22014
### Motivation
https://github.com/apache/pulsar/pull/21445 introduces an implicit requirement that if the topic policies cannot be loaded, `BrokerService#getTopic` will fail and then the client will retry loading the topic.
It could break some existing usages like the tests in the C++ client: https://github.com/apache/pulsar-client-cpp/pull/394
This change is applied only to avoid the race condition when unloading a namespace bundle. However, the client should not fail if the topic-level policies are not available.
### Modifications
If the topic policies cannot be loaded due to a non-retryable error, we should not fail the `getTopic`. For retryable errors, the client will still retry until the broker gets the topic-level policies successfully after some attempts.
Modify `TokenAuthenticatedProducerConsumerTest` to protect the change.
**Note: This PR also changes the `isRetriableError` semantics. To keep the compatibility, should we avoid this change?**
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Ignore non-retryable errors when loading topic-level policies before loading topics [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower closed pull request #22014: [fix][broker] Ignore non-retryable errors when loading topic-level policies before loading topics
URL: https://github.com/apache/pulsar/pull/22014
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Ignore non-retryable errors when loading topic-level policies before loading topics [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on PR #22014:
URL: https://github.com/apache/pulsar/pull/22014#issuecomment-1924897805
> This change is applied only to avoid the race condition when unloading a namespace bundle. However, the client should not fail if the topic-level policies are not available.
The risk is if users have data retention policy on the topic but no data retention on the namespace, the produced data will be removed by the broker if no subscriptions.
For example, you might have 10GB produced data with data retention policy before. If the topic been unloaded and reload by another broker without the topic policy, all the 10GB data will be removed permanently. It could be high risk to users.
And other policies will also have the similar problem, persistence policy (replicas), backlog policy, geo-replication …
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Ignore non-retryable errors when loading topic-level policies before loading topics [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #22014:
URL: https://github.com/apache/pulsar/pull/22014#issuecomment-1925614471
After estimating the risks again, I agree that we should not skip the system topic load exception even if it's non-retryable. Otherwise, the configured system topic policy could be lost even if `topicLevelPoliciesEnabled` is true. We should prevent the topic from being created anyway.
BTW, there is another issue that the clients would retry even if there is a non-retryable error when loading the system topic. But it can be improved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [fix][broker] Ignore non-retryable errors when loading topic-level policies before loading topics [pulsar]
Posted by "BewareMyPower (via GitHub)" <gi...@apache.org>.
BewareMyPower commented on PR #22014:
URL: https://github.com/apache/pulsar/pull/22014#issuecomment-1925588141
> For example, you might have 10GB produced data with data retention policy before. If the topic been unloaded and reload by another broker without the topic policy,
I don't think this issue is avoided. See https://github.com/apache/pulsar/blob/e7c2a75473b545134a3b292ae0e87a79d65cb756/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L1064
If the broker without topic policy does not configure topic policy or system topic, `getTopicPoliciesBypassSystemTopic` will return `Optional.empty()`, which is just the same behavior with this PR when there are non-retryable errors
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org