You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "zhanghaou (via GitHub)" <gi...@apache.org> on 2023/06/27 13:14:52 UTC
[GitHub] [pulsar] zhanghaou opened a new pull request, #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
zhanghaou opened a new pull request, #20613:
URL: https://github.com/apache/pulsar/pull/20613
### Motivation
If listener.onUpdate has any exception, the future can not be complete. We should catch the exception and make the method more safety, otherwise thread would be block.
### Does this pull request potentially affect one of the following parts:
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
*If the box was checked, please highlight the changes*
- [ ] Dependencies (add or upgrade a dependency)
- [ ] The public API
- [ ] The schema
- [ ] The default values of configurations
- [ ] The threading model
- [ ] The binary protocol
- [ ] The REST endpoints
- [ ] The admin CLI options
- [ ] The metrics
- [ ] Anything that affects deployment
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: <!-- ENTER URL HERE -->
<!--
After opening this PR, the build in apache/pulsar will fail and instructions will
be provided for opening a PR in the PR author's forked repository.
apache/pulsar pull requests should be first tested in your own fork since the
apache/pulsar CI based on GitHub Actions has constrained resources and quota.
GitHub Actions provides separate quota for pull requests that are executed in
a forked repository.
The tests will be run in the forked repository until all PR review comments have
been handled, the tests pass and the PR is approved by a reviewer.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zhanghaou commented on pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "zhanghaou (via GitHub)" <gi...@apache.org>.
zhanghaou commented on PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#issuecomment-1600328433
/pulsarbot rerun-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#discussion_r1241005146
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java:
##########
@@ -362,13 +362,20 @@ private void initPolicesCache(SystemTopicClient.Reader<PulsarEvent> reader, Comp
policyCacheInitMap.computeIfPresent(
reader.getSystemTopic().getTopicName().getNamespaceObject(), (k, v) -> true);
// replay policy message
- policiesCache.forEach(((topicName, topicPolicies) -> {
- if (listeners.get(topicName) != null) {
- for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
- listener.onUpdate(topicPolicies);
+ try {
+ policiesCache.forEach(((topicName, topicPolicies) -> {
+ if (listeners.get(topicName) != null) {
+ for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
+ listener.onUpdate(topicPolicies);
Review Comment:
Can we easily catch the `onUpdate` method's exception with the warn log and then normally complete the future?
IMO, Because it is an implementation problem(bug) in the listener logic, we can't let one bad apple spoils the whole bunch. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- closed pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- closed pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
URL: https://github.com/apache/pulsar/pull/20613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zhanghaou commented on pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "zhanghaou (via GitHub)" <gi...@apache.org>.
zhanghaou commented on PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#issuecomment-1607481219
@Technoboy- Makes sense. Thanks for your explanation. I also caught exceptions in the notifyListener() method to prevent a listener callback from failing to terminate the remaining listener callbacks. Please take a look again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zhanghaou commented on a diff in pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "zhanghaou (via GitHub)" <gi...@apache.org>.
zhanghaou commented on code in PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#discussion_r1236212373
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java:
##########
@@ -362,13 +362,20 @@ private void initPolicesCache(SystemTopicClient.Reader<PulsarEvent> reader, Comp
policyCacheInitMap.computeIfPresent(
reader.getSystemTopic().getTopicName().getNamespaceObject(), (k, v) -> true);
// replay policy message
- policiesCache.forEach(((topicName, topicPolicies) -> {
- if (listeners.get(topicName) != null) {
- for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
- listener.onUpdate(topicPolicies);
+ try {
+ policiesCache.forEach(((topicName, topicPolicies) -> {
+ if (listeners.get(topicName) != null) {
+ for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
+ listener.onUpdate(topicPolicies);
+ }
}
- }
- }));
+ }));
+ } catch (Exception e) {
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#issuecomment-1607054187
I think we need to process like this :
```
policiesCache.forEach(((topicName, topicPolicies) -> {
if (listeners.get(topicName) != null) {
for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
try {
listener.onUpdate(topicPolicies);
} catch (Throwable error) {
log.error("[{}] call listener error.", topicName, error);
}
}
}
}));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zhanghaou commented on a diff in pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "zhanghaou (via GitHub)" <gi...@apache.org>.
zhanghaou commented on code in PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#discussion_r1241049899
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java:
##########
@@ -362,13 +362,20 @@ private void initPolicesCache(SystemTopicClient.Reader<PulsarEvent> reader, Comp
policyCacheInitMap.computeIfPresent(
reader.getSystemTopic().getTopicName().getNamespaceObject(), (k, v) -> true);
// replay policy message
- policiesCache.forEach(((topicName, topicPolicies) -> {
- if (listeners.get(topicName) != null) {
- for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
- listener.onUpdate(topicPolicies);
+ try {
+ policiesCache.forEach(((topicName, topicPolicies) -> {
+ if (listeners.get(topicName) != null) {
+ for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
+ listener.onUpdate(topicPolicies);
Review Comment:
If we complete directly, it may introduce data inconsistency? Complete directly or throws exception depends on the importance of this method. From the source pr(https://github.com/apache/pulsar/pull/11136) of this code, I think this method is very important.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zhanghaou commented on pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "zhanghaou (via GitHub)" <gi...@apache.org>.
zhanghaou commented on PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#issuecomment-1605902350
> > We shouldn't close the reader there
>
> cleanCacheAndCloseReader had delete from here. And I will make a new pr to fix some other use case.
https://github.com/apache/pulsar/pull/20640
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- merged pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- merged PR #20613:
URL: https://github.com/apache/pulsar/pull/20613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#discussion_r1241076829
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java:
##########
@@ -362,13 +362,20 @@ private void initPolicesCache(SystemTopicClient.Reader<PulsarEvent> reader, Comp
policyCacheInitMap.computeIfPresent(
reader.getSystemTopic().getTopicName().getNamespaceObject(), (k, v) -> true);
// replay policy message
- policiesCache.forEach(((topicName, topicPolicies) -> {
- if (listeners.get(topicName) != null) {
- for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
- listener.onUpdate(topicPolicies);
+ try {
+ policiesCache.forEach(((topicName, topicPolicies) -> {
+ if (listeners.get(topicName) != null) {
+ for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
+ listener.onUpdate(topicPolicies);
Review Comment:
If the listener throws an exception, it should be a bug. Do you have any ability to auto-fix it by retry?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#discussion_r1241005632
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java:
##########
@@ -362,13 +362,20 @@ private void initPolicesCache(SystemTopicClient.Reader<PulsarEvent> reader, Comp
policyCacheInitMap.computeIfPresent(
reader.getSystemTopic().getTopicName().getNamespaceObject(), (k, v) -> true);
// replay policy message
- policiesCache.forEach(((topicName, topicPolicies) -> {
- if (listeners.get(topicName) != null) {
- for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
- listener.onUpdate(topicPolicies);
+ try {
+ policiesCache.forEach(((topicName, topicPolicies) -> {
+ if (listeners.get(topicName) != null) {
+ for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
+ listener.onUpdate(topicPolicies);
+ }
}
- }
- }));
+ }));
+ } catch (Throwable e) {
+ log.error("[{}] Failed to replay policy message.", reader.getSystemTopic().getTopicName(), e);
+ future.completeExceptionally(e);
Review Comment:
The callback will invoke before `cleanCacheAndCloseReader`, which may cause some inconsistent status. :)
Considering someone who checks the cache and reader status in the future callback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zhanghaou commented on a diff in pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "zhanghaou (via GitHub)" <gi...@apache.org>.
zhanghaou commented on code in PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#discussion_r1241050135
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java:
##########
@@ -362,13 +362,20 @@ private void initPolicesCache(SystemTopicClient.Reader<PulsarEvent> reader, Comp
policyCacheInitMap.computeIfPresent(
reader.getSystemTopic().getTopicName().getNamespaceObject(), (k, v) -> true);
// replay policy message
- policiesCache.forEach(((topicName, topicPolicies) -> {
- if (listeners.get(topicName) != null) {
- for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
- listener.onUpdate(topicPolicies);
+ try {
+ policiesCache.forEach(((topicName, topicPolicies) -> {
+ if (listeners.get(topicName) != null) {
+ for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
+ listener.onUpdate(topicPolicies);
+ }
}
- }
- }));
+ }));
+ } catch (Throwable e) {
+ log.error("[{}] Failed to replay policy message.", reader.getSystemTopic().getTopicName(), e);
+ future.completeExceptionally(e);
+ cleanCacheAndCloseReader(reader.getSystemTopic().getTopicName().getNamespaceObject(), false);
Review Comment:
cleanCacheAndCloseReader had delete from here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] poorbarcode commented on a diff in pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#discussion_r1241023422
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java:
##########
@@ -362,13 +362,20 @@ private void initPolicesCache(SystemTopicClient.Reader<PulsarEvent> reader, Comp
policyCacheInitMap.computeIfPresent(
reader.getSystemTopic().getTopicName().getNamespaceObject(), (k, v) -> true);
// replay policy message
- policiesCache.forEach(((topicName, topicPolicies) -> {
- if (listeners.get(topicName) != null) {
- for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
- listener.onUpdate(topicPolicies);
+ try {
+ policiesCache.forEach(((topicName, topicPolicies) -> {
+ if (listeners.get(topicName) != null) {
+ for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
+ listener.onUpdate(topicPolicies);
+ }
}
- }
- }));
+ }));
+ } catch (Throwable e) {
+ log.error("[{}] Failed to replay policy message.", reader.getSystemTopic().getTopicName(), e);
+ future.completeExceptionally(e);
+ cleanCacheAndCloseReader(reader.getSystemTopic().getTopicName().getNamespaceObject(), false);
Review Comment:
We should not close the reader here because the caller will close it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zhanghaou commented on a diff in pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "zhanghaou (via GitHub)" <gi...@apache.org>.
zhanghaou commented on code in PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#discussion_r1241088229
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java:
##########
@@ -362,13 +362,20 @@ private void initPolicesCache(SystemTopicClient.Reader<PulsarEvent> reader, Comp
policyCacheInitMap.computeIfPresent(
reader.getSystemTopic().getTopicName().getNamespaceObject(), (k, v) -> true);
// replay policy message
- policiesCache.forEach(((topicName, topicPolicies) -> {
- if (listeners.get(topicName) != null) {
- for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
- listener.onUpdate(topicPolicies);
+ try {
+ policiesCache.forEach(((topicName, topicPolicies) -> {
+ if (listeners.get(topicName) != null) {
+ for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
+ listener.onUpdate(topicPolicies);
Review Comment:
We need to fix this bug instead of ignoring it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zhanghaou commented on pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "zhanghaou (via GitHub)" <gi...@apache.org>.
zhanghaou commented on PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#issuecomment-1605892695
> We shouldn't close the reader there
cleanCacheAndCloseReader had delete from here. And I will make a new pr to fix some other use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] mattisonchao commented on a diff in pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "mattisonchao (via GitHub)" <gi...@apache.org>.
mattisonchao commented on code in PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#discussion_r1241005632
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java:
##########
@@ -362,13 +362,20 @@ private void initPolicesCache(SystemTopicClient.Reader<PulsarEvent> reader, Comp
policyCacheInitMap.computeIfPresent(
reader.getSystemTopic().getTopicName().getNamespaceObject(), (k, v) -> true);
// replay policy message
- policiesCache.forEach(((topicName, topicPolicies) -> {
- if (listeners.get(topicName) != null) {
- for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
- listener.onUpdate(topicPolicies);
+ try {
+ policiesCache.forEach(((topicName, topicPolicies) -> {
+ if (listeners.get(topicName) != null) {
+ for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
+ listener.onUpdate(topicPolicies);
+ }
}
- }
- }));
+ }));
+ } catch (Throwable e) {
+ log.error("[{}] Failed to replay policy message.", reader.getSystemTopic().getTopicName(), e);
+ future.completeExceptionally(e);
Review Comment:
The callback will invoke before `cleanCacheAndCloseReader`, which may cause some inconsistent status. :)
1. Considering someone who checks the cache and reader status in the future callback.
2. Considering `completeExceptionally` throw exception. (logically)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zhanghaou commented on pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "zhanghaou (via GitHub)" <gi...@apache.org>.
zhanghaou commented on PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#issuecomment-1607172962
> I think we need to process like this :
>
> ```
> policiesCache.forEach(((topicName, topicPolicies) -> {
> if (listeners.get(topicName) != null) {
> for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
> try {
> listener.onUpdate(topicPolicies);
> } catch (Throwable error) {
> log.error("[{}] call listener error.", topicName, error);
> }
> }
> }
> }));
> ```
why?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on a diff in pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on code in PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#discussion_r1241029886
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java:
##########
@@ -362,13 +362,20 @@ private void initPolicesCache(SystemTopicClient.Reader<PulsarEvent> reader, Comp
policyCacheInitMap.computeIfPresent(
reader.getSystemTopic().getTopicName().getNamespaceObject(), (k, v) -> true);
// replay policy message
- policiesCache.forEach(((topicName, topicPolicies) -> {
- if (listeners.get(topicName) != null) {
- for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
- listener.onUpdate(topicPolicies);
+ try {
+ policiesCache.forEach(((topicName, topicPolicies) -> {
+ if (listeners.get(topicName) != null) {
+ for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
+ listener.onUpdate(topicPolicies);
+ }
}
- }
- }));
+ }));
+ } catch (Throwable e) {
+ log.error("[{}] Failed to replay policy message.", reader.getSystemTopic().getTopicName(), e);
+ future.completeExceptionally(e);
+ cleanCacheAndCloseReader(reader.getSystemTopic().getTopicName().getNamespaceObject(), false);
Review Comment:
yes, right. We shouldn't close the reader here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on a diff in pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on code in PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#discussion_r1236204247
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java:
##########
@@ -362,13 +362,20 @@ private void initPolicesCache(SystemTopicClient.Reader<PulsarEvent> reader, Comp
policyCacheInitMap.computeIfPresent(
reader.getSystemTopic().getTopicName().getNamespaceObject(), (k, v) -> true);
// replay policy message
- policiesCache.forEach(((topicName, topicPolicies) -> {
- if (listeners.get(topicName) != null) {
- for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
- listener.onUpdate(topicPolicies);
+ try {
+ policiesCache.forEach(((topicName, topicPolicies) -> {
+ if (listeners.get(topicName) != null) {
+ for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
+ listener.onUpdate(topicPolicies);
+ }
}
- }
- }));
+ }));
+ } catch (Exception e) {
Review Comment:
`Throwable` is better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codecov-commenter commented on pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#issuecomment-1612894833
## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/20613?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#20613](https://app.codecov.io/gh/apache/pulsar/pull/20613?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (2871cc9) into [master](https://app.codecov.io/gh/apache/pulsar/commit/2b01f83ea4f3ab7b22f4ea2b0290cc183e79bbb8?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (2b01f83) will **increase** coverage by `3.33%`.
> The diff coverage is `22.64%`.
[](https://app.codecov.io/gh/apache/pulsar/pull/20613?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
```diff
@@ Coverage Diff @@
## master #20613 +/- ##
============================================
+ Coverage 33.58% 36.92% +3.33%
+ Complexity 12127 12118 -9
============================================
Files 1613 1690 +77
Lines 126241 129192 +2951
Branches 13770 14091 +321
============================================
+ Hits 42396 47698 +5302
+ Misses 78331 75232 -3099
- Partials 5514 6262 +748
```
| Flag | Coverage Δ | |
|---|---|---|
| inttests | `24.14% <19.81%> (-0.05%)` | :arrow_down: |
| systests | `25.05% <16.98%> (?)` | |
| unittests | `32.03% <56.75%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://app.codecov.io/gh/apache/pulsar/pull/20613?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/pulsar/admin/cli/CmdNamespaces.java](https://app.codecov.io/gh/apache/pulsar/pull/20613?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC10b29scy9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2FkbWluL2NsaS9DbWROYW1lc3BhY2VzLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...in/java/org/apache/pulsar/admin/cli/CmdTopics.java](https://app.codecov.io/gh/apache/pulsar/pull/20613?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC10b29scy9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2FkbWluL2NsaS9DbWRUb3BpY3MuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [.../java/org/apache/pulsar/client/cli/CmdProduce.java](https://app.codecov.io/gh/apache/pulsar/pull/20613?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC10b29scy9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9jbGkvQ21kUHJvZHVjZS5qYXZh) | `25.90% <9.09%> (-0.76%)` | :arrow_down: |
| [.../service/SystemTopicBasedTopicPoliciesService.java](https://app.codecov.io/gh/apache/pulsar/pull/20613?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL1N5c3RlbVRvcGljQmFzZWRUb3BpY1BvbGljaWVzU2VydmljZS5qYXZh) | `57.14% <50.00%> (+0.87%)` | :arrow_up: |
| [...roker/service/persistent/MessageDeduplication.java](https://app.codecov.io/gh/apache/pulsar/pull/20613?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL3BlcnNpc3RlbnQvTWVzc2FnZURlZHVwbGljYXRpb24uamF2YQ==) | `40.17% <50.00%> (ø)` | |
| [...he/pulsar/client/impl/MultiTopicsConsumerImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/20613?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL011bHRpVG9waWNzQ29uc3VtZXJJbXBsLmphdmE=) | `32.81% <50.00%> (+0.44%)` | :arrow_up: |
| [.../pulsar/broker/stats/BrokerOperabilityMetrics.java](https://app.codecov.io/gh/apache/pulsar/pull/20613?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zdGF0cy9Ccm9rZXJPcGVyYWJpbGl0eU1ldHJpY3MuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...org/apache/pulsar/broker/stats/DimensionStats.java](https://app.codecov.io/gh/apache/pulsar/pull/20613?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zdGF0cy9EaW1lbnNpb25TdGF0cy5qYXZh) | `85.45% <100.00%> (ø)` | |
... and [400 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/20613/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- closed pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- closed pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
URL: https://github.com/apache/pulsar/pull/20613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Technoboy- commented on pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "Technoboy- (via GitHub)" <gi...@apache.org>.
Technoboy- commented on PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#issuecomment-1607329276
> > I think we need to process like this :
> > ```
> > policiesCache.forEach(((topicName, topicPolicies) -> {
> > if (listeners.get(topicName) != null) {
> > for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
> > try {
> > listener.onUpdate(topicPolicies);
> > } catch (Throwable error) {
> > log.error("[{}] call listener error.", topicName, error);
> > }
> > }
> > }
> > }));
> > ```
>
> why?
If the listener has an error, the reader cannot be closed here, because the reader is at the namespace level. Nor can other listeners not receive notifications because of a problem with one topic listener. So, only log output can be printed here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] zhanghaou commented on a diff in pull request #20613: [fix][broker] Topic policy can not be work well if replay policy message has any exception.
Posted by "zhanghaou (via GitHub)" <gi...@apache.org>.
zhanghaou commented on code in PR #20613:
URL: https://github.com/apache/pulsar/pull/20613#discussion_r1241050082
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/SystemTopicBasedTopicPoliciesService.java:
##########
@@ -362,13 +362,20 @@ private void initPolicesCache(SystemTopicClient.Reader<PulsarEvent> reader, Comp
policyCacheInitMap.computeIfPresent(
reader.getSystemTopic().getTopicName().getNamespaceObject(), (k, v) -> true);
// replay policy message
- policiesCache.forEach(((topicName, topicPolicies) -> {
- if (listeners.get(topicName) != null) {
- for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
- listener.onUpdate(topicPolicies);
+ try {
+ policiesCache.forEach(((topicName, topicPolicies) -> {
+ if (listeners.get(topicName) != null) {
+ for (TopicPolicyListener<TopicPolicies> listener : listeners.get(topicName)) {
+ listener.onUpdate(topicPolicies);
+ }
}
- }
- }));
+ }));
+ } catch (Throwable e) {
+ log.error("[{}] Failed to replay policy message.", reader.getSystemTopic().getTopicName(), e);
+ future.completeExceptionally(e);
Review Comment:
cleanCacheAndCloseReader had delete from here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org