You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/05/29 08:29:05 UTC
[GitHub] [pulsar] AnonHxy opened a new pull request, #15825: [fix][broker]Fix topiclevel replicator rate limiter not init
AnonHxy opened a new pull request, #15825:
URL: https://github.com/apache/pulsar/pull/15825
### Motivation
* Fix bug: If only topic-level replicator rate limiter set, the limiter will not be initialized
### Modifications
* Add the method `org.apache.pulsar.broker.service.Replicator#updateRateLimiter` to initialize or update the replicator rate limiter.
* Use this method to initialize or update all level replicator rate limiter
### Verifying this change
- [x] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
*(example:)*
- *Added UT `org.apache.pulsar.broker.service.ReplicatorRateLimiterTest#testReplicatorRateLimiterWithOnlyTopicLevel`
### Documentation
- [x] `doc-not-needed`
(Please explain why)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui merged pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
codelipenghui merged PR #15825:
URL: https://github.com/apache/pulsar/pull/15825
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on a diff in pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
Jason918 commented on code in PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#discussion_r886263458
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java:
##########
@@ -704,10 +704,19 @@ public Optional<DispatchRateLimiter> getRateLimiter() {
}
@Override
- public void initializeDispatchRateLimiterIfNeeded() {
+ public boolean initializeDispatchRateLimiterIfNeeded() {
if (!dispatchRateLimiter.isPresent()
&& DispatchRateLimiter.isDispatchRateEnabled(topic.getReplicatorDispatchRate())) {
this.dispatchRateLimiter = Optional.of(new DispatchRateLimiter(topic, Type.REPLICATOR));
+ return true;
+ }
+ return false;
+ }
+
+ @Override
+ public void updateRateLimiter() {
Review Comment:
Is it possible that this method will be called concurrently?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#issuecomment-1149594825
@mattisonchao @Technoboy- @codelipenghui PTAL ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#issuecomment-1151961781
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#issuecomment-1140618281
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#issuecomment-1151353059
Thx, I have updated. PTAL again @codelipenghui
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#issuecomment-1152230333
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on a diff in pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on code in PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#discussion_r886695725
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java:
##########
@@ -704,10 +705,23 @@ public Optional<DispatchRateLimiter> getRateLimiter() {
}
@Override
- public void initializeDispatchRateLimiterIfNeeded() {
- if (!dispatchRateLimiter.isPresent()
- && DispatchRateLimiter.isDispatchRateEnabled(topic.getReplicatorDispatchRate())) {
- this.dispatchRateLimiter = Optional.of(new DispatchRateLimiter(topic, Type.REPLICATOR));
+ public boolean initializeDispatchRateLimiterIfNeeded() {
+ synchronized (dispatchRateLimiterLock) {
+ if (!dispatchRateLimiter.isPresent()
+ && DispatchRateLimiter.isDispatchRateEnabled(topic.getReplicatorDispatchRate())) {
+ this.dispatchRateLimiter = Optional.of(new DispatchRateLimiter(topic, Type.REPLICATOR));
+ return true;
+ }
+ return false;
+ }
+ }
+
+ @Override
+ public void updateRateLimiter() {
+ synchronized (dispatchRateLimiterLock) {
Review Comment:
@Jason918 It seems not necessary. I will remove the `synchronized ` here. Because `updateDispatchRate(dispatchRate)`, which is located at Line203 is synchronized:
https://github.com/apache/pulsar/blob/d4be6bb05204c270122e159c5c361cd7a2412805/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/DispatchRateLimiter.java#L178-L204
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#issuecomment-1152150385
> 1. Why not initialize replicator like initializeDispatchRateLimiterIfNeeded ?
If we do that, when broker updating replicator limiter dynamicly, it need call `PersistentReplicator#initializeDispatchRateLimiterIfNeeded` before line2420 to initialize the limiter. But this method is not `synchronized`.
https://github.com/apache/pulsar/blob/3525fe478e123734f04864811089411b4a3fdeb4/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java#L2412-L2421
However if we `synchronized` this method, updating or initializing replicator limiter will both hold `dispatchRateLimiterLock` and `PersistentReplicatorLock`, see line:380 and line 388. So I think it is better to abstract `Replicator#updateRateLimiter()`, which will benefits for broker-level 、namespace-level and topic-level.
https://github.com/apache/pulsar/blob/3525fe478e123734f04864811089411b4a3fdeb4/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentTopic.java#L379-L391
> 2. Even if we can do like this patch, no need to change the return type of initializeDispatchRateLimiterIfNeeded
Make sense, I have update to fix this. Thanks very much @Technoboy-
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#issuecomment-1153501529
> @AnonHxy Could you please help create a PR for branch-2.10 directly? Due to the topic-level policy optimization, there are lots of conflicts when cherry-picking.
Yes. PTAL @codelipenghui https://github.com/apache/pulsar/pull/16034
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on a diff in pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on code in PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#discussion_r886341619
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java:
##########
@@ -704,10 +704,19 @@ public Optional<DispatchRateLimiter> getRateLimiter() {
}
@Override
- public void initializeDispatchRateLimiterIfNeeded() {
+ public boolean initializeDispatchRateLimiterIfNeeded() {
if (!dispatchRateLimiter.isPresent()
&& DispatchRateLimiter.isDispatchRateEnabled(topic.getReplicatorDispatchRate())) {
this.dispatchRateLimiter = Optional.of(new DispatchRateLimiter(topic, Type.REPLICATOR));
+ return true;
+ }
+ return false;
+ }
+
+ @Override
+ public void updateRateLimiter() {
Review Comment:
Yes, I have updated the code to synchronize `initializeDispatchRateLimiterIfNeeded` and `updateRateLimiter`. PTAL again @Jason918 ~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] Jason918 commented on a diff in pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
Jason918 commented on code in PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#discussion_r886374143
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java:
##########
@@ -704,10 +705,23 @@ public Optional<DispatchRateLimiter> getRateLimiter() {
}
@Override
- public void initializeDispatchRateLimiterIfNeeded() {
- if (!dispatchRateLimiter.isPresent()
- && DispatchRateLimiter.isDispatchRateEnabled(topic.getReplicatorDispatchRate())) {
- this.dispatchRateLimiter = Optional.of(new DispatchRateLimiter(topic, Type.REPLICATOR));
+ public boolean initializeDispatchRateLimiterIfNeeded() {
+ synchronized (dispatchRateLimiterLock) {
+ if (!dispatchRateLimiter.isPresent()
+ && DispatchRateLimiter.isDispatchRateEnabled(topic.getReplicatorDispatchRate())) {
+ this.dispatchRateLimiter = Optional.of(new DispatchRateLimiter(topic, Type.REPLICATOR));
+ return true;
+ }
+ return false;
+ }
+ }
+
+ @Override
+ public void updateRateLimiter() {
+ synchronized (dispatchRateLimiterLock) {
Review Comment:
Is this necessary? `initializeDispatchRateLimiterIfNeeded` already has synchronized `dispatchRateLimiterLock`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on a diff in pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on code in PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#discussion_r886695725
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentReplicator.java:
##########
@@ -704,10 +705,23 @@ public Optional<DispatchRateLimiter> getRateLimiter() {
}
@Override
- public void initializeDispatchRateLimiterIfNeeded() {
- if (!dispatchRateLimiter.isPresent()
- && DispatchRateLimiter.isDispatchRateEnabled(topic.getReplicatorDispatchRate())) {
- this.dispatchRateLimiter = Optional.of(new DispatchRateLimiter(topic, Type.REPLICATOR));
+ public boolean initializeDispatchRateLimiterIfNeeded() {
+ synchronized (dispatchRateLimiterLock) {
+ if (!dispatchRateLimiter.isPresent()
+ && DispatchRateLimiter.isDispatchRateEnabled(topic.getReplicatorDispatchRate())) {
+ this.dispatchRateLimiter = Optional.of(new DispatchRateLimiter(topic, Type.REPLICATOR));
+ return true;
+ }
+ return false;
+ }
+ }
+
+ @Override
+ public void updateRateLimiter() {
+ synchronized (dispatchRateLimiterLock) {
Review Comment:
@Jason918 It seems not necessary. I will remove the `synchronized ` here. Because `updateDispatchRate(dispatchRate)` is synchronized, which located at Line203 :
https://github.com/apache/pulsar/blob/d4be6bb05204c270122e159c5c361cd7a2412805/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/DispatchRateLimiter.java#L178-L204
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#issuecomment-1145666322
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#issuecomment-1142179424
@Jason918 @HQebupt PTAL
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#issuecomment-1152133601
> 1. Why not initialize replicator like initializeDispatchRateLimiterIfNeeded ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#issuecomment-1144850584
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#issuecomment-1140451070
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] AnonHxy commented on pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
AnonHxy commented on PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#issuecomment-1152273690
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #15825: [fix][broker]Fix topic-level replicator rate limiter not init
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on PR #15825:
URL: https://github.com/apache/pulsar/pull/15825#issuecomment-1153391736
@AnonHxy Could you please help create a PR for branch-2.10 directly? Due to the topic-level policy optimization, there are lots of conflicts when cherry-picking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org