You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by "poorbarcode (via GitHub)" <gi...@apache.org> on 2024/08/14 09:52:28 UTC
[PR] [draft] [improve] [broker] Phase 1 of PIP-370 support disable create topics on remote cluster through replication [pulsar]
poorbarcode opened a new pull request, #23169:
URL: https://github.com/apache/pulsar/pull/23169
### Motivation
see also [PIP-370: configurable remote topic creation in geo-replication ](https://github.com/apache/pulsar/pull/23124)
### Modifications
- Phase 1 defined in PIP-370: link...
### Next PRs
- Phase 2:
- Phase 3:
### Documentation
<!-- DO NOT REMOVE THIS SECTION. CHECK THE PROPER BOX ONLY. -->
- [ ] `doc` <!-- Your PR contains doc changes. -->
- [ ] `doc-required` <!-- Your PR changes impact docs and you will update later -->
- [x] `doc-not-needed` <!-- Your PR changes do not impact docs -->
- [ ] `doc-complete` <!-- Docs have been already added -->
### Matching PR in forked repository
PR in forked repository: x
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve] [broker] Phase 1 of PIP-370 support disable create topics on remote cluster through replication [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #23169:
URL: https://github.com/apache/pulsar/pull/23169#discussion_r1723164876
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/OneWayReplicatorTest.java:
##########
@@ -1160,4 +1160,77 @@ public void testDifferentTopicCreationRule(ReplicationMode replicationMode) thro
admin1.namespaces().deleteNamespace(ns);
admin2.namespaces().deleteNamespace(ns);
}
+
+ @Test(dataProvider = "replicationModes")
+ public void testDisabledCreateTopicToRemoteClusterForReplication(ReplicationMode replicationMode) throws Exception {
Review Comment:
This test was added in mistake, I removed this test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve] [broker] Phase 1 of PIP-370 support disable create topics on remote cluster through replication [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #23169:
URL: https://github.com/apache/pulsar/pull/23169#discussion_r1723164547
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java:
##########
@@ -3332,6 +3332,11 @@ protected CompletableFuture<Void> internalSetReplicationClusters(List<String> cl
}
return FutureUtil.waitForAll(futures);
}).thenCompose(__ -> {
+ if (!pulsar().getConfig().isCreateTopicToRemoteClusterForReplication()) {
+ log.info("[{}] Skip creating partitions for topic {} for the remote clusters {}",
+ clientAppId(), topicName, pulsar().getConfiguration().getClusterName());
Review Comment:
Added remote clusters in the log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve] [broker] Phase 1 of PIP-370 support disable create topics on remote cluster through replication [pulsar]
Posted by "lhotari (via GitHub)" <gi...@apache.org>.
lhotari commented on PR #23169:
URL: https://github.com/apache/pulsar/pull/23169#issuecomment-2342706792
@poorbarcode why did we cherry-picked PIP-370 changes to branch-3.0, but branch-3.3 was skipped? Asked that on the dev mailing list thread, https://lists.apache.org/thread/onvgpx4vr1184tzjl2ngxog674zgltgk .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve] [broker] Phase 1 of PIP-370 support disable create topics on remote cluster through replication [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode merged PR #23169:
URL: https://github.com/apache/pulsar/pull/23169
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve] [broker] Phase 1 of PIP-370 support disable create topics on remote cluster through replication [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on code in PR #23169:
URL: https://github.com/apache/pulsar/pull/23169#discussion_r1722698834
##########
pulsar-broker/src/test/java/org/apache/pulsar/broker/service/OneWayReplicatorTest.java:
##########
@@ -1160,4 +1160,77 @@ public void testDifferentTopicCreationRule(ReplicationMode replicationMode) thro
admin1.namespaces().deleteNamespace(ns);
admin2.namespaces().deleteNamespace(ns);
}
+
+ @Test(dataProvider = "replicationModes")
+ public void testDisabledCreateTopicToRemoteClusterForReplication(ReplicationMode replicationMode) throws Exception {
Review Comment:
Could you point out where disabled `createTopicToRemoteClusterForReplication` ?
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java:
##########
@@ -609,11 +609,15 @@ protected void internalCreatePartitionedTopic(AsyncResponse asyncResponse, int n
.thenCompose(__ -> provisionPartitionedTopicPath(numPartitions, createLocalTopicOnly, properties))
.thenCompose(__ -> tryCreatePartitionsAsync(numPartitions))
.thenRun(() -> {
- if (!createLocalTopicOnly && topicName.isGlobal()) {
+ if (!createLocalTopicOnly && topicName.isGlobal()
+ && pulsar().getConfig().isCreateTopicToRemoteClusterForReplication()) {
internalCreatePartitionedTopicToReplicatedClustersInBackground(numPartitions);
+ log.info("[{}] Successfully created partitions for topic {} for the remote clusters {}",
Review Comment:
```suggestion
log.info("[{}] Successfully created partitioned topic {} to remote clusters {}",
```
Can we list the remote clusters in the log? It will be better than log the current cluster.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/PersistentTopicsBase.java:
##########
@@ -3332,6 +3332,11 @@ protected CompletableFuture<Void> internalSetReplicationClusters(List<String> cl
}
return FutureUtil.waitForAll(futures);
}).thenCompose(__ -> {
+ if (!pulsar().getConfig().isCreateTopicToRemoteClusterForReplication()) {
+ log.info("[{}] Skip creating partitions for topic {} for the remote clusters {}",
+ clientAppId(), topicName, pulsar().getConfiguration().getClusterName());
Review Comment:
Same as the above comment.
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java:
##########
@@ -609,11 +609,15 @@ protected void internalCreatePartitionedTopic(AsyncResponse asyncResponse, int n
.thenCompose(__ -> provisionPartitionedTopicPath(numPartitions, createLocalTopicOnly, properties))
.thenCompose(__ -> tryCreatePartitionsAsync(numPartitions))
.thenRun(() -> {
- if (!createLocalTopicOnly && topicName.isGlobal()) {
+ if (!createLocalTopicOnly && topicName.isGlobal()
+ && pulsar().getConfig().isCreateTopicToRemoteClusterForReplication()) {
internalCreatePartitionedTopicToReplicatedClustersInBackground(numPartitions);
+ log.info("[{}] Successfully created partitions for topic {} for the remote clusters {}",
+ clientAppId(), topicName, pulsar().getConfiguration().getClusterName());
+ } else {
+ log.info("[{}] Skip creating partitions for topic {} for the remote clusters {}",
Review Comment:
```suggestion
log.info("[{}] Skip creating partitioned topic {} to remote clusters {}",
```
Can we list the remote clusters in the log? It will be better than log the current cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve] [broker] Phase 1 of PIP-370 support disable create topics on remote cluster through replication [pulsar]
Posted by "poorbarcode (via GitHub)" <gi...@apache.org>.
poorbarcode commented on code in PR #23169:
URL: https://github.com/apache/pulsar/pull/23169#discussion_r1723164163
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java:
##########
@@ -609,11 +609,15 @@ protected void internalCreatePartitionedTopic(AsyncResponse asyncResponse, int n
.thenCompose(__ -> provisionPartitionedTopicPath(numPartitions, createLocalTopicOnly, properties))
.thenCompose(__ -> tryCreatePartitionsAsync(numPartitions))
.thenRun(() -> {
- if (!createLocalTopicOnly && topicName.isGlobal()) {
+ if (!createLocalTopicOnly && topicName.isGlobal()
+ && pulsar().getConfig().isCreateTopicToRemoteClusterForReplication()) {
internalCreatePartitionedTopicToReplicatedClustersInBackground(numPartitions);
+ log.info("[{}] Successfully created partitions for topic {} for the remote clusters {}",
Review Comment:
Since we can hardly get remote clusters here, I removed the part of the log
##########
pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java:
##########
@@ -609,11 +609,15 @@ protected void internalCreatePartitionedTopic(AsyncResponse asyncResponse, int n
.thenCompose(__ -> provisionPartitionedTopicPath(numPartitions, createLocalTopicOnly, properties))
.thenCompose(__ -> tryCreatePartitionsAsync(numPartitions))
.thenRun(() -> {
- if (!createLocalTopicOnly && topicName.isGlobal()) {
+ if (!createLocalTopicOnly && topicName.isGlobal()
+ && pulsar().getConfig().isCreateTopicToRemoteClusterForReplication()) {
internalCreatePartitionedTopicToReplicatedClustersInBackground(numPartitions);
+ log.info("[{}] Successfully created partitions for topic {} for the remote clusters {}",
+ clientAppId(), topicName, pulsar().getConfiguration().getClusterName());
+ } else {
+ log.info("[{}] Skip creating partitions for topic {} for the remote clusters {}",
Review Comment:
Since we can hardly get remote clusters here, I removed the part of the log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve] [broker] Phase 1 of PIP-370 support disable create topics on remote cluster through replication [pulsar]
Posted by "codelipenghui (via GitHub)" <gi...@apache.org>.
codelipenghui commented on PR #23169:
URL: https://github.com/apache/pulsar/pull/23169#issuecomment-2299178805
/pulsarbot run-failure-checks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [improve] [broker] Phase 1 of PIP-370 support disable create topics on remote cluster through replication [pulsar]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #23169:
URL: https://github.com/apache/pulsar/pull/23169#issuecomment-2300544627
## [Codecov](https://app.codecov.io/gh/apache/pulsar/pull/23169?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `83.78378%` with `6 lines` in your changes missing coverage. Please review.
> Project coverage is 74.58%. Comparing base [(`bbc6224`)](https://app.codecov.io/gh/apache/pulsar/commit/bbc62245c5ddba1de4b1e7cee4ab49334bc36277?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`34a9816`)](https://app.codecov.io/gh/apache/pulsar/commit/34a981684dcd0fffe70888abaf7d8f240e985816?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 539 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/pulsar/pull/23169?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...er/service/persistent/GeoPersistentReplicator.java](https://app.codecov.io/gh/apache/pulsar/pull/23169?src=pr&el=tree&filepath=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2Fpersistent%2FGeoPersistentReplicator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL3BlcnNpc3RlbnQvR2VvUGVyc2lzdGVudFJlcGxpY2F0b3IuamF2YQ==) | 62.50% | [5 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pulsar/pull/23169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
[](https://app.codecov.io/gh/apache/pulsar/pull/23169?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
```diff
@@ Coverage Diff @@
## master #23169 +/- ##
============================================
+ Coverage 73.57% 74.58% +1.01%
- Complexity 32624 34211 +1587
============================================
Files 1877 1918 +41
Lines 139502 144657 +5155
Branches 15299 15813 +514
============================================
+ Hits 102638 107898 +5260
+ Misses 28908 28503 -405
- Partials 7956 8256 +300
```
| [Flag](https://app.codecov.io/gh/apache/pulsar/pull/23169/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [inttests](https://app.codecov.io/gh/apache/pulsar/pull/23169/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `27.69% <37.83%> (+3.10%)` | :arrow_up: |
| [systests](https://app.codecov.io/gh/apache/pulsar/pull/23169/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `24.75% <10.81%> (+0.42%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pulsar/pull/23169/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `73.93% <83.78%> (+1.09%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files](https://app.codecov.io/gh/apache/pulsar/pull/23169?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...org/apache/pulsar/broker/ServiceConfiguration.java](https://app.codecov.io/gh/apache/pulsar/pull/23169?src=pr&el=tree&filepath=pulsar-broker-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2FServiceConfiguration.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3B1bHNhci9icm9rZXIvU2VydmljZUNvbmZpZ3VyYXRpb24uamF2YQ==) | `99.03% <100.00%> (-0.36%)` | :arrow_down: |
| [.../org/apache/pulsar/broker/admin/AdminResource.java](https://app.codecov.io/gh/apache/pulsar/pull/23169?src=pr&el=tree&filepath=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fadmin%2FAdminResource.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi9BZG1pblJlc291cmNlLmphdmE=) | `76.26% <100.00%> (-1.37%)` | :arrow_down: |
| [...pulsar/broker/admin/impl/PersistentTopicsBase.java](https://app.codecov.io/gh/apache/pulsar/pull/23169?src=pr&el=tree&filepath=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fadmin%2Fimpl%2FPersistentTopicsBase.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9hZG1pbi9pbXBsL1BlcnNpc3RlbnRUb3BpY3NCYXNlLmphdmE=) | `69.71% <100.00%> (+4.25%)` | :arrow_up: |
| [...ache/pulsar/broker/service/AbstractReplicator.java](https://app.codecov.io/gh/apache/pulsar/pull/23169?src=pr&el=tree&filepath=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2FAbstractReplicator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL0Fic3RyYWN0UmVwbGljYXRvci5qYXZh) | `65.80% <100.00%> (-19.20%)` | :arrow_down: |
| [...rg/apache/pulsar/client/impl/PulsarClientImpl.java](https://app.codecov.io/gh/apache/pulsar/pull/23169?src=pr&el=tree&filepath=pulsar-client%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fclient%2Fimpl%2FPulsarClientImpl.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2NsaWVudC9pbXBsL1B1bHNhckNsaWVudEltcGwuamF2YQ==) | `75.55% <100.00%> (+1.24%)` | :arrow_up: |
| [...er/service/persistent/GeoPersistentReplicator.java](https://app.codecov.io/gh/apache/pulsar/pull/23169?src=pr&el=tree&filepath=pulsar-broker%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpulsar%2Fbroker%2Fservice%2Fpersistent%2FGeoPersistentReplicator.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cHVsc2FyLWJyb2tlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcHVsc2FyL2Jyb2tlci9zZXJ2aWNlL3BlcnNpc3RlbnQvR2VvUGVyc2lzdGVudFJlcGxpY2F0b3IuamF2YQ==) | `76.14% <62.50%> (-1.88%)` | :arrow_down: |
... and [525 files with indirect coverage changes](https://app.codecov.io/gh/apache/pulsar/pull/23169/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org