You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2021/11/09 15:32:11 UTC
[GitHub] [pulsar] cbornet opened a new pull request #12699: Fix namespace policy override ignored when creating subscription
cbornet opened a new pull request #12699:
URL: https://github.com/apache/pulsar/pull/12699
### Motivation
PR #6471 introduced the possibility to set auto topic creation settings at the namespace level
PR #6685 is a fix to take into account the auto topic creation setting when creating a subscription
but it uses the broker configuration and doesn't see the namespace level setting that was introduced in #6471
### Modifications
This fix uses the method that fetches config overrides in ZK instead of the one from the broker configuration.
This way the method of the pulsar-admin uses the value set for the namespace for auto-topic-creation
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
This change added tests and can be verified as follows:
* Undo the changes in `PersistentTopicsBase.java` and the tests won't pass anymore
### Does this pull request potentially affect one of the following parts:
*If `yes` was chosen, please highlight the changes*
- Dependencies (does it add or upgrade a dependency): (yes / no) no
- The public API: (yes / no) no
- The schema: (yes / no / don't know) no
- The default values of configurations: (yes / no) no
- The wire protocol: (yes / no) no
- The rest endpoints: (yes / no) no
- The admin cli options: (yes / no) no
- Anything that affects deployment: (yes / no / don't know) no
### Documentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?
- [ ] `doc-required`
- [x] `no-need-doc`
It's a fix
- [ ] `doc`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat merged pull request #12699: Fix namespace policy override ignored when creating subscription
Posted by GitBox <gi...@apache.org>.
merlimat merged pull request #12699:
URL: https://github.com/apache/pulsar/pull/12699
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] cbornet commented on a change in pull request #12699: Fix namespace policy override ignored when creating subscription
Posted by GitBox <gi...@apache.org>.
cbornet commented on a change in pull request #12699:
URL: https://github.com/apache/pulsar/pull/12699#discussion_r746467768
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceAutoTopicCreationTest.java
##########
@@ -350,6 +350,37 @@ public void testNotAllowSubscriptionTopicCreation() throws Exception{
}
+ @Test
+ public void testAutoCreationNamespaceOverridesSubscriptionTopicCreation() throws Exception {
+ pulsar.getConfiguration().setAllowAutoTopicCreation(false);
+ String topicString = "persistent://prop/ns-abc/non-partitioned-topic" + System.currentTimeMillis();
+ String subscriptionName = "non-partitioned-topic-sub";
+ final TopicName topicName = TopicName.get(topicString);
+ pulsar.getAdminClient().namespaces().setAutoTopicCreation(topicName.getNamespace(),
+ AutoTopicCreationOverride.builder()
+ .allowAutoTopicCreation(true)
+ .topicType(TopicType.NON_PARTITIONED.toString())
+ .build());
+
+ admin.topics().createSubscription(topicString, subscriptionName, MessageId.earliest);
+ }
+
+ @Test
+ public void testAutoCreationNamespaceOverridesSubscriptionPartitionedTopicCreation() throws Exception {
+ pulsar.getConfiguration().setAllowAutoTopicCreation(false);
+ String topicString = "persistent://prop/ns-abc/partitioned-topic" + System.currentTimeMillis();
+ String subscriptionName = "partitioned-topic-sub";
+ final TopicName topicName = TopicName.get(topicString);
+ pulsar.getAdminClient().namespaces().setAutoTopicCreation(topicName.getNamespace(),
+ AutoTopicCreationOverride.builder()
+ .allowAutoTopicCreation(true)
+ .topicType(TopicType.PARTITIONED.toString())
+ .defaultNumPartitions(3)
+ .build());
+
+ admin.topics().createSubscription(topicString + "-partition-0", subscriptionName, MessageId.earliest);
Review comment:
Oh, I see. This test doesn't really make sense. I removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #12699: Fix namespace policy override ignored when creating subscription
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #12699:
URL: https://github.com/apache/pulsar/pull/12699#discussion_r746168576
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceAutoTopicCreationTest.java
##########
@@ -350,6 +350,37 @@ public void testNotAllowSubscriptionTopicCreation() throws Exception{
}
+ @Test
+ public void testAutoCreationNamespaceOverridesSubscriptionTopicCreation() throws Exception {
+ pulsar.getConfiguration().setAllowAutoTopicCreation(false);
+ String topicString = "persistent://prop/ns-abc/non-partitioned-topic" + System.currentTimeMillis();
+ String subscriptionName = "non-partitioned-topic-sub";
+ final TopicName topicName = TopicName.get(topicString);
+ pulsar.getAdminClient().namespaces().setAutoTopicCreation(topicName.getNamespace(),
+ AutoTopicCreationOverride.builder()
+ .allowAutoTopicCreation(true)
+ .topicType(TopicType.NON_PARTITIONED.toString())
+ .build());
+
+ admin.topics().createSubscription(topicString, subscriptionName, MessageId.earliest);
+ }
+
+ @Test
+ public void testAutoCreationNamespaceOverridesSubscriptionPartitionedTopicCreation() throws Exception {
+ pulsar.getConfiguration().setAllowAutoTopicCreation(false);
+ String topicString = "persistent://prop/ns-abc/partitioned-topic" + System.currentTimeMillis();
+ String subscriptionName = "partitioned-topic-sub";
+ final TopicName topicName = TopicName.get(topicString);
+ pulsar.getAdminClient().namespaces().setAutoTopicCreation(topicName.getNamespace(),
+ AutoTopicCreationOverride.builder()
+ .allowAutoTopicCreation(true)
+ .topicType(TopicType.PARTITIONED.toString())
+ .defaultNumPartitions(3)
+ .build());
+
+ admin.topics().createSubscription(topicString + "-partition-0", subscriptionName, MessageId.earliest);
Review comment:
Sorry, I don't understand why change the namespace level policy for topic auto-creation to create partitioned topic with 3 partitions, but here check if able to subscribe to one partition, looks like not able to ensure the partitioned topic has been created?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] cbornet commented on a change in pull request #12699: Fix namespace policy override ignored when creating subscription
Posted by GitBox <gi...@apache.org>.
cbornet commented on a change in pull request #12699:
URL: https://github.com/apache/pulsar/pull/12699#discussion_r746470768
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceAutoTopicCreationTest.java
##########
@@ -350,6 +350,37 @@ public void testNotAllowSubscriptionTopicCreation() throws Exception{
}
+ @Test
+ public void testAutoCreationNamespaceOverridesSubscriptionTopicCreation() throws Exception {
+ pulsar.getConfiguration().setAllowAutoTopicCreation(false);
+ String topicString = "persistent://prop/ns-abc/non-partitioned-topic" + System.currentTimeMillis();
+ String subscriptionName = "non-partitioned-topic-sub";
+ final TopicName topicName = TopicName.get(topicString);
+ pulsar.getAdminClient().namespaces().setAutoTopicCreation(topicName.getNamespace(),
+ AutoTopicCreationOverride.builder()
+ .allowAutoTopicCreation(true)
+ .topicType(TopicType.NON_PARTITIONED.toString())
+ .build());
+
+ admin.topics().createSubscription(topicString, subscriptionName, MessageId.earliest);
+ }
+
+ @Test
+ public void testAutoCreationNamespaceOverridesSubscriptionPartitionedTopicCreation() throws Exception {
+ pulsar.getConfiguration().setAllowAutoTopicCreation(false);
+ String topicString = "persistent://prop/ns-abc/partitioned-topic" + System.currentTimeMillis();
+ String subscriptionName = "partitioned-topic-sub";
+ final TopicName topicName = TopicName.get(topicString);
+ pulsar.getAdminClient().namespaces().setAutoTopicCreation(topicName.getNamespace(),
+ AutoTopicCreationOverride.builder()
+ .allowAutoTopicCreation(true)
+ .topicType(TopicType.PARTITIONED.toString())
+ .defaultNumPartitions(3)
+ .build());
+
+ admin.topics().createSubscription(topicString + "-partition-0", subscriptionName, MessageId.earliest);
Review comment:
Note that currently it's possible to create a topic partition from the subscription creation without the full partitioned topic being created. Is it OK to be able to do that ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] cbornet commented on a change in pull request #12699: Fix namespace policy override ignored when creating subscription
Posted by GitBox <gi...@apache.org>.
cbornet commented on a change in pull request #12699:
URL: https://github.com/apache/pulsar/pull/12699#discussion_r745780962
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceAutoTopicCreationTest.java
##########
@@ -350,6 +350,37 @@ public void testNotAllowSubscriptionTopicCreation() throws Exception{
}
+ @Test
+ public void testAutoCreationNamespaceOverridesSubscriptionTopicCreation() throws Exception {
+ pulsar.getConfiguration().setAllowAutoTopicCreation(false);
+ String topicString = "persistent://prop/ns-abc/non-partitioned-topic" + System.currentTimeMillis();
+ String subscriptionName = "non-partitioned-topic-sub";
+ final TopicName topicName = TopicName.get(topicString);
+ pulsar.getAdminClient().namespaces().setAutoTopicCreation(topicName.getNamespace(),
+ AutoTopicCreationOverride.builder()
+ .allowAutoTopicCreation(true)
+ .topicType(TopicType.NON_PARTITIONED.toString())
+ .build());
+
+ admin.topics().createSubscription(topicString, subscriptionName, MessageId.earliest);
+ }
+
+ @Test
+ public void testAutoCreationNamespaceOverridesSubscriptionPartitionedTopicCreation() throws Exception {
+ pulsar.getConfiguration().setAllowAutoTopicCreation(false);
+ String topicString = "persistent://prop/ns-abc/partitioned-topic" + System.currentTimeMillis();
+ String subscriptionName = "partitioned-topic-sub";
+ final TopicName topicName = TopicName.get(topicString);
+ pulsar.getAdminClient().namespaces().setAutoTopicCreation(topicName.getNamespace(),
+ AutoTopicCreationOverride.builder()
+ .allowAutoTopicCreation(true)
+ .topicType(TopicType.PARTITIONED.toString())
+ .defaultNumPartitions(3)
+ .build());
+
+ admin.topics().createSubscription(topicString + "-partition-0", subscriptionName, MessageId.earliest);
Review comment:
Yes that's the intent. I've seen it done [here](https://github.com/apache/pulsar/blob/da71ec2b2a6682ffbab60843756f638bedbd4460/pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/CreateSubscriptionTest.java#L112) so I guess it's OK.
Also this way the topic is created by `pulsar().getBrokerService().getTopic()` in `internalCreateSubscriptionForNonPartitionedTopic` instead of `pulsar().getBrokerService().fetchPartitionedTopicMetadataCheckAllowAutoCreationAsync()` when you don't specify a partition.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on a change in pull request #12699: Fix namespace policy override ignored when creating subscription
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on a change in pull request #12699:
URL: https://github.com/apache/pulsar/pull/12699#discussion_r745755144
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceAutoTopicCreationTest.java
##########
@@ -350,6 +350,37 @@ public void testNotAllowSubscriptionTopicCreation() throws Exception{
}
+ @Test
+ public void testAutoCreationNamespaceOverridesSubscriptionTopicCreation() throws Exception {
+ pulsar.getConfiguration().setAllowAutoTopicCreation(false);
+ String topicString = "persistent://prop/ns-abc/non-partitioned-topic" + System.currentTimeMillis();
+ String subscriptionName = "non-partitioned-topic-sub";
+ final TopicName topicName = TopicName.get(topicString);
+ pulsar.getAdminClient().namespaces().setAutoTopicCreation(topicName.getNamespace(),
+ AutoTopicCreationOverride.builder()
+ .allowAutoTopicCreation(true)
+ .topicType(TopicType.NON_PARTITIONED.toString())
+ .build());
+
+ admin.topics().createSubscription(topicString, subscriptionName, MessageId.earliest);
+ }
+
+ @Test
+ public void testAutoCreationNamespaceOverridesSubscriptionPartitionedTopicCreation() throws Exception {
+ pulsar.getConfiguration().setAllowAutoTopicCreation(false);
+ String topicString = "persistent://prop/ns-abc/partitioned-topic" + System.currentTimeMillis();
+ String subscriptionName = "partitioned-topic-sub";
+ final TopicName topicName = TopicName.get(topicString);
+ pulsar.getAdminClient().namespaces().setAutoTopicCreation(topicName.getNamespace(),
+ AutoTopicCreationOverride.builder()
+ .allowAutoTopicCreation(true)
+ .topicType(TopicType.PARTITIONED.toString())
+ .defaultNumPartitions(3)
+ .build());
+
+ admin.topics().createSubscription(topicString + "-partition-0", subscriptionName, MessageId.earliest);
Review comment:
Do you want to subscribe a to specific partition?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] cbornet commented on a change in pull request #12699: Fix namespace policy override ignored when creating subscription
Posted by GitBox <gi...@apache.org>.
cbornet commented on a change in pull request #12699:
URL: https://github.com/apache/pulsar/pull/12699#discussion_r755880055
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/service/BrokerServiceAutoTopicCreationTest.java
##########
@@ -350,6 +350,37 @@ public void testNotAllowSubscriptionTopicCreation() throws Exception{
}
+ @Test
+ public void testAutoCreationNamespaceOverridesSubscriptionTopicCreation() throws Exception {
+ pulsar.getConfiguration().setAllowAutoTopicCreation(false);
+ String topicString = "persistent://prop/ns-abc/non-partitioned-topic" + System.currentTimeMillis();
+ String subscriptionName = "non-partitioned-topic-sub";
+ final TopicName topicName = TopicName.get(topicString);
+ pulsar.getAdminClient().namespaces().setAutoTopicCreation(topicName.getNamespace(),
+ AutoTopicCreationOverride.builder()
+ .allowAutoTopicCreation(true)
+ .topicType(TopicType.NON_PARTITIONED.toString())
+ .build());
+
+ admin.topics().createSubscription(topicString, subscriptionName, MessageId.earliest);
+ }
+
+ @Test
+ public void testAutoCreationNamespaceOverridesSubscriptionPartitionedTopicCreation() throws Exception {
+ pulsar.getConfiguration().setAllowAutoTopicCreation(false);
+ String topicString = "persistent://prop/ns-abc/partitioned-topic" + System.currentTimeMillis();
+ String subscriptionName = "partitioned-topic-sub";
+ final TopicName topicName = TopicName.get(topicString);
+ pulsar.getAdminClient().namespaces().setAutoTopicCreation(topicName.getNamespace(),
+ AutoTopicCreationOverride.builder()
+ .allowAutoTopicCreation(true)
+ .topicType(TopicType.PARTITIONED.toString())
+ .defaultNumPartitions(3)
+ .build());
+
+ admin.topics().createSubscription(topicString + "-partition-0", subscriptionName, MessageId.earliest);
Review comment:
@codelipenghui is it OK for you ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org