You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/12/23 12:58:36 UTC
[GitHub] [pulsar] hangc0276 opened a new pull request #9042: support namespace policy for maxTopicsPerNamespace limit
hangc0276 opened a new pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042
Related to #8225
#### Changes
1. Support namespace policy for `maxTopicsPerNamespace` limit
2. Add the tests for this feature
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
315157973 commented on pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042#issuecomment-753251035
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042#issuecomment-750714700
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] codelipenghui commented on pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
codelipenghui commented on pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042#issuecomment-750296223
@315157973 Could you please help review this PR?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on a change in pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on a change in pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042#discussion_r548363020
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
##########
@@ -2517,9 +2517,18 @@ private void checkTopicLevelPolicyEnable() {
private <T> boolean checkMaxTopicsPerNamespace(TopicName topicName, int numPartitions,
CompletableFuture<T> topicFuture) {
- final int maxTopicsPerNamespace = pulsar().getConfig().getMaxTopicsPerNamespace();
- if (maxTopicsPerNamespace > 0) {
- try {
+ Integer maxTopicsPerNamespace;
+ try {
+ maxTopicsPerNamespace = pulsar.getConfigurationCache().policiesCache()
+ .get(AdminResource.path(POLICIES, topicName.getNamespace()))
+ .map(p -> p.max_topics_per_namespace)
+ .orElse(null);
+
+ if (maxTopicsPerNamespace == null || maxTopicsPerNamespace < 0) {
Review comment:
I remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 removed a comment on pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
315157973 removed a comment on pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042#issuecomment-753251035
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on a change in pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on a change in pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042#discussion_r548338780
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java
##########
@@ -811,9 +811,14 @@ protected boolean isNamespaceReplicated(NamespaceName namespaceName) {
}
protected void internalCreatePartitionedTopic(AsyncResponse asyncResponse, int numPartitions) {
- final int maxTopicsPerNamespace = pulsar().getConfig().getMaxTopicsPerNamespace();
- if (maxTopicsPerNamespace > 0) {
- try {
+ Integer maxTopicsPerNamespace;
+ try {
+ maxTopicsPerNamespace = getNamespacePolicies(namespaceName).max_topics_per_namespace;
+ if (maxTopicsPerNamespace == null || maxTopicsPerNamespace < 0) {
Review comment:
OK, `maxTopicsPerNamespace` is get from namespace policy, it must be greater than 0, i update the code.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
315157973 commented on pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042#issuecomment-751158482
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] sijie merged pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
sijie merged pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on a change in pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on a change in pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042#discussion_r548339312
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/NamespacesTest.java
##########
@@ -1413,6 +1414,132 @@ public void testRetentionPolicyValidation() throws Exception {
admin.namespaces().deleteNamespace(namespace);
}
+ @Test(timeOut = 30000)
+ public void testMaxTopicsPerNamespace() throws Exception {
+ super.internalCleanup();
+ conf.setMaxTopicsPerNamespace(15);
+ super.internalSetup();
+
+ String namespace = "testTenant/ns1";
+ admin.clusters().createCluster("use", new ClusterData(brokerUrl.toString()));
+ TenantInfo tenantInfo = new TenantInfo(Sets.newHashSet("role1", "role2"),
+ Sets.newHashSet("use"));
+ admin.tenants().createTenant("testTenant", tenantInfo);
+ admin.namespaces().createNamespace(namespace, Sets.newHashSet("use"));
+
+ admin.namespaces().setMaxTopicsPerNamespace(namespace, 10);
+ assertEquals(10, admin.namespaces().getMaxTopicsPerNamespace(namespace));
+
+ // check create partitioned/non-partitioned topics using namespace policy
+ String topic = "persistent://testTenant/ns1/test_create_topic_v";
+ admin.topics().createPartitionedTopic(topic + "1", 2);
+ admin.topics().createPartitionedTopic(topic + "2", 3);
+ admin.topics().createPartitionedTopic(topic + "3", 4);
+ admin.topics().createNonPartitionedTopic(topic + "4");
+
+ try {
+ admin.topics().createPartitionedTopic(topic + "5", 2);
+ fail();
+ } catch (PulsarAdminException e) {
+ assertEquals(e.getStatusCode(), 412);
+ assertEquals(e.getHttpError(), "Exceed maximum number of topics in namespace.");
+ }
+
+ // remove namespace policy limit, use broker configuration instead.
+ admin.namespaces().removeMaxTopicsPerNamespace(namespace);
+ admin.topics().createPartitionedTopic(topic + "6", 4);
+ try {
+ admin.topics().createPartitionedTopic(topic + "7", 3);
+ fail();
+ } catch (PulsarAdminException e) {
+ assertEquals(e.getStatusCode(), 412);
+ assertEquals(e.getHttpError(), "Exceed maximum number of topics in namespace.");
+ }
+
+ admin.namespaces().setMaxTopicsPerNamespace(namespace, 0);
+ // set namespace policy to no limit
+ for (int i = 0; i< 10; ++i) {
+ admin.topics().createPartitionedTopic(topic + "_v" + i, 2);
+ admin.topics().createNonPartitionedTopic(topic + "_vn" + i);
+ }
+
+
+ // check producer/consumer auto create partitioned topic
+ super.internalCleanup();
+ conf.setMaxTopicsPerNamespace(0);
+ conf.setDefaultNumPartitions(3);
+ conf.setAllowAutoTopicCreationType("partitioned");
+ super.internalSetup();
+
+ admin.clusters().createCluster("use", new ClusterData(brokerUrl.toString()));
+ admin.tenants().createTenant("testTenant", tenantInfo);
+ admin.namespaces().createNamespace(namespace, Sets.newHashSet("use"));
+ admin.namespaces().setMaxTopicsPerNamespace(namespace, 10);
+
+ pulsarClient.newProducer().topic(topic + "1").create();
Review comment:
OK, i will close them.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
315157973 commented on pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042#issuecomment-753251691
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 removed a comment on pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
315157973 removed a comment on pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042#issuecomment-753251691
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on a change in pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on a change in pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042#discussion_r548339245
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
##########
@@ -3182,5 +3182,26 @@ protected OffloadPolicies internalGetOffloadPolicies() {
return policies.offload_policies;
}
- private static final Logger log = LoggerFactory.getLogger(NamespacesBase.class);
+ protected int internalGetMaxTopicsPerNamespace() {
+ validateNamespacePolicyOperation(namespaceName, PolicyName.MAX_TOPICS, PolicyOperation.READ);
+ return getNamespacePolicies(namespaceName).max_topics_per_namespace;
+ }
+
+ protected void internalRemoveMaxTopicsPerNamespace() {
+ validateNamespacePolicyOperation(namespaceName, PolicyName.MAX_TOPICS, PolicyOperation.WRITE);
+ internalSetMaxTopicsPerNamespace(null);
+ }
+
+ protected void internalSetMaxTopicsPerNamespace(Integer maxTopicsPerNamespace) {
+ validateNamespacePolicyOperation(namespaceName, PolicyName.MAX_TOPICS, PolicyOperation.WRITE);
+ validatePoliciesReadOnlyAccess();
+
+ if (maxTopicsPerNamespace != null && maxTopicsPerNamespace < 0) {
Review comment:
The `maxTopicsPerNamespace` is passed by user, we should forbid user set the value < 0, so the check must be added.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on a change in pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
315157973 commented on a change in pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042#discussion_r548359438
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BrokerService.java
##########
@@ -2517,9 +2517,18 @@ private void checkTopicLevelPolicyEnable() {
private <T> boolean checkMaxTopicsPerNamespace(TopicName topicName, int numPartitions,
CompletableFuture<T> topicFuture) {
- final int maxTopicsPerNamespace = pulsar().getConfig().getMaxTopicsPerNamespace();
- if (maxTopicsPerNamespace > 0) {
- try {
+ Integer maxTopicsPerNamespace;
+ try {
+ maxTopicsPerNamespace = pulsar.getConfigurationCache().policiesCache()
+ .get(AdminResource.path(POLICIES, topicName.getNamespace()))
+ .map(p -> p.max_topics_per_namespace)
+ .orElse(null);
+
+ if (maxTopicsPerNamespace == null || maxTopicsPerNamespace < 0) {
Review comment:
Will there be no <0？
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] 315157973 commented on a change in pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
315157973 commented on a change in pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042#discussion_r547976954
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/impl/NamespacesBase.java
##########
@@ -3182,5 +3182,26 @@ protected OffloadPolicies internalGetOffloadPolicies() {
return policies.offload_policies;
}
- private static final Logger log = LoggerFactory.getLogger(NamespacesBase.class);
+ protected int internalGetMaxTopicsPerNamespace() {
+ validateNamespacePolicyOperation(namespaceName, PolicyName.MAX_TOPICS, PolicyOperation.READ);
+ return getNamespacePolicies(namespaceName).max_topics_per_namespace;
+ }
+
+ protected void internalRemoveMaxTopicsPerNamespace() {
+ validateNamespacePolicyOperation(namespaceName, PolicyName.MAX_TOPICS, PolicyOperation.WRITE);
+ internalSetMaxTopicsPerNamespace(null);
+ }
+
+ protected void internalSetMaxTopicsPerNamespace(Integer maxTopicsPerNamespace) {
+ validateNamespacePolicyOperation(namespaceName, PolicyName.MAX_TOPICS, PolicyOperation.WRITE);
+ validatePoliciesReadOnlyAccess();
+
+ if (maxTopicsPerNamespace != null && maxTopicsPerNamespace < 0) {
Review comment:
When setting the value, it must be greater than or equal to 0, so there should be no case of <0.
##########
File path: pulsar-broker/src/main/java/org/apache/pulsar/broker/admin/AdminResource.java
##########
@@ -811,9 +811,14 @@ protected boolean isNamespaceReplicated(NamespaceName namespaceName) {
}
protected void internalCreatePartitionedTopic(AsyncResponse asyncResponse, int numPartitions) {
- final int maxTopicsPerNamespace = pulsar().getConfig().getMaxTopicsPerNamespace();
- if (maxTopicsPerNamespace > 0) {
- try {
+ Integer maxTopicsPerNamespace;
+ try {
+ maxTopicsPerNamespace = getNamespacePolicies(namespaceName).max_topics_per_namespace;
+ if (maxTopicsPerNamespace == null || maxTopicsPerNamespace < 0) {
Review comment:
When setting the value, it must be greater than or equal to 0, so there should be no case of <0.
##########
File path: pulsar-broker/src/test/java/org/apache/pulsar/broker/admin/NamespacesTest.java
##########
@@ -1413,6 +1414,132 @@ public void testRetentionPolicyValidation() throws Exception {
admin.namespaces().deleteNamespace(namespace);
}
+ @Test(timeOut = 30000)
+ public void testMaxTopicsPerNamespace() throws Exception {
+ super.internalCleanup();
+ conf.setMaxTopicsPerNamespace(15);
+ super.internalSetup();
+
+ String namespace = "testTenant/ns1";
+ admin.clusters().createCluster("use", new ClusterData(brokerUrl.toString()));
+ TenantInfo tenantInfo = new TenantInfo(Sets.newHashSet("role1", "role2"),
+ Sets.newHashSet("use"));
+ admin.tenants().createTenant("testTenant", tenantInfo);
+ admin.namespaces().createNamespace(namespace, Sets.newHashSet("use"));
+
+ admin.namespaces().setMaxTopicsPerNamespace(namespace, 10);
+ assertEquals(10, admin.namespaces().getMaxTopicsPerNamespace(namespace));
+
+ // check create partitioned/non-partitioned topics using namespace policy
+ String topic = "persistent://testTenant/ns1/test_create_topic_v";
+ admin.topics().createPartitionedTopic(topic + "1", 2);
+ admin.topics().createPartitionedTopic(topic + "2", 3);
+ admin.topics().createPartitionedTopic(topic + "3", 4);
+ admin.topics().createNonPartitionedTopic(topic + "4");
+
+ try {
+ admin.topics().createPartitionedTopic(topic + "5", 2);
+ fail();
+ } catch (PulsarAdminException e) {
+ assertEquals(e.getStatusCode(), 412);
+ assertEquals(e.getHttpError(), "Exceed maximum number of topics in namespace.");
+ }
+
+ // remove namespace policy limit, use broker configuration instead.
+ admin.namespaces().removeMaxTopicsPerNamespace(namespace);
+ admin.topics().createPartitionedTopic(topic + "6", 4);
+ try {
+ admin.topics().createPartitionedTopic(topic + "7", 3);
+ fail();
+ } catch (PulsarAdminException e) {
+ assertEquals(e.getStatusCode(), 412);
+ assertEquals(e.getHttpError(), "Exceed maximum number of topics in namespace.");
+ }
+
+ admin.namespaces().setMaxTopicsPerNamespace(namespace, 0);
+ // set namespace policy to no limit
+ for (int i = 0; i< 10; ++i) {
+ admin.topics().createPartitionedTopic(topic + "_v" + i, 2);
+ admin.topics().createNonPartitionedTopic(topic + "_vn" + i);
+ }
+
+
+ // check producer/consumer auto create partitioned topic
+ super.internalCleanup();
+ conf.setMaxTopicsPerNamespace(0);
+ conf.setDefaultNumPartitions(3);
+ conf.setAllowAutoTopicCreationType("partitioned");
+ super.internalSetup();
+
+ admin.clusters().createCluster("use", new ClusterData(brokerUrl.toString()));
+ admin.tenants().createTenant("testTenant", tenantInfo);
+ admin.namespaces().createNamespace(namespace, Sets.newHashSet("use"));
+ admin.namespaces().setMaxTopicsPerNamespace(namespace, 10);
+
+ pulsarClient.newProducer().topic(topic + "1").create();
Review comment:
Will there be a problem if the created producer is not closed?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hangc0276 commented on pull request #9042: support namespace policy for maxTopicsPerNamespace limit
Posted by GitBox <gi...@apache.org>.
hangc0276 commented on pull request #9042:
URL: https://github.com/apache/pulsar/pull/9042#issuecomment-751139035
/pulsarbot run-failure-checks
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org