You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2023/01/13 20:29:12 UTC
[GitHub] [kafka] rondagostino opened a new pull request, #13116: KAFKA-14351: Controller Mutation Quota for KRaft
rondagostino opened a new pull request, #13116:
URL: https://github.com/apache/kafka/pull/13116
Adds hooks to check controller mutation quotas in KRaft.
It won't actually work in practice until controllers support dynamic reconfiguration (https://issues.apache.org/jira/browse/KAFKA-14350), but this adds the appropriate logic for when that happens.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1127191300
##########
core/src/test/scala/unit/kafka/utils/TestUtils.scala:
##########
@@ -378,6 +378,118 @@ object TestUtils extends Logging {
props
}
+ def createCombinedControllerConfig(nodeId: Int,
Review Comment:
Can't we just add a boolean to `TestUtils.createBrokerConfig`? Two whole new functions seems like overkill.
Also, while you're changing this code, it would be good to use the standard indentation style. Having the parameter list be indented halfway through the page really hurts the eyes. Thsee days we do:
```
def myVeryLongFunctionName(
foo: Foo,
bar: Bar,
...
quux: Quux
): ReturnValueType = {
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1127186242
##########
core/src/test/scala/unit/kafka/server/DynamicBrokerConfigTest.scala:
##########
@@ -479,6 +489,42 @@ class DynamicBrokerConfigTest {
assertEquals("User:admin", authorizer.superUsers)
}
+ @Test
+ def testCombinedControllerAuthorizerConfig(): Unit = {
+ val props = TestUtils.createCombinedControllerConfig(0, port = 9092)
+ val oldConfig = KafkaConfig.fromProps(props)
+ oldConfig.dynamicConfig.initialize(None)
+
+ val controllerServer: ControllerServer = mock(classOf[kafka.server.ControllerServer])
+
+ val authorizer = new TestAuthorizer
+ when(controllerServer.config).thenReturn(oldConfig)
+ when(controllerServer.authorizer).thenReturn(Some(authorizer))
+ // We are only testing authorizer reconfiguration, ignore any exceptions due to incomplete mock
+ assertThrows(classOf[Throwable], () => controllerServer.config.dynamicConfig.addReconfigurables(controllerServer))
Review Comment:
I sympathize with not wanting to debug the mocking mess here. There are just so many functions that get called, and having to do thenReturn for all of them is a huge pain. Still, just putting an `assertThrows(classOf[Throwable], ...)` seems like it could hide errors over time.
Should we just add the authorizer as a reconfigurable directly? Or is that too hard to do here?
##########
core/src/test/scala/unit/kafka/server/DynamicBrokerConfigTest.scala:
##########
@@ -479,6 +489,42 @@ class DynamicBrokerConfigTest {
assertEquals("User:admin", authorizer.superUsers)
}
+ @Test
+ def testCombinedControllerAuthorizerConfig(): Unit = {
+ val props = TestUtils.createCombinedControllerConfig(0, port = 9092)
+ val oldConfig = KafkaConfig.fromProps(props)
+ oldConfig.dynamicConfig.initialize(None)
+
+ val controllerServer: ControllerServer = mock(classOf[kafka.server.ControllerServer])
+
+ val authorizer = new TestAuthorizer
+ when(controllerServer.config).thenReturn(oldConfig)
+ when(controllerServer.authorizer).thenReturn(Some(authorizer))
+ // We are only testing authorizer reconfiguration, ignore any exceptions due to incomplete mock
+ assertThrows(classOf[Throwable], () => controllerServer.config.dynamicConfig.addReconfigurables(controllerServer))
+ props.put("super.users", "User:admin")
+ controllerServer.config.dynamicConfig.updateBrokerConfig(0, props)
+ assertEquals("User:admin", authorizer.superUsers)
+ }
+
+ @Test
+ def testIsolatedControllerAuthorizerConfig(): Unit = {
+ val props = TestUtils.createIsolatedControllerConfig(0, port = 9092)
+ val oldConfig = KafkaConfig.fromProps(props)
+ oldConfig.dynamicConfig.initialize(None)
+
+ val controllerServer: ControllerServer = mock(classOf[kafka.server.ControllerServer])
+
+ val authorizer = new TestAuthorizer
+ when(controllerServer.config).thenReturn(oldConfig)
+ when(controllerServer.authorizer).thenReturn(Some(authorizer))
+ // We are only testing authorizer reconfiguration, ignore any exceptions due to incomplete mock
+ assertThrows(classOf[Throwable], () => controllerServer.config.dynamicConfig.addReconfigurables(controllerServer))
Review Comment:
Same question as above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1127213105
##########
core/src/test/scala/unit/kafka/utils/TestUtils.scala:
##########
@@ -378,6 +378,118 @@ object TestUtils extends Logging {
props
}
+ def createCombinedControllerConfig(nodeId: Int,
Review Comment:
`createBrokerConfig` has a ton of parameters, I don't think we should be adding more parameters to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "rondagostino (via GitHub)" <gi...@apache.org>.
rondagostino commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1128069250
##########
core/src/test/scala/unit/kafka/server/DynamicBrokerConfigTest.scala:
##########
@@ -479,6 +489,42 @@ class DynamicBrokerConfigTest {
assertEquals("User:admin", authorizer.superUsers)
}
+ @Test
+ def testCombinedControllerAuthorizerConfig(): Unit = {
+ val props = TestUtils.createCombinedControllerConfig(0, port = 9092)
+ val oldConfig = KafkaConfig.fromProps(props)
+ oldConfig.dynamicConfig.initialize(None)
+
+ val controllerServer: ControllerServer = mock(classOf[kafka.server.ControllerServer])
+
+ val authorizer = new TestAuthorizer
+ when(controllerServer.config).thenReturn(oldConfig)
+ when(controllerServer.authorizer).thenReturn(Some(authorizer))
+ // We are only testing authorizer reconfiguration, ignore any exceptions due to incomplete mock
+ assertThrows(classOf[Throwable], () => controllerServer.config.dynamicConfig.addReconfigurables(controllerServer))
Review Comment:
I fixed it.
##########
core/src/test/scala/unit/kafka/server/DynamicBrokerConfigTest.scala:
##########
@@ -479,6 +489,42 @@ class DynamicBrokerConfigTest {
assertEquals("User:admin", authorizer.superUsers)
}
+ @Test
+ def testCombinedControllerAuthorizerConfig(): Unit = {
+ val props = TestUtils.createCombinedControllerConfig(0, port = 9092)
+ val oldConfig = KafkaConfig.fromProps(props)
+ oldConfig.dynamicConfig.initialize(None)
+
+ val controllerServer: ControllerServer = mock(classOf[kafka.server.ControllerServer])
+
+ val authorizer = new TestAuthorizer
+ when(controllerServer.config).thenReturn(oldConfig)
+ when(controllerServer.authorizer).thenReturn(Some(authorizer))
+ // We are only testing authorizer reconfiguration, ignore any exceptions due to incomplete mock
+ assertThrows(classOf[Throwable], () => controllerServer.config.dynamicConfig.addReconfigurables(controllerServer))
+ props.put("super.users", "User:admin")
+ controllerServer.config.dynamicConfig.updateBrokerConfig(0, props)
+ assertEquals("User:admin", authorizer.superUsers)
+ }
+
+ @Test
+ def testIsolatedControllerAuthorizerConfig(): Unit = {
+ val props = TestUtils.createIsolatedControllerConfig(0, port = 9092)
+ val oldConfig = KafkaConfig.fromProps(props)
+ oldConfig.dynamicConfig.initialize(None)
+
+ val controllerServer: ControllerServer = mock(classOf[kafka.server.ControllerServer])
+
+ val authorizer = new TestAuthorizer
+ when(controllerServer.config).thenReturn(oldConfig)
+ when(controllerServer.authorizer).thenReturn(Some(authorizer))
+ // We are only testing authorizer reconfiguration, ignore any exceptions due to incomplete mock
+ assertThrows(classOf[Throwable], () => controllerServer.config.dynamicConfig.addReconfigurables(controllerServer))
Review Comment:
Same here -- I fixed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on PR #13116:
URL: https://github.com/apache/kafka/pull/13116#issuecomment-1458687762
Yes. It is frustrating that we have so many flaky tests. @mumrah is fixing the `KafkaServerKRaftRegistrationTest`. We may need to fix `SocketServerTest` too in the near future since it seems to flake a lot. I agree that the test failures are unrelated. Will commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on PR #13116:
URL: https://github.com/apache/kafka/pull/13116#issuecomment-1400816232
> Also note that I didn't implement the full logic -- QuorumController has additional checks that could in fact cause a specific topic request to be rejected (e.g. explicit assignments not contiguous partition numbers starting with 0). This could lead to a calculation of partition counts for throttling purposes that exceeds the actual partition counts requested -- which could cause throttling in cases where technically it should not.
I think arguably if you are making requests that fail, your quota should be dinged for those requests. Surely we don't want to let the guy spinning in a tight loop making tons of bad requests go un-throttled? I would even maybe include requests that fail authorization in the quota, although I know the ZK implementation might not. Authorization is expensive; you should get charged for wasting the authorizer's time with bad rqeuests.
Maybe we should charge for validateOnly requests as well. They are almost as expensive as non-validateOnly requests... the only difference is no records written.
> The benefit of the Scala approach is that we avoid doing work in the controller itself when throttling does occur. We don't generate all of the Metadata Log records, for example. Note that createPartitions and delateTopics will have to get access to the partition counts, which currently I think is not available.
One big issue is that ControllerMutationQuota takes some locks, and I am worried that the controller thread could get blocked behind those locks. This is similar to the Authorizer where for better or worse, we went with a blocking implementation that would be too slow to call from the quorum controller thread (for example, the Authorizer implementation might send out a blocking request to ZK or other external system, etc.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1127176224
##########
core/src/main/scala/kafka/server/DynamicBrokerConfig.scala:
##########
@@ -730,6 +744,45 @@ class DynamicThreadPool(server: KafkaBroker) extends BrokerReconfigurable {
}
}
+ def getValue(config: KafkaConfig, name: String): Int = {
+ name match {
+ case KafkaConfig.NumIoThreadsProp => config.numIoThreads
+ case KafkaConfig.NumReplicaFetchersProp => config.numReplicaFetchers
+ case KafkaConfig.NumRecoveryThreadsPerDataDirProp => config.numRecoveryThreadsPerDataDir
+ case KafkaConfig.BackgroundThreadsProp => config.backgroundThreads
+ case n => throw new IllegalStateException(s"Unexpected config $n")
+ }
+ }
+}
+
+class ControllerDynamicThreadPool(controller: ControllerServer) extends BrokerReconfigurable {
+
+ override def reconfigurableConfigs: Set[String] = {
+ DynamicThreadPool.ReconfigurableConfigs // common configs
Review Comment:
This is a small thing but, we probably shouldn't return `num.replica.fetchers`, `num.recovery.threads.per.data.dir`, and `background.threads` here, since they're not actually reconfigurable on the controller.
We could still share most of the code here if we pass in the `ReconfigurableConfigs` `Set[String]` to `DynamicThreadPool.validateReconfiguration`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1127179616
##########
core/src/test/java/kafka/test/MockController.java:
##########
@@ -181,10 +194,16 @@ public CompletableFuture<Void> unregisterBroker(
static class MockTopic {
private final String name;
private final Uuid id;
+ private final int numPartitions;
MockTopic(String name, Uuid id) {
+ this(name, id, 10); // hard-code the number of partitions if left unspecified
Review Comment:
I guess this is a nitpick but `num.partitions` defaults to 1, so having a default of 10 partitions is a bit surprising here. Is there a reason to use 10 instead of 1?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1127197623
##########
metadata/src/main/java/org/apache/kafka/controller/ControllerRequestContext.java:
##########
@@ -42,19 +45,39 @@ public static OptionalLong requestTimeoutMsToDeadlineNs(
private final OptionalLong deadlineNs;
private final RequestHeaderData requestHeader;
+ private final Optional<Consumer<Double>> requestedPartitionCountRecorder;
+
Review Comment:
Yes, this is messy. I have a lot of ideas for improving this, but most of them would require substantial refactoring.
For now, let's just get rid of the obvious bad stuff.
* Using a double for number of partitions is a bit silly
* "Recorder" is also a bit of a misnomer -- we are doing more than recording the number of partitions.
* The Optional isn't needed -- just pass a Consumer that does nothing if you don't want to do anything.
So maybe change it to something like `Consumer<Integer> applyPartitionChangeQuota` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1127191300
##########
core/src/test/scala/unit/kafka/utils/TestUtils.scala:
##########
@@ -378,6 +378,118 @@ object TestUtils extends Logging {
props
}
+ def createCombinedControllerConfig(nodeId: Int,
Review Comment:
Can't we just add a boolean to `TestUtils.createBrokerConfig`? Two whole new functions seems like overkill.
Also, while you're changing this code, it would be good to use the standard indentation style. Having this be indented halfway through the page really hurts the eyes. Thsee days we do:
```
def myVeryLongFunctionName(
foo: Foo,
bar: Bar,
...
quux: Quux
): ReturnValueType = {
...
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe merged pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe merged PR #13116:
URL: https://github.com/apache/kafka/pull/13116
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by GitBox <gi...@apache.org>.
rondagostino commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1070162021
##########
core/src/test/scala/unit/kafka/server/ControllerApisTest.scala:
##########
@@ -523,8 +523,23 @@ class ControllerApisTest {
new ListPartitionReassignmentsRequestData()).build())))
}
- @Test
- def testCreateTopics(): Unit = {
+ object AlwaysExceededControllerMutationQuota extends ControllerMutationQuota {
+ override def isExceeded: Boolean = true
+ override def record(permits: Double): Unit = throw new ThrottlingQuotaExceededException(throttleTime, "quota exceeded in test")
+ override def throttleTime: Int = 1000
+ }
+
+ @ParameterizedTest(name = "testCreateTopics with mutationQuotaExceeded: {0}, validateOnly: {1}")
+ @CsvSource(value = Array(
+ "no,no",
+ "no,yes",
+ "yes,no",
+ "yes,yes"
+ ))
Review Comment:
Instead of `yes/no` for the quota maybe this should be an int so that we can test that we are actually sending the correct partition count. We are requesting 6 potentially-valid partitions, so it might be good to ensure that a limit of 6 is throttled but a limit of 7 is not.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1127176972
##########
core/src/main/scala/kafka/server/DynamicBrokerConfig.scala:
##########
@@ -911,7 +954,22 @@ object DynamicListenerConfig {
)
}
-class DynamicClientQuotaCallback(server: KafkaBroker) extends Reconfigurable {
+trait DynamicClientQuotaCallbackServer {
Review Comment:
We don't really need a trait for this, do we? It's just two objects that we can pass to the constructor of `DynamicClientQuotaCallback`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1127191979
##########
core/src/test/scala/unit/kafka/utils/TestUtils.scala:
##########
@@ -378,6 +378,118 @@ object TestUtils extends Logging {
props
}
+ def createCombinedControllerConfig(nodeId: Int,
+ enableControlledShutdown: Boolean = true,
+ enableDeleteTopic: Boolean = true,
+ port: Int = RandomPort,
+ interBrokerSecurityProtocol: Option[SecurityProtocol] = None,
+ trustStoreFile: Option[File] = None,
+ saslProperties: Option[Properties] = None,
+ enablePlaintext: Boolean = true,
+ enableSaslPlaintext: Boolean = false,
+ saslPlaintextPort: Int = RandomPort,
+ enableSsl: Boolean = false,
+ sslPort: Int = RandomPort,
+ enableSaslSsl: Boolean = false,
+ saslSslPort: Int = RandomPort,
+ rack: Option[String] = None,
+ logDirCount: Int = 1,
+ enableToken: Boolean = false,
+ numPartitions: Int = 1,
+ defaultReplicationFactor: Short = 1,
+ enableFetchFromFollower: Boolean = false): Properties = {
+ val retval = createBrokerConfig(nodeId,
+ zkConnect = null,
+ enableControlledShutdown,
+ enableDeleteTopic,
+ port,
+ interBrokerSecurityProtocol,
+ trustStoreFile,
+ saslProperties,
+ enablePlaintext,
+ enableSaslPlaintext,
+ saslPlaintextPort,
+ enableSsl,
+ sslPort,
+ enableSaslSsl,
+ saslSslPort,
+ rack,
+ logDirCount,
+ enableToken,
+ numPartitions,
+ defaultReplicationFactor,
+ enableFetchFromFollower,
+ )
+ retval.put(KafkaConfig.ProcessRolesProp, "broker,controller")
+ retval.put(KafkaConfig.ControllerListenerNamesProp, "CONTROLLER")
+ retval.put(KafkaConfig.ListenersProp, s"${retval.get(KafkaConfig.ListenersProp)},CONTROLLER://localhost:0")
+ retval.put(KafkaConfig.QuorumVotersProp, s"${nodeId}@localhost:0")
+ retval
+ }
+
+ def createIsolatedControllerConfig(nodeId: Int,
Review Comment:
Same question as above, why not just a boolean on `createBrokerConfig`?
`combinedMode: Boolean = false` or similar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1127176972
##########
core/src/main/scala/kafka/server/DynamicBrokerConfig.scala:
##########
@@ -911,7 +954,22 @@ object DynamicListenerConfig {
)
}
-class DynamicClientQuotaCallback(server: KafkaBroker) extends Reconfigurable {
+trait DynamicClientQuotaCallbackServer {
Review Comment:
We don't really need a trait for this, do we? It's just two objects that we can pass directly to the constructor of `DynamicClientQuotaCallback`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by GitBox <gi...@apache.org>.
rondagostino commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1070162021
##########
core/src/test/scala/unit/kafka/server/ControllerApisTest.scala:
##########
@@ -523,8 +523,23 @@ class ControllerApisTest {
new ListPartitionReassignmentsRequestData()).build())))
}
- @Test
- def testCreateTopics(): Unit = {
+ object AlwaysExceededControllerMutationQuota extends ControllerMutationQuota {
+ override def isExceeded: Boolean = true
+ override def record(permits: Double): Unit = throw new ThrottlingQuotaExceededException(throttleTime, "quota exceeded in test")
+ override def throttleTime: Int = 1000
+ }
+
+ @ParameterizedTest(name = "testCreateTopics with mutationQuotaExceeded: {0}, validateOnly: {1}")
+ @CsvSource(value = Array(
+ "no,no",
+ "no,yes",
+ "yes,no",
+ "yes,yes"
+ ))
Review Comment:
Instead of `yes/no` for the quota maybe this should be an int so that we can test that we are actually sending the correct partition count. We are requesting 6 potentially-valid partitions, so it might be good to ensure that 5 is not throttled and 6 is throttle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "rondagostino (via GitHub)" <gi...@apache.org>.
rondagostino commented on PR #13116:
URL: https://github.com/apache/kafka/pull/13116#issuecomment-1455123012
```
================================================================================
SESSION REPORT (ALL TESTS)
ducktape version: 0.11.1
session_id: 2023-03-05--001
run time: 1 minute 33.639 seconds
tests run: 3
passed: 3
flaky: 0
failed: 0
ignored: 0
================================================================================
test_id: kafkatest.tests.core.controller_mutation_quota_test.ControllerMutationQuotaTest.test_controller_mutation_quota.metadata_quorum=COLOCATED_KRAFT
status: PASS
run time: 26.262 seconds
--------------------------------------------------------------------------------
test_id: kafkatest.tests.core.controller_mutation_quota_test.ControllerMutationQuotaTest.test_controller_mutation_quota.metadata_quorum=REMOTE_KRAFT
status: PASS
run time: 30.448 seconds
--------------------------------------------------------------------------------
test_id: kafkatest.tests.core.controller_mutation_quota_test.ControllerMutationQuotaTest.test_controller_mutation_quota.metadata_quorum=ZK
status: PASS
run time: 36.590 seconds
--------------------------------------------------------------------------------
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ijuma commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "ijuma (via GitHub)" <gi...@apache.org>.
ijuma commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1125408496
##########
metadata/src/main/java/org/apache/kafka/controller/ControllerRequestContext.java:
##########
@@ -42,19 +45,39 @@ public static OptionalLong requestTimeoutMsToDeadlineNs(
private final OptionalLong deadlineNs;
private final RequestHeaderData requestHeader;
+ private final Optional<Consumer<Double>> requestedPartitionCountRecorder;
+
Review Comment:
We could have an interface in server-common that the scala classes implement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by GitBox <gi...@apache.org>.
rondagostino commented on PR #13116:
URL: https://github.com/apache/kafka/pull/13116#issuecomment-1382381390
Only implemented/tested the create-topics case so far.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1084419497
##########
core/src/main/scala/kafka/server/ControllerApis.scala:
##########
@@ -392,14 +394,36 @@ class ControllerApis(val requestChannel: RequestChannel,
val describableTopicNames = getDescribableTopics.apply(allowedTopicNames).asJava
val effectiveRequest = request.duplicate()
val iterator = effectiveRequest.topics().iterator()
+ var totalRequestedPartitionCount = 0
+ val defaultPartitionCont = config.numPartitions.intValue()
while (iterator.hasNext) {
val creatableTopic = iterator.next()
if (duplicateTopicNames.contains(creatableTopic.name()) ||
!authorizedTopicNames.contains(creatableTopic.name())) {
iterator.remove()
- }
- }
- controller.createTopics(context, effectiveRequest, describableTopicNames).thenApply { response =>
+ } else {
+ if (!creatableTopic.assignments().isEmpty) {
+ totalRequestedPartitionCount += creatableTopic.assignments().size()
+ } else if (creatableTopic.numPartitions() > 0)
+ totalRequestedPartitionCount += creatableTopic.numPartitions()
+ else
+ totalRequestedPartitionCount += defaultPartitionCont
+ }
+ }
+ val future = try {
+ if (!effectiveRequest.validateOnly())
+ controllerMutationQuota.record(totalRequestedPartitionCount)
+ controller.createTopics(context, effectiveRequest, describableTopicNames)
+ } catch {
+ case e: ThrottlingQuotaExceededException =>
+ val apiError = ApiError.fromThrowable(e)
+ val data = new CreateTopicsResponseData
+ effectiveRequest.topics().forEach(topic =>
+ data.topics.add(new CreateTopicsResponseData.CreatableTopicResult().setName(topic.name).setErrorCode(apiError.error.code).setErrorMessage(apiError.message)))
+ data.setThrottleTimeMs(e.throttleTimeMs())
+ CompletableFuture.completedFuture(data)
+ }
Review Comment:
I don't think you need to catch this here, since we have a catch block on the request handler that will deal with "send back error X to response Y"
If you do have a catch block here you'd certainly need a return statement as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by GitBox <gi...@apache.org>.
rondagostino commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1070000035
##########
docs/ops.html:
##########
@@ -1197,7 +1197,7 @@ <h4 class="anchor-heading"><a id="multitenancy-isolation" class="anchor-link"></
</p>
<p>
- <strong>Client quotas:</strong> Kafka supports different types of (per-user principal) client quotas. Because a client's quotas apply irrespective of which topics the client is writing to or reading from, they are a convenient and effective tool to allocate resources in a multi-tenant cluster. <a href="#design_quotascpu">Request rate quotas</a>, for example, help to limit a user's impact on broker CPU usage by limiting the time a broker spends on the <a href="/protocol.html">request handling path</a> for that user, after which throttling kicks in. In many situations, isolating users with request rate quotas has a bigger impact in multi-tenant clusters than setting incoming/outgoing network bandwidth quotas, because excessive broker CPU usage for processing requests reduces the effective bandwidth the broker can serve. Furthermore, administrators can also define quotas on topic operations—such as create, delete, and alter—to prevent Kafka clusters from being overwhelmed by high
ly concurrent topic operations (see <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-599%3A+Throttle+Create+Topic%2C+Create+Partition+and+Delete+Topic+Operations">KIP-599</a> and the quota type <code>controller_mutations_rate</code>).
+ <strong>Client quotas:</strong> Kafka supports different types of (per-user principal) client quotas. Because a client's quotas apply irrespective of which topics the client is writing to or reading from, they are a convenient and effective tool to allocate resources in a multi-tenant cluster. <a href="#design_quotascpu">Request rate quotas</a>, for example, help to limit a user's impact on broker CPU usage by limiting the time a broker spends on the <a href="/protocol.html">request handling path</a> for that user, after which throttling kicks in. In many situations, isolating users with request rate quotas has a bigger impact in multi-tenant clusters than setting incoming/outgoing network bandwidth quotas, because excessive broker CPU usage for processing requests reduces the effective bandwidth the broker can serve. Furthermore, administrators can also define quotas on topic operations—such as create, delete, and alter—to prevent Kafka clusters from being overwhelmed by high
ly concurrent topic operations (see <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-599%3A+Throttle+Create+Topic%2C+Create+Partition+and+Delete+Topic+Operations">KIP-599</a> and the quota type <code>controller_mutation_rate</code>).
Review Comment:
Hard to see the change, which is on the last line: changes `controller_mutations_rate` to `controller_mutation_rate` since the latter is what was implemented despite what the KIP says.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by GitBox <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1070071187
##########
docs/ops.html:
##########
@@ -1197,7 +1197,7 @@ <h4 class="anchor-heading"><a id="multitenancy-isolation" class="anchor-link"></
</p>
<p>
- <strong>Client quotas:</strong> Kafka supports different types of (per-user principal) client quotas. Because a client's quotas apply irrespective of which topics the client is writing to or reading from, they are a convenient and effective tool to allocate resources in a multi-tenant cluster. <a href="#design_quotascpu">Request rate quotas</a>, for example, help to limit a user's impact on broker CPU usage by limiting the time a broker spends on the <a href="/protocol.html">request handling path</a> for that user, after which throttling kicks in. In many situations, isolating users with request rate quotas has a bigger impact in multi-tenant clusters than setting incoming/outgoing network bandwidth quotas, because excessive broker CPU usage for processing requests reduces the effective bandwidth the broker can serve. Furthermore, administrators can also define quotas on topic operations—such as create, delete, and alter—to prevent Kafka clusters from being overwhelmed by high
ly concurrent topic operations (see <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-599%3A+Throttle+Create+Topic%2C+Create+Partition+and+Delete+Topic+Operations">KIP-599</a> and the quota type <code>controller_mutations_rate</code>).
+ <strong>Client quotas:</strong> Kafka supports different types of (per-user principal) client quotas. Because a client's quotas apply irrespective of which topics the client is writing to or reading from, they are a convenient and effective tool to allocate resources in a multi-tenant cluster. <a href="#design_quotascpu">Request rate quotas</a>, for example, help to limit a user's impact on broker CPU usage by limiting the time a broker spends on the <a href="/protocol.html">request handling path</a> for that user, after which throttling kicks in. In many situations, isolating users with request rate quotas has a bigger impact in multi-tenant clusters than setting incoming/outgoing network bandwidth quotas, because excessive broker CPU usage for processing requests reduces the effective bandwidth the broker can serve. Furthermore, administrators can also define quotas on topic operations—such as create, delete, and alter—to prevent Kafka clusters from being overwhelmed by high
ly concurrent topic operations (see <a href="https://cwiki.apache.org/confluence/display/KAFKA/KIP-599%3A+Throttle+Create+Topic%2C+Create+Partition+and+Delete+Topic+Operations">KIP-599</a> and the quota type <code>controller_mutation_rate</code>).
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "rondagostino (via GitHub)" <gi...@apache.org>.
rondagostino commented on PR #13116:
URL: https://github.com/apache/kafka/pull/13116#issuecomment-1401088949
Thanks, Colin. That all makes sense. The KIP indicates that `validateOnly` is not throttled because it does not load the controller (https://cwiki.apache.org/confluence/display/KAFKA/KIP-599%3A+Throttle+Create+Topic%2C+Create+Partition+and+Delete+Topic+Operations#KIP599:ThrottleCreateTopic,CreatePartitionandDeleteTopicOperations-HandlingofValidateOnly), but that is not the case here as you point out. So I will proceed as follows:
1. Keep the logic inside `ControllerApis.scala` to avoid taking locks in the controller thread
2. Expose any necessary information required via the `org.apache.kafka.controller.Controller` interface.
It isn't clear to me that we should charge for unauthorized requests. First, this is not mentioned in the KIP, so it is a different case than the `validateOnly` situation (where the spirit of the KIP implies we should charge for `validateOnly` in KRaft). Also, I wonder if it is possible that charging for unauthorized requests could lead to indirectly exposing partition information that a client is unauthorized to know. For now I will leave this piece alone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "rondagostino (via GitHub)" <gi...@apache.org>.
rondagostino commented on PR #13116:
URL: https://github.com/apache/kafka/pull/13116#issuecomment-1458306158
There were 6 test failures across 3 separate builds. All tests pass locally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1127186242
##########
core/src/test/scala/unit/kafka/server/DynamicBrokerConfigTest.scala:
##########
@@ -479,6 +489,42 @@ class DynamicBrokerConfigTest {
assertEquals("User:admin", authorizer.superUsers)
}
+ @Test
+ def testCombinedControllerAuthorizerConfig(): Unit = {
+ val props = TestUtils.createCombinedControllerConfig(0, port = 9092)
+ val oldConfig = KafkaConfig.fromProps(props)
+ oldConfig.dynamicConfig.initialize(None)
+
+ val controllerServer: ControllerServer = mock(classOf[kafka.server.ControllerServer])
+
+ val authorizer = new TestAuthorizer
+ when(controllerServer.config).thenReturn(oldConfig)
+ when(controllerServer.authorizer).thenReturn(Some(authorizer))
+ // We are only testing authorizer reconfiguration, ignore any exceptions due to incomplete mock
+ assertThrows(classOf[Throwable], () => controllerServer.config.dynamicConfig.addReconfigurables(controllerServer))
Review Comment:
I sympathize with not wanting to debug the mocking mess here. There are just so many functions that get called, and having to do thenReturn for all of them is a huge pain. Still, this seems like it could hide errors over time.
Should we just add the authorizer as a reconfigurable directly? Or is that too hard to do here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1127182652
##########
core/src/main/scala/kafka/server/metadata/DynamicClientQuotaPublisher.scala:
##########
@@ -0,0 +1,82 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package kafka.server.metadata
+
+import kafka.server.KafkaConfig
+import kafka.utils.Logging
+import org.apache.kafka.image.loader.{LogDeltaManifest, SnapshotManifest}
+import org.apache.kafka.image.{MetadataDelta, MetadataImage}
+import org.apache.kafka.server.fault.FaultHandler
+
+
+class DynamicClientQuotaPublisher(
+ conf: KafkaConfig,
+ faultHandler: FaultHandler,
+ nodeType: String,
+ clientQuotaMetadataManager: ClientQuotaMetadataManager,
+) extends Logging with org.apache.kafka.image.publisher.MetadataPublisher {
Review Comment:
I didn't think about this previously, but it makes sense to include `controller` or `broker` in the name that is returned here. Good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1127176224
##########
core/src/main/scala/kafka/server/DynamicBrokerConfig.scala:
##########
@@ -730,6 +744,45 @@ class DynamicThreadPool(server: KafkaBroker) extends BrokerReconfigurable {
}
}
+ def getValue(config: KafkaConfig, name: String): Int = {
+ name match {
+ case KafkaConfig.NumIoThreadsProp => config.numIoThreads
+ case KafkaConfig.NumReplicaFetchersProp => config.numReplicaFetchers
+ case KafkaConfig.NumRecoveryThreadsPerDataDirProp => config.numRecoveryThreadsPerDataDir
+ case KafkaConfig.BackgroundThreadsProp => config.backgroundThreads
+ case n => throw new IllegalStateException(s"Unexpected config $n")
+ }
+ }
+}
+
+class ControllerDynamicThreadPool(controller: ControllerServer) extends BrokerReconfigurable {
+
+ override def reconfigurableConfigs: Set[String] = {
+ DynamicThreadPool.ReconfigurableConfigs // common configs
Review Comment:
We probably shouldn't return `num.replica.fetchers`, `num.recovery.threads.per.data.dir`, and `background.threads` here, since they're not actually reconfigurable on the controller.
I don't think we have to fix that now, but I will file a JIRA to clean this up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] dajac commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "dajac (via GitHub)" <gi...@apache.org>.
dajac commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1098623901
##########
core/src/main/scala/kafka/server/ControllerApis.scala:
##########
@@ -322,15 +325,37 @@ class ControllerApis(val requestChannel: RequestChannel,
}
}
// Finally, the idToName map contains all the topics that we are authorized to delete.
- // Perform the deletion and create responses for each one.
- controller.deleteTopics(context, idToName.keySet).thenApply { idToError =>
- idToError.forEach { (id, error) =>
- appendResponse(idToName.get(id), id, error)
+ // First check the controller mutation quota if necessary, and then
Review Comment:
The quota is basically a `Sensor` which would only be updated by the QuorumController thread. However, I suppose that the internal metric lock is also acquired when the metric is reported so the QuorumController would not be the only one trying to acquire it, I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by GitBox <gi...@apache.org>.
cmccabe commented on PR #13116:
URL: https://github.com/apache/kafka/pull/13116#issuecomment-1382406862
Thanks very much for this, @rondagostino !
Anyway what I’d like to do here is two things:
1. split all the spelling corrections and docs changes into a separate PR which we can do today. They look like no-brainers
2. second PR do everything in Scala, see if that is cleaner. I think it will be. It would be better not to involve the core controller in this. we should be able to see which partition(s) were successfully created from ControllerServer (check the controller response)
also we'll need a test. As you said, we need to get a bit of dynamic config code in first, but that is pretty small.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by GitBox <gi...@apache.org>.
rondagostino commented on PR #13116:
URL: https://github.com/apache/kafka/pull/13116#issuecomment-1382580751
Thanks, Colin. I moved the minor stuff into https://github.com/apache/kafka/pull/13118 and put the check into `ControllerApis`. The only difference with what you suggested is that I consider the partitions requested (except for the ones with obvious issues like lack of authorization or duplicate topic names) rather than the partitions that the controller creates. When a quota is exceeded the request is not accepted, so we can't let the controller mutate the partition count and then decide -- we have to decide before letting the controller do its thing.
Again, the create-topic only case is the only one I've done. I've added a test as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1091311692
##########
core/src/main/scala/kafka/server/ControllerApis.scala:
##########
@@ -322,15 +325,37 @@ class ControllerApis(val requestChannel: RequestChannel,
}
}
// Finally, the idToName map contains all the topics that we are authorized to delete.
- // Perform the deletion and create responses for each one.
- controller.deleteTopics(context, idToName.keySet).thenApply { idToError =>
- idToError.forEach { (id, error) =>
- appendResponse(idToName.get(id), id, error)
+ // First check the controller mutation quota if necessary, and then
Review Comment:
This is a lot more logic (and expense) than what I wanted, and unfortunately it also has a TOCTOU, because partitions may be added in between checking here and doing the deletion.
I guess I simply didn't forsee this, sorry.
I wonder if we could convert the throttle to be lockless so that we could use it in the QuorumController thread itself. Like I said earlier, the big concern is that blocking the controller thread is really bad and not something we want to do under really any condition. It seems like the throttle should be simple... just a number or something right? Or can we somehow guarantee that only the controller thread itself is taking that lock under ordinary conditions?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "rondagostino (via GitHub)" <gi...@apache.org>.
rondagostino commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1097911296
##########
core/src/main/scala/kafka/server/ControllerApis.scala:
##########
@@ -322,15 +325,37 @@ class ControllerApis(val requestChannel: RequestChannel,
}
}
// Finally, the idToName map contains all the topics that we are authorized to delete.
- // Perform the deletion and create responses for each one.
- controller.deleteTopics(context, idToName.keySet).thenApply { idToError =>
- idToError.forEach { (id, error) =>
- appendResponse(idToName.get(id), id, error)
+ // First check the controller mutation quota if necessary, and then
Review Comment:
I believe these locks are scoped by the controller mutation quota manager such that nobody else would contest for them aside from the single-threaded controller. @dajac might you be able to confirm this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cmccabe commented on pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "cmccabe (via GitHub)" <gi...@apache.org>.
cmccabe commented on PR #13116:
URL: https://github.com/apache/kafka/pull/13116#issuecomment-1457277048
Thanks for the PR, @rondagostino . You've been very patient here and I'm sorry that the review wasn't quicker. We did a major pivot from implementing the quota in `ControllerApis` like I originally wanted, to implementing it in `QuorumController` like you originally wanted. And I think the QC way is the way it has to be, for all the reasons discussed above. So you were right all along :)
I spent a few hours looking through this code today to see if there were any obvious ways to improve the performance characteristics. The main thing I worry about is the impact of locking. The locking in `StrictControllerMutationQuota.record`kj is certainly frustrating, since I could see it colliding with metrics collection. There are several small locks that might be contended (sensor locks, KafkaMetric locks). None of them seems to be held for a very long time, though. So overall I cannot find any easy way to avoid this worry. We will have to benchmark, I guess.
I left some minor comments that probably you can do in a day or two. I think we're pretty close now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "rondagostino (via GitHub)" <gi...@apache.org>.
rondagostino commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1088113461
##########
core/src/main/scala/kafka/server/ControllerApis.scala:
##########
@@ -392,14 +394,36 @@ class ControllerApis(val requestChannel: RequestChannel,
val describableTopicNames = getDescribableTopics.apply(allowedTopicNames).asJava
val effectiveRequest = request.duplicate()
val iterator = effectiveRequest.topics().iterator()
+ var totalRequestedPartitionCount = 0
+ val defaultPartitionCont = config.numPartitions.intValue()
while (iterator.hasNext) {
val creatableTopic = iterator.next()
if (duplicateTopicNames.contains(creatableTopic.name()) ||
!authorizedTopicNames.contains(creatableTopic.name())) {
iterator.remove()
- }
- }
- controller.createTopics(context, effectiveRequest, describableTopicNames).thenApply { response =>
+ } else {
+ if (!creatableTopic.assignments().isEmpty) {
+ totalRequestedPartitionCount += creatableTopic.assignments().size()
+ } else if (creatableTopic.numPartitions() > 0)
+ totalRequestedPartitionCount += creatableTopic.numPartitions()
+ else
+ totalRequestedPartitionCount += defaultPartitionCont
+ }
+ }
+ val future = try {
+ if (!effectiveRequest.validateOnly())
+ controllerMutationQuota.record(totalRequestedPartitionCount)
+ controller.createTopics(context, effectiveRequest, describableTopicNames)
+ } catch {
+ case e: ThrottlingQuotaExceededException =>
+ val apiError = ApiError.fromThrowable(e)
+ val data = new CreateTopicsResponseData
+ effectiveRequest.topics().forEach(topic =>
+ data.topics.add(new CreateTopicsResponseData.CreatableTopicResult().setName(topic.name).setErrorCode(apiError.error.code).setErrorMessage(apiError.message)))
+ data.setThrottleTimeMs(e.throttleTimeMs())
+ CompletableFuture.completedFuture(data)
+ }
Review Comment:
I actually believe the code is correct as written. We mark all of the requested topics in `effectiveRequest` with the throttle exception, and then we fallthrough to mark all of the topics that were not in the effective request with the appropriate error codes (duplicate topics get marked with invalid request and unauthorized topics get marked with unauthorized request). Clearly this is not clear though! I'll add a comment describing this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by "rondagostino (via GitHub)" <gi...@apache.org>.
rondagostino commented on PR #13116:
URL: https://github.com/apache/kafka/pull/13116#issuecomment-1458662433
There were several test failures (SelectorTest, Tls12SelectorTest, KafkaServerKRaftRegistrationTest, ListOffsetsRequestWithRemoteStoreTest, SocketServerTest) but I have seen them in the recent past sporadically, and they all passed locally. Some builds still running/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by GitBox <gi...@apache.org>.
rondagostino commented on PR #13116:
URL: https://github.com/apache/kafka/pull/13116#issuecomment-1383262505
@cmccabe I updated this to deal with create topic requests that either set an explicit assignment or that use the default partition count. Note that this is logic that exists within `QuorumController`, and to take it into account with the throttling logic being invoked in Scala means we have to have it in `ControllerApis`. Also note that I didn't implement the full logic -- `QuorumController` has additional checks that could in fact cause a specific topic request to be rejected (e.g. explicit assignments not contiguous partition numbers starting with 0). This could lead to a calculation of partition counts for throttling purposes that exceeds the actual partition counts requested -- which could cause throttling in cases where technically it should not. I think this would be a rare occurrence -- so a small detriment to the Scala approach.
The benefit of the Scala approach is that we avoid doing work in the controller itself when throttling does occur. We don't generate all of the Metadata Log records, for example. Note that createPartitions and delateTopics will have to get access to the partition counts, which currently I think is not available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] rondagostino commented on a diff in pull request #13116: KAFKA-14351: Controller Mutation Quota for KRaft
Posted by GitBox <gi...@apache.org>.
rondagostino commented on code in PR #13116:
URL: https://github.com/apache/kafka/pull/13116#discussion_r1070011382
##########
metadata/src/main/java/org/apache/kafka/controller/ControllerRequestContext.java:
##########
@@ -42,19 +45,39 @@ public static OptionalLong requestTimeoutMsToDeadlineNs(
private final OptionalLong deadlineNs;
private final RequestHeaderData requestHeader;
+ private final Optional<Consumer<Double>> requestedPartitionCountRecorder;
+
Review Comment:
We add this here because `class ControllerMutationQuotaManager` and `trait ControllerMutationQuota` are written in Scala while the KRaft controller and such are written in Java with no access to the Scala class/trait -- we can't pass in anything without wrapping it in Java somehow. All we need to do is record a request for some number of partitions, and this matches that requirement precisely.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org