You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2020/11/20 07:35:08 UTC

[GitHub] [kafka] splett2 opened a new pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

splett2 opened a new pull request #9628:
URL: https://github.com/apache/kafka/pull/9628


   This PR adds support for IP entities to the `DescribeClientQuotas` and `AlterClientQuotas` APIs. This change does not require any protocol version bumps. 
   
   This PR also adds support for describing/altering IP quotas via `kafka-configs` tooling.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r534494510



##########
File path: core/src/main/scala/kafka/server/AdminManager.scala
##########
@@ -920,32 +954,49 @@ class AdminManager(val config: KafkaConfig,
         !name.isDefined || !strict
     }
 
-    def fromProps(props: Map[String, String]): Map[String, Double] = {
-      props.map { case (key, value) =>
-        val doubleValue = try value.toDouble catch {
-          case _: NumberFormatException =>
-            throw new IllegalStateException(s"Unexpected client quota configuration value: $key -> $value")
-        }
-        key -> doubleValue
-      }
-    }
-
-    (userEntries ++ clientIdEntries ++ bothEntries).map { case ((u, c), p) =>
+    (userEntries ++ clientIdEntries ++ bothEntries).flatMap { case ((u, c), p) =>
       val quotaProps = p.asScala.filter { case (key, _) => QuotaConfigs.isQuotaConfig(key) }
       if (quotaProps.nonEmpty && matches(userComponent, u) && matches(clientIdComponent, c))
         Some(userClientIdToEntity(u, c) -> fromProps(quotaProps))
       else
         None
-    }.flatten.toMap
+    }.toMap

Review comment:
       this was just a coding style change, from calling `entries.map(...).flatten()` => `entries.flatMap`

##########
File path: core/src/main/scala/kafka/server/AdminManager.scala
##########
@@ -920,32 +954,49 @@ class AdminManager(val config: KafkaConfig,
         !name.isDefined || !strict
     }
 
-    def fromProps(props: Map[String, String]): Map[String, Double] = {
-      props.map { case (key, value) =>
-        val doubleValue = try value.toDouble catch {
-          case _: NumberFormatException =>
-            throw new IllegalStateException(s"Unexpected client quota configuration value: $key -> $value")
-        }
-        key -> doubleValue
-      }
-    }
-
-    (userEntries ++ clientIdEntries ++ bothEntries).map { case ((u, c), p) =>
+    (userEntries ++ clientIdEntries ++ bothEntries).flatMap { case ((u, c), p) =>
       val quotaProps = p.asScala.filter { case (key, _) => QuotaConfigs.isQuotaConfig(key) }
       if (quotaProps.nonEmpty && matches(userComponent, u) && matches(clientIdComponent, c))
         Some(userClientIdToEntity(u, c) -> fromProps(quotaProps))
       else
         None
-    }.flatten.toMap
+    }.toMap

Review comment:
       this was just a coding style change, from calling `entries.map(...).flatten()` => `entries.flatMap()`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r536257595



##########
File path: core/src/test/scala/unit/kafka/admin/ConfigCommandTest.scala
##########
@@ -472,6 +512,134 @@ class ConfigCommandTest extends ZooKeeperTestHarness with Logging {
     EasyMock.reset(alterResult, describeResult)
   }
 
+  @Test
+  def shouldNotAlterNonQuotaIpConfigsUsingBootstrapServer(): Unit = {
+    // when using --bootstrap-server, it should be illegal to alter anything that is not a connection quota
+    // for ip entities
+    val node = new Node(1, "localhost", 9092)
+    val mockAdminClient = new MockAdminClient(util.Collections.singletonList(node), node)
+
+    def verifyCommand(entityType: String, alterOpts: String*): Unit = {
+      val opts = new ConfigCommandOptions(Array("--bootstrap-server", "localhost:9092",
+        "--entity-type", entityType, "--entity-name", "admin",
+        "--alter") ++ alterOpts)
+      val e = intercept[IllegalArgumentException] {
+        ConfigCommand.alterConfig(mockAdminClient, opts)
+      }
+      assertTrue(s"Unexpected exception: $e", e.getMessage.contains("some_config"))
+    }
+
+    verifyCommand("ips", "--add-config", "connection_creation_rate=10000,some_config=10")
+    verifyCommand("ips", "--add-config", "some_config=10")
+    verifyCommand("ips", "--delete-config", "connection_creation_rate=10000,some_config=10")
+    verifyCommand("ips", "--delete-config", "some_config=10")
+  }
+
+  @Test
+  def testDescribeIpConfigs(): Unit = {
+    def testShouldDescribeIpConfig(ip: Option[String]): Unit = {
+      val entityType = ClientQuotaEntity.IP
+      val (ipArgs, filterComponent) = ip match {
+        case Some(null) => (Array("--entity-default"), ClientQuotaFilterComponent.ofDefaultEntity(entityType))
+        case Some(addr) => (Array("--entity-name", addr), ClientQuotaFilterComponent.ofEntity(entityType, addr))
+        case None => (Array.empty[String], ClientQuotaFilterComponent.ofEntityType(entityType))
+      }
+
+      val describeOpts = new ConfigCommandOptions(Array("--bootstrap-server", "localhost:9092",
+        "--describe", "--entity-type", "ips") ++ ipArgs)
+
+      val expectedFilter = ClientQuotaFilter.containsOnly(List(filterComponent).asJava)
+
+      var describedConfigs = false
+      val describeFuture = new KafkaFutureImpl[util.Map[ClientQuotaEntity, util.Map[String, java.lang.Double]]]
+      describeFuture.complete(Map.empty[ClientQuotaEntity, util.Map[String, java.lang.Double]].asJava)
+      val describeResult: DescribeClientQuotasResult = EasyMock.createNiceMock(classOf[DescribeClientQuotasResult])
+      EasyMock.expect(describeResult.entities()).andReturn(describeFuture)
+
+      val node = new Node(1, "localhost", 9092)
+      val mockAdminClient = new MockAdminClient(util.Collections.singletonList(node), node) {
+        override def describeClientQuotas(filter: ClientQuotaFilter, options: DescribeClientQuotasOptions): DescribeClientQuotasResult = {
+          assertTrue(filter.strict)
+          assertEquals(expectedFilter, filter)
+          describedConfigs = true
+          describeResult
+        }
+      }

Review comment:
       hmm, good point. I did this because it's what a similar quotas config command test did, but you're right that a real mock would be more appropriate here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r534496946



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -56,11 +56,12 @@ import scala.collection._
  *                          --entity-type users --entity-name <user-principal> --entity-type clients --entity-name <client-id>
  *     <li> broker: --broker <broker-id> OR --entity-type brokers --entity-name <broker-id>
  *     <li> broker-logger: --broker-logger <broker-id> OR --entity-type broker-loggers --entity-name <broker-id>
+ *     <li> ip: --IP <ip> OR --entity-type IPs --entity-name <ip>
  * </ul>
- * --entity-type <users|clients|brokers> --entity-default may be specified in place of --entity-type <users|clients|brokers> --entity-name <entityName>
- * when describing or altering default configuration for users, clients, or brokers, respectively.
- * Alternatively, --user-defaults, --client-defaults, or --broker-defaults may be specified in place of
- * --entity-type <users|clients|brokers> --entity-default, respectively.
+ * --entity-type <users|clients|brokers|IPs> --entity-default may be specified in place of --entity-type <users|clients|brokers|IPs> --entity-name <entityName>
+ * when describing or altering default configuration for users, clients, brokers, or IPs, respectively.
+ * Alternatively, --user-defaults, --client-defaults, --broker-defaults, or --IP-defaults may be specified in place of

Review comment:
       sounds good. In this case, do we want to change all mentions of IP to lower-case? including entity type/ZK path.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] dajac commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r538477244



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -864,13 +885,20 @@ object ConfigCommand extends Config {
         }
       }
 
+      if (hasEntityName && entityTypeVals.contains(ConfigType.Ip)) {
+        Seq(entityName, ip).filter(options.has(_)).map(options.valueOf(_)).foreach(DynamicConfig.Ip.validateIpOrHost)

Review comment:
       ditto here. It is a bit weird to have an `InvalidRequestException` thrown here.

##########
File path: core/src/main/scala/kafka/server/DynamicConfig.scala
##########
@@ -141,6 +144,18 @@ object DynamicConfig {
     def names = ipConfigs.names
 
     def validate(props: Properties) = DynamicConfig.validate(ipConfigs, props, customPropsAllowed = false)
+
+    def validateIpOrHost(ip: String): Unit = {
+      if (ip != ConfigEntityName.Default) {
+        if (!Utils.validHostPattern(ip))
+          throw new InvalidRequestException(s"$ip is not a valid hostname")

Review comment:
       I wonder if this step is really necessary as `InetAddress.getByName` also ensures that the hostname is valid. What do you think?

##########
File path: core/src/main/scala/kafka/server/DynamicConfig.scala
##########
@@ -141,6 +144,18 @@ object DynamicConfig {
     def names = ipConfigs.names
 
     def validate(props: Properties) = DynamicConfig.validate(ipConfigs, props, customPropsAllowed = false)
+
+    def validateIpOrHost(ip: String): Unit = {
+      if (ip != ConfigEntityName.Default) {
+        if (!Utils.validHostPattern(ip))
+          throw new InvalidRequestException(s"$ip is not a valid hostname")
+        try {
+          InetAddress.getByName(ip)
+        } catch {
+          case _ :UnknownHostException => throw new InvalidRequestException(s"$ip is not a valid IP or resolvable hostname")

Review comment:
       nit: `_ :UnknownHostException` => `_: UnknownHostException`

##########
File path: core/src/main/scala/kafka/zk/AdminZkClient.scala
##########
@@ -394,8 +393,7 @@ class AdminZkClient(zkClient: KafkaZkClient) extends Logging {
    * @param configs properties to validate for the IP
    */
   def validateIpConfig(ip: String, configs: Properties): Unit = {
-    if (ip != ConfigEntityName.Default && !Utils.validHostPattern(ip))
-      throw new AdminOperationException(s"IP $ip is not a valid address.")
+    DynamicConfig.Ip.validateIpOrHost(ip)

Review comment:
       It is a little weird to throw `InvalidRequestException` here because there is not request involved. Users of `AdminZkClient` may not expect this. I think that it would be better to throw a `IllegalArgumentException`. Then in the `AdminManager#alterClientQuotas`, we can catch it and transform it into `InvalidRequestException` by copying the internal message.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r537937180



##########
File path: core/src/test/scala/unit/kafka/admin/ConfigCommandTest.scala
##########
@@ -472,6 +512,134 @@ class ConfigCommandTest extends ZooKeeperTestHarness with Logging {
     EasyMock.reset(alterResult, describeResult)
   }
 
+  @Test
+  def shouldNotAlterNonQuotaIpConfigsUsingBootstrapServer(): Unit = {
+    // when using --bootstrap-server, it should be illegal to alter anything that is not a connection quota
+    // for ip entities
+    val node = new Node(1, "localhost", 9092)
+    val mockAdminClient = new MockAdminClient(util.Collections.singletonList(node), node)
+
+    def verifyCommand(entityType: String, alterOpts: String*): Unit = {
+      val opts = new ConfigCommandOptions(Array("--bootstrap-server", "localhost:9092",
+        "--entity-type", entityType, "--entity-name", "admin",
+        "--alter") ++ alterOpts)
+      val e = intercept[IllegalArgumentException] {
+        ConfigCommand.alterConfig(mockAdminClient, opts)
+      }
+      assertTrue(s"Unexpected exception: $e", e.getMessage.contains("some_config"))
+    }
+
+    verifyCommand("ips", "--add-config", "connection_creation_rate=10000,some_config=10")
+    verifyCommand("ips", "--add-config", "some_config=10")
+    verifyCommand("ips", "--delete-config", "connection_creation_rate=10000,some_config=10")
+    verifyCommand("ips", "--delete-config", "some_config=10")
+  }
+
+  @Test
+  def testDescribeIpConfigs(): Unit = {
+    def testShouldDescribeIpConfig(ip: Option[String]): Unit = {
+      val entityType = ClientQuotaEntity.IP
+      val (ipArgs, filterComponent) = ip match {
+        case Some(null) => (Array("--entity-default"), ClientQuotaFilterComponent.ofDefaultEntity(entityType))
+        case Some(addr) => (Array("--entity-name", addr), ClientQuotaFilterComponent.ofEntity(entityType, addr))
+        case None => (Array.empty[String], ClientQuotaFilterComponent.ofEntityType(entityType))
+      }
+
+      val describeOpts = new ConfigCommandOptions(Array("--bootstrap-server", "localhost:9092",
+        "--describe", "--entity-type", "ips") ++ ipArgs)
+
+      val expectedFilter = ClientQuotaFilter.containsOnly(List(filterComponent).asJava)
+
+      var describedConfigs = false
+      val describeFuture = new KafkaFutureImpl[util.Map[ClientQuotaEntity, util.Map[String, java.lang.Double]]]
+      describeFuture.complete(Map.empty[ClientQuotaEntity, util.Map[String, java.lang.Double]].asJava)
+      val describeResult: DescribeClientQuotasResult = EasyMock.createNiceMock(classOf[DescribeClientQuotasResult])
+      EasyMock.expect(describeResult.entities()).andReturn(describeFuture)
+
+      val node = new Node(1, "localhost", 9092)
+      val mockAdminClient = new MockAdminClient(util.Collections.singletonList(node), node) {
+        override def describeClientQuotas(filter: ClientQuotaFilter, options: DescribeClientQuotasOptions): DescribeClientQuotasResult = {
+          assertTrue(filter.strict)
+          assertEquals(expectedFilter, filter)
+          describedConfigs = true
+          describeResult
+        }
+      }

Review comment:
       I took an attempt at this, and realized that the reason we use `MockAdminClient` is because `ClientQuotaFilter` and `ClientQuotaAlteration` take `Collection` in their constructors.
   
   If we used a real mock, it would make the tests unnecessarily brittle, since we'd have to match the collection type used in the `ConfigCommand` implementation.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r537937603



##########
File path: core/src/test/scala/unit/kafka/admin/ConfigCommandTest.scala
##########
@@ -472,6 +512,134 @@ class ConfigCommandTest extends ZooKeeperTestHarness with Logging {
     EasyMock.reset(alterResult, describeResult)
   }
 
+  @Test
+  def shouldNotAlterNonQuotaIpConfigsUsingBootstrapServer(): Unit = {
+    // when using --bootstrap-server, it should be illegal to alter anything that is not a connection quota
+    // for ip entities
+    val node = new Node(1, "localhost", 9092)
+    val mockAdminClient = new MockAdminClient(util.Collections.singletonList(node), node)
+
+    def verifyCommand(entityType: String, alterOpts: String*): Unit = {
+      val opts = new ConfigCommandOptions(Array("--bootstrap-server", "localhost:9092",
+        "--entity-type", entityType, "--entity-name", "admin",
+        "--alter") ++ alterOpts)
+      val e = intercept[IllegalArgumentException] {
+        ConfigCommand.alterConfig(mockAdminClient, opts)
+      }
+      assertTrue(s"Unexpected exception: $e", e.getMessage.contains("some_config"))
+    }
+
+    verifyCommand("ips", "--add-config", "connection_creation_rate=10000,some_config=10")
+    verifyCommand("ips", "--add-config", "some_config=10")
+    verifyCommand("ips", "--delete-config", "connection_creation_rate=10000,some_config=10")
+    verifyCommand("ips", "--delete-config", "some_config=10")
+  }
+
+  @Test
+  def testDescribeIpConfigs(): Unit = {
+    def testShouldDescribeIpConfig(ip: Option[String]): Unit = {

Review comment:
       done. refactored the method to be more generic as well, to reduce some code duplication with an existing test.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] apovzner commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
apovzner commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r535830690



##########
File path: core/src/test/scala/unit/kafka/admin/ConfigCommandTest.scala
##########
@@ -472,6 +512,134 @@ class ConfigCommandTest extends ZooKeeperTestHarness with Logging {
     EasyMock.reset(alterResult, describeResult)
   }
 
+  @Test
+  def shouldNotAlterNonQuotaIpConfigsUsingBootstrapServer(): Unit = {
+    // when using --bootstrap-server, it should be illegal to alter anything that is not a connection quota
+    // for ip entities
+    val node = new Node(1, "localhost", 9092)
+    val mockAdminClient = new MockAdminClient(util.Collections.singletonList(node), node)
+
+    def verifyCommand(entityType: String, alterOpts: String*): Unit = {
+      val opts = new ConfigCommandOptions(Array("--bootstrap-server", "localhost:9092",
+        "--entity-type", entityType, "--entity-name", "admin",
+        "--alter") ++ alterOpts)
+      val e = intercept[IllegalArgumentException] {
+        ConfigCommand.alterConfig(mockAdminClient, opts)
+      }
+      assertTrue(s"Unexpected exception: $e", e.getMessage.contains("some_config"))
+    }
+
+    verifyCommand("ips", "--add-config", "connection_creation_rate=10000,some_config=10")

Review comment:
       super tiny nit: `verifyCommand` name makes the code a bit confusing since the command is expected to fail. What about `verifyCommandFails` or `verifyFails`?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] apovzner commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
apovzner commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r534570444



##########
File path: clients/src/main/java/org/apache/kafka/common/quota/ClientQuotaEntity.java
##########
@@ -32,6 +32,7 @@
      */
     public static final String USER = "user";
     public static final String CLIENT_ID = "client-id";
+    public static final String IP = "IP";

Review comment:
       as I mentioned in my other comment, agreed with using lower-case.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] dajac commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r536134783



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -864,11 +885,21 @@ object ConfigCommand extends Config {
         }
       }
 
+      if (hasEntityName && entityTypeVals.contains(ConfigType.Ip)) {
+        Seq(entityName, ip).filter(options.has(_)).map(options.valueOf(_)).foreach { ipAddress =>
+          if (!Utils.validHostPattern(ipAddress))
+            throw new IllegalArgumentException(s"The entity name for ${entityTypeVals.head} must be a valid IP, but it is: $ipAddress")
+        }
+      }
+
       if (options.has(describeOpt) && entityTypeVals.contains(BrokerLoggerConfigType) && !hasEntityName)
         throw new IllegalArgumentException(s"an entity name must be specified with --describe of ${entityTypeVals.mkString(",")}")
 
       if (options.has(alterOpt)) {
-        if (entityTypeVals.contains(ConfigType.User) || entityTypeVals.contains(ConfigType.Client) || entityTypeVals.contains(ConfigType.Broker)) {
+        if (entityTypeVals.contains(ConfigType.User) ||
+            entityTypeVals.contains(ConfigType.Client) ||
+            entityTypeVals.contains(ConfigType.Broker) ||
+            entityTypeVals.contains(ConfigType.Ip)) {
           if (!hasEntityName && !hasEntityDefault)
             throw new IllegalArgumentException("an entity-name or default entity must be specified with --alter of users, clients or brokers")

Review comment:
       nit: Could we add ip in the list as well?

##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -778,6 +793,10 @@ object ConfigCommand extends Config {
     val brokerLogger = parser.accepts("broker-logger", "The broker's ID for its logger config.")
       .withRequiredArg
       .ofType(classOf[String])
+    val ipDefaults = parser.accepts("ip-defaults", "The config defaults for all IPs.")
+    val ip = parser.accepts("ip", "The IP address.")

Review comment:
       It would be good to add these two flags in the KIP as well for completeness.

##########
File path: core/src/test/scala/unit/kafka/admin/ConfigCommandTest.scala
##########
@@ -472,6 +512,134 @@ class ConfigCommandTest extends ZooKeeperTestHarness with Logging {
     EasyMock.reset(alterResult, describeResult)
   }
 
+  @Test
+  def shouldNotAlterNonQuotaIpConfigsUsingBootstrapServer(): Unit = {
+    // when using --bootstrap-server, it should be illegal to alter anything that is not a connection quota
+    // for ip entities
+    val node = new Node(1, "localhost", 9092)
+    val mockAdminClient = new MockAdminClient(util.Collections.singletonList(node), node)
+
+    def verifyCommand(entityType: String, alterOpts: String*): Unit = {
+      val opts = new ConfigCommandOptions(Array("--bootstrap-server", "localhost:9092",
+        "--entity-type", entityType, "--entity-name", "admin",
+        "--alter") ++ alterOpts)
+      val e = intercept[IllegalArgumentException] {
+        ConfigCommand.alterConfig(mockAdminClient, opts)
+      }
+      assertTrue(s"Unexpected exception: $e", e.getMessage.contains("some_config"))
+    }
+
+    verifyCommand("ips", "--add-config", "connection_creation_rate=10000,some_config=10")
+    verifyCommand("ips", "--add-config", "some_config=10")
+    verifyCommand("ips", "--delete-config", "connection_creation_rate=10000,some_config=10")
+    verifyCommand("ips", "--delete-config", "some_config=10")
+  }
+
+  @Test
+  def testDescribeIpConfigs(): Unit = {
+    def testShouldDescribeIpConfig(ip: Option[String]): Unit = {
+      val entityType = ClientQuotaEntity.IP
+      val (ipArgs, filterComponent) = ip match {
+        case Some(null) => (Array("--entity-default"), ClientQuotaFilterComponent.ofDefaultEntity(entityType))
+        case Some(addr) => (Array("--entity-name", addr), ClientQuotaFilterComponent.ofEntity(entityType, addr))
+        case None => (Array.empty[String], ClientQuotaFilterComponent.ofEntityType(entityType))
+      }
+
+      val describeOpts = new ConfigCommandOptions(Array("--bootstrap-server", "localhost:9092",
+        "--describe", "--entity-type", "ips") ++ ipArgs)
+
+      val expectedFilter = ClientQuotaFilter.containsOnly(List(filterComponent).asJava)
+
+      var describedConfigs = false
+      val describeFuture = new KafkaFutureImpl[util.Map[ClientQuotaEntity, util.Map[String, java.lang.Double]]]
+      describeFuture.complete(Map.empty[ClientQuotaEntity, util.Map[String, java.lang.Double]].asJava)
+      val describeResult: DescribeClientQuotasResult = EasyMock.createNiceMock(classOf[DescribeClientQuotasResult])
+      EasyMock.expect(describeResult.entities()).andReturn(describeFuture)
+
+      val node = new Node(1, "localhost", 9092)
+      val mockAdminClient = new MockAdminClient(util.Collections.singletonList(node), node) {
+        override def describeClientQuotas(filter: ClientQuotaFilter, options: DescribeClientQuotasOptions): DescribeClientQuotasResult = {
+          assertTrue(filter.strict)
+          assertEquals(expectedFilter, filter)
+          describedConfigs = true
+          describeResult
+        }
+      }

Review comment:
       Have you considered using a real mock instead of doing this? Is there a reason not to? It seems to be that it would be more appropriate.

##########
File path: core/src/test/scala/unit/kafka/admin/ConfigCommandTest.scala
##########
@@ -472,6 +512,134 @@ class ConfigCommandTest extends ZooKeeperTestHarness with Logging {
     EasyMock.reset(alterResult, describeResult)
   }
 
+  @Test
+  def shouldNotAlterNonQuotaIpConfigsUsingBootstrapServer(): Unit = {
+    // when using --bootstrap-server, it should be illegal to alter anything that is not a connection quota
+    // for ip entities
+    val node = new Node(1, "localhost", 9092)
+    val mockAdminClient = new MockAdminClient(util.Collections.singletonList(node), node)
+
+    def verifyCommand(entityType: String, alterOpts: String*): Unit = {
+      val opts = new ConfigCommandOptions(Array("--bootstrap-server", "localhost:9092",
+        "--entity-type", entityType, "--entity-name", "admin",
+        "--alter") ++ alterOpts)
+      val e = intercept[IllegalArgumentException] {
+        ConfigCommand.alterConfig(mockAdminClient, opts)
+      }
+      assertTrue(s"Unexpected exception: $e", e.getMessage.contains("some_config"))
+    }
+
+    verifyCommand("ips", "--add-config", "connection_creation_rate=10000,some_config=10")
+    verifyCommand("ips", "--add-config", "some_config=10")
+    verifyCommand("ips", "--delete-config", "connection_creation_rate=10000,some_config=10")
+    verifyCommand("ips", "--delete-config", "some_config=10")
+  }
+
+  @Test
+  def testDescribeIpConfigs(): Unit = {
+    def testShouldDescribeIpConfig(ip: Option[String]): Unit = {

Review comment:
       nit: I feel like this inner method is quite large and may merit to be extracted. It feels a bit weird when the inner method is way larger than the actual body of the method. I guess that it is a matter of taste but I believe that the code would be more readable. I leave this up to you.

##########
File path: core/src/test/scala/unit/kafka/admin/ConfigCommandTest.scala
##########
@@ -472,6 +512,134 @@ class ConfigCommandTest extends ZooKeeperTestHarness with Logging {
     EasyMock.reset(alterResult, describeResult)
   }
 
+  @Test
+  def shouldNotAlterNonQuotaIpConfigsUsingBootstrapServer(): Unit = {
+    // when using --bootstrap-server, it should be illegal to alter anything that is not a connection quota
+    // for ip entities
+    val node = new Node(1, "localhost", 9092)
+    val mockAdminClient = new MockAdminClient(util.Collections.singletonList(node), node)
+
+    def verifyCommand(entityType: String, alterOpts: String*): Unit = {
+      val opts = new ConfigCommandOptions(Array("--bootstrap-server", "localhost:9092",
+        "--entity-type", entityType, "--entity-name", "admin",

Review comment:
       Actually, I believe that the command should fail due to `admin` which is not valid IP. It may be better to use a valid IP address here as you want to tests the configs.

##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -864,11 +885,21 @@ object ConfigCommand extends Config {
         }
       }
 
+      if (hasEntityName && entityTypeVals.contains(ConfigType.Ip)) {
+        Seq(entityName, ip).filter(options.has(_)).map(options.valueOf(_)).foreach { ipAddress =>
+          if (!Utils.validHostPattern(ipAddress))

Review comment:
       I think that we should validate that `ipAddress` is actually an IP. `validHostPattern` only ensure that it is a valid host name.

##########
File path: core/src/test/scala/unit/kafka/admin/ConfigCommandTest.scala
##########
@@ -398,19 +418,39 @@ class ConfigCommandTest extends ZooKeeperTestHarness with Logging {
     ConfigCommand.alterConfigWithZk(null, createOpts, new TestAdminZkClient(zkClient))
   }
 
-  def testShouldAddClientConfig(user: Option[String], clientId: Option[String]): Unit = {
-    def toValues(entityName: Option[String], entityType: String, command: String):
-        (Array[String], Option[String], Option[ClientQuotaFilterComponent]) = {
-      entityName match {
-        case Some(null) =>
-          (Array("--entity-type", command, "--entity-default"), Some(null),
-            Some(ClientQuotaFilterComponent.ofDefaultEntity(entityType)))
-        case Some(name) =>
-          (Array("--entity-type", command, "--entity-name", name), Some(name),
-            Some(ClientQuotaFilterComponent.ofEntity(entityType, name)))
-        case None => (Array.empty, None, None)
+  @Test
+  def shouldAddIpConfigsUsingZookeeper(): Unit = {
+    val createOpts = new ConfigCommandOptions(Array("--zookeeper", zkConnect,
+      "--entity-name", "1.2.3.4",
+      "--entity-type", "ips",
+      "--alter",
+      "--add-config", "a=b,c=d"))
+
+    class TestAdminZkClient(zkClient: KafkaZkClient) extends AdminZkClient(zkClient) {
+      override def changeIpConfig(ip: String, configChange: Properties): Unit = {
+        assertEquals("1.2.3.4", ip)
+        assertEquals("b", configChange.get("a"))
+        assertEquals("d", configChange.get("c"))
       }
     }
+
+    ConfigCommand.alterConfigWithZk(null, createOpts, new TestAdminZkClient(zkClient))
+  }
+
+  private def toValues(entityName: Option[String], entityType: String, command: String):
+  (Array[String], Option[String], Option[ClientQuotaFilterComponent]) = {

Review comment:
       nit: Indentation seems wrong here.

##########
File path: core/src/main/scala/kafka/server/AdminManager.scala
##########
@@ -920,32 +954,51 @@ class AdminManager(val config: KafkaConfig,
         !name.isDefined || !strict
     }
 
-    def fromProps(props: Map[String, String]): Map[String, Double] = {
-      props.map { case (key, value) =>
-        val doubleValue = try value.toDouble catch {
-          case _: NumberFormatException =>
-            throw new IllegalStateException(s"Unexpected client quota configuration value: $key -> $value")
-        }
-        key -> doubleValue
-      }
-    }
-
-    (userEntries ++ clientIdEntries ++ bothEntries).map { case ((u, c), p) =>
+    (userEntries ++ clientIdEntries ++ bothEntries).flatMap { case ((u, c), p) =>
       val quotaProps = p.asScala.filter { case (key, _) => QuotaConfigs.isQuotaConfig(key) }
       if (quotaProps.nonEmpty && matches(userComponent, u) && matches(clientIdComponent, c))
         Some(userClientIdToEntity(u, c) -> fromProps(quotaProps))
       else
         None
-    }.flatten.toMap
+    }.toMap
+  }
+
+  def handleDescribeIpQuotas(ipComponent: Option[ClientQuotaFilterComponent], strict: Boolean): Map[ClientQuotaEntity, Map[String, Double]] = {
+    val ip = ipComponent.flatMap(c => toOption(c.`match`))
+    val exactIp = wantExact(ipComponent)
+    val allIps = ipComponent.exists(_.`match` == null) || (ipComponent.isEmpty && !strict)
+    val ipEntries = if (exactIp)
+      Map(Some(ip.get) -> adminZkClient.fetchEntityConfig(ConfigType.Ip, sanitized(ip)))
+    else if (allIps)
+      adminZkClient.fetchAllEntityConfigs(ConfigType.Ip).map { case (name, props) =>
+        Some(desanitizeEntityName(name)) -> props
+      }
+    else
+      Map.empty
+
+    def ipToQuotaEntity(ip: Option[String]): ClientQuotaEntity = {
+      new ClientQuotaEntity(ip.map(ipName => ClientQuotaEntity.IP -> ipName).toMap.asJava)
+    }
+
+    ipEntries.flatMap { case (ip, props) =>
+      val ipQuotaProps = props.asScala.filter { case (key, _) => DynamicConfig.Ip.names.contains(key) }
+      if (ipQuotaProps.nonEmpty)
+        Some(ipToQuotaEntity(ip) -> fromProps(ipQuotaProps))
+      else
+        None
+    }
   }
 
   def alterClientQuotas(entries: Seq[ClientQuotaAlteration], validateOnly: Boolean): Map[ClientQuotaEntity, ApiError] = {
     def alterEntityQuotas(entity: ClientQuotaEntity, ops: Iterable[ClientQuotaAlteration.Op]): Unit = {
-      val (path, configType, configKeys) = entityToSanitizedUserClientId(entity) match {
-        case (Some(user), Some(clientId)) => (user + "/clients/" + clientId, ConfigType.User, DynamicConfig.User.configKeys)
-        case (Some(user), None) => (user, ConfigType.User, DynamicConfig.User.configKeys)
-        case (None, Some(clientId)) => (clientId, ConfigType.Client, DynamicConfig.Client.configKeys)
-        case _ => throw new InvalidRequestException("Invalid empty client quota entity")
+      val (path, configType, configKeys) = parseAndSanitizeQuotaEntity(entity) match {
+        case (Some(user), Some(clientId), None) => (user + "/clients/" + clientId, ConfigType.User, DynamicConfig.User.configKeys)
+        case (Some(user), None, None) => (user, ConfigType.User, DynamicConfig.User.configKeys)
+        case (None, Some(clientId), None) => (clientId, ConfigType.Client, DynamicConfig.Client.configKeys)
+        case (None, None, Some(ip)) => (ip, ConfigType.Ip, DynamicConfig.Ip.configKeys)
+        case (_, _, Some(_)) => throw new InvalidRequestException(s"Invalid quota entity combination, " +

Review comment:
       nit: Shouldn't we use `(Some(_), Some(_), Some(_))` here in order to be really accurate? As it is, the first two could be anything. That works because it is after all the others but I would prefer to be explicit if possible.

##########
File path: core/src/main/scala/kafka/server/AdminManager.scala
##########
@@ -920,32 +954,51 @@ class AdminManager(val config: KafkaConfig,
         !name.isDefined || !strict
     }
 
-    def fromProps(props: Map[String, String]): Map[String, Double] = {
-      props.map { case (key, value) =>
-        val doubleValue = try value.toDouble catch {
-          case _: NumberFormatException =>
-            throw new IllegalStateException(s"Unexpected client quota configuration value: $key -> $value")
-        }
-        key -> doubleValue
-      }
-    }
-
-    (userEntries ++ clientIdEntries ++ bothEntries).map { case ((u, c), p) =>
+    (userEntries ++ clientIdEntries ++ bothEntries).flatMap { case ((u, c), p) =>
       val quotaProps = p.asScala.filter { case (key, _) => QuotaConfigs.isQuotaConfig(key) }
       if (quotaProps.nonEmpty && matches(userComponent, u) && matches(clientIdComponent, c))
         Some(userClientIdToEntity(u, c) -> fromProps(quotaProps))
       else
         None
-    }.flatten.toMap
+    }.toMap
+  }
+
+  def handleDescribeIpQuotas(ipComponent: Option[ClientQuotaFilterComponent], strict: Boolean): Map[ClientQuotaEntity, Map[String, Double]] = {
+    val ip = ipComponent.flatMap(c => toOption(c.`match`))
+    val exactIp = wantExact(ipComponent)
+    val allIps = ipComponent.exists(_.`match` == null) || (ipComponent.isEmpty && !strict)
+    val ipEntries = if (exactIp)
+      Map(Some(ip.get) -> adminZkClient.fetchEntityConfig(ConfigType.Ip, sanitized(ip)))
+    else if (allIps)
+      adminZkClient.fetchAllEntityConfigs(ConfigType.Ip).map { case (name, props) =>
+        Some(desanitizeEntityName(name)) -> props
+      }
+    else
+      Map.empty
+
+    def ipToQuotaEntity(ip: Option[String]): ClientQuotaEntity = {
+      new ClientQuotaEntity(ip.map(ipName => ClientQuotaEntity.IP -> ipName).toMap.asJava)
+    }
+
+    ipEntries.flatMap { case (ip, props) =>
+      val ipQuotaProps = props.asScala.filter { case (key, _) => DynamicConfig.Ip.names.contains(key) }
+      if (ipQuotaProps.nonEmpty)
+        Some(ipToQuotaEntity(ip) -> fromProps(ipQuotaProps))
+      else
+        None
+    }
   }
 
   def alterClientQuotas(entries: Seq[ClientQuotaAlteration], validateOnly: Boolean): Map[ClientQuotaEntity, ApiError] = {
     def alterEntityQuotas(entity: ClientQuotaEntity, ops: Iterable[ClientQuotaAlteration.Op]): Unit = {
-      val (path, configType, configKeys) = entityToSanitizedUserClientId(entity) match {
-        case (Some(user), Some(clientId)) => (user + "/clients/" + clientId, ConfigType.User, DynamicConfig.User.configKeys)
-        case (Some(user), None) => (user, ConfigType.User, DynamicConfig.User.configKeys)
-        case (None, Some(clientId)) => (clientId, ConfigType.Client, DynamicConfig.Client.configKeys)
-        case _ => throw new InvalidRequestException("Invalid empty client quota entity")
+      val (path, configType, configKeys) = parseAndSanitizeQuotaEntity(entity) match {

Review comment:
       It seems that we should validate the provided `ip` somewhere here. I just tried with the following command:
   ```
   ./bin/kafka-configs.sh --bootstrap-server localhost:9092 --alter --entity-type ips --entity-name ip --add-config  connection_creation_rate=10
   ```
   The command completes but then the broker fails while applying the configuration:
   ```
   [2020-12-04 15:31:01,951] ERROR error processing change notification {"version":2,"entity_path":"ips/ip"} from /config/changes/config_change_0000000001 (kafka.common.ZkNodeChangeNotificationListener)
   java.lang.IllegalArgumentException: Unable to resolve address ip
   	at kafka.server.IpConfigHandler.processConfigChanges(ConfigHandler.scala:199)
   	at kafka.server.DynamicConfigManager$ConfigChangedNotificationHandler$.processEntityConfigChangeVersion2(DynamicConfigManager.scala:151)
   	at kafka.server.DynamicConfigManager$ConfigChangedNotificationHandler$.$anonfun$processNotification$1(DynamicConfigManager.scala:105)
   	at kafka.server.DynamicConfigManager$ConfigChangedNotificationHandler$.processNotification(DynamicConfigManager.scala:96)
   	at kafka.common.ZkNodeChangeNotificationListener.$anonfun$processNotification$1(ZkNodeChangeNotificationListener.scala:106)
   	at kafka.common.ZkNodeChangeNotificationListener.processNotification(ZkNodeChangeNotificationListener.scala:106)
   	at kafka.common.ZkNodeChangeNotificationListener.$anonfun$processNotifications$2(ZkNodeChangeNotificationListener.scala:90)
   	at kafka.common.ZkNodeChangeNotificationListener.$anonfun$processNotifications$2$adapted(ZkNodeChangeNotificationListener.scala:87)
   	at scala.collection.IterableOnceOps.foreach(IterableOnce.scala:563)
   	at scala.collection.IterableOnceOps.foreach$(IterableOnce.scala:561)
   	at scala.collection.AbstractIterable.foreach(Iterable.scala:919)
   	at kafka.common.ZkNodeChangeNotificationListener.kafka$common$ZkNodeChangeNotificationListener$$processNotifications(ZkNodeChangeNotificationListener.scala:87)
   	at kafka.common.ZkNodeChangeNotificationListener$ChangeNotification.process(ZkNodeChangeNotificationListener.scala:120)
   	at kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread.doWork(ZkNodeChangeNotificationListener.scala:146)
   	at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
   ```
   
   It would be good to cover this in the tests, both for the command and the api.
   
   This is also true with:
   ```
   ./bin/kafka-configs.sh --zookeeper localhost:2181 --alter --ip ip --add-config connection_creation_rate=2
   ```

##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -864,11 +885,21 @@ object ConfigCommand extends Config {
         }
       }
 
+      if (hasEntityName && entityTypeVals.contains(ConfigType.Ip)) {
+        Seq(entityName, ip).filter(options.has(_)).map(options.valueOf(_)).foreach { ipAddress =>

Review comment:
       nit: We can probably remove `(_)` in `filter` and `map`. 

##########
File path: core/src/test/scala/unit/kafka/server/ClientQuotasRequestTest.scala
##########
@@ -236,13 +310,22 @@ class ClientQuotasRequestTest extends BaseRequestTest {
     (Some("user-3"), None, 58.58),
     (Some(null), None, 59.59),
     (None, Some("client-id-2"), 60.60)
-  ).map { case (u, c, v) => (toEntity(u, c), v) }
+  ).map { case (u, c, v) => (toClientEntity(u, c), v) }
+
+  private val matchIpEntities = List(
+    (Some("1.2.3.4"), 10.0),
+    (Some("2.3.4.5"), 20.0)
+  ).map { case (ip, quota) => (toIpEntity(ip), quota)}
 
   private def setupDescribeClientQuotasMatchTest() = {
-    val result = alterClientQuotas(matchEntities.map { case (e, v) =>
-      (e -> Map((RequestPercentageProp, Some(v))))
-    }.toMap, validateOnly = false)
-    matchEntities.foreach(e => result.get(e._1).get.get(10, TimeUnit.SECONDS))
+    val userClientQuotas = matchUserClientEntities.map { case (e, v) =>
+      e -> Map((RequestPercentageProp, Some(v)))
+    }.toMap
+    val ipQuotas = matchIpEntities.map { case (e, v) =>
+      e -> Map((IpConnectionRateProp, Some(v)))
+    }.toMap
+    val result = alterClientQuotas(userClientQuotas ++ ipQuotas, validateOnly = false)
+    (matchUserClientEntities ++ matchIpEntities).foreach(e => result(e._1).get(10, TimeUnit.SECONDS))
 
     // Allow time for watch callbacks to be triggered.
     Thread.sleep(500)

Review comment:
       This is not related to your PR but I wonder if it would be possible to replace this sleep by `retry` or `waitUntilTrue`. Thoughts?

##########
File path: core/src/test/scala/unit/kafka/admin/ConfigCommandTest.scala
##########
@@ -472,6 +512,134 @@ class ConfigCommandTest extends ZooKeeperTestHarness with Logging {
     EasyMock.reset(alterResult, describeResult)
   }
 
+  @Test
+  def shouldNotAlterNonQuotaIpConfigsUsingBootstrapServer(): Unit = {
+    // when using --bootstrap-server, it should be illegal to alter anything that is not a connection quota
+    // for ip entities
+    val node = new Node(1, "localhost", 9092)
+    val mockAdminClient = new MockAdminClient(util.Collections.singletonList(node), node)
+
+    def verifyCommand(entityType: String, alterOpts: String*): Unit = {
+      val opts = new ConfigCommandOptions(Array("--bootstrap-server", "localhost:9092",
+        "--entity-type", entityType, "--entity-name", "admin",
+        "--alter") ++ alterOpts)
+      val e = intercept[IllegalArgumentException] {
+        ConfigCommand.alterConfig(mockAdminClient, opts)
+      }
+      assertTrue(s"Unexpected exception: $e", e.getMessage.contains("some_config"))
+    }
+
+    verifyCommand("ips", "--add-config", "connection_creation_rate=10000,some_config=10")
+    verifyCommand("ips", "--add-config", "some_config=10")
+    verifyCommand("ips", "--delete-config", "connection_creation_rate=10000,some_config=10")
+    verifyCommand("ips", "--delete-config", "some_config=10")
+  }
+
+  @Test
+  def testDescribeIpConfigs(): Unit = {
+    def testShouldDescribeIpConfig(ip: Option[String]): Unit = {
+      val entityType = ClientQuotaEntity.IP
+      val (ipArgs, filterComponent) = ip match {
+        case Some(null) => (Array("--entity-default"), ClientQuotaFilterComponent.ofDefaultEntity(entityType))
+        case Some(addr) => (Array("--entity-name", addr), ClientQuotaFilterComponent.ofEntity(entityType, addr))
+        case None => (Array.empty[String], ClientQuotaFilterComponent.ofEntityType(entityType))
+      }
+
+      val describeOpts = new ConfigCommandOptions(Array("--bootstrap-server", "localhost:9092",
+        "--describe", "--entity-type", "ips") ++ ipArgs)
+
+      val expectedFilter = ClientQuotaFilter.containsOnly(List(filterComponent).asJava)
+
+      var describedConfigs = false
+      val describeFuture = new KafkaFutureImpl[util.Map[ClientQuotaEntity, util.Map[String, java.lang.Double]]]
+      describeFuture.complete(Map.empty[ClientQuotaEntity, util.Map[String, java.lang.Double]].asJava)
+      val describeResult: DescribeClientQuotasResult = EasyMock.createNiceMock(classOf[DescribeClientQuotasResult])
+      EasyMock.expect(describeResult.entities()).andReturn(describeFuture)
+
+      val node = new Node(1, "localhost", 9092)
+      val mockAdminClient = new MockAdminClient(util.Collections.singletonList(node), node) {
+        override def describeClientQuotas(filter: ClientQuotaFilter, options: DescribeClientQuotasOptions): DescribeClientQuotasResult = {
+          assertTrue(filter.strict)
+          assertEquals(expectedFilter, filter)
+          describedConfigs = true
+          describeResult
+        }
+      }
+      EasyMock.replay(describeResult)
+      ConfigCommand.describeConfig(mockAdminClient, describeOpts)
+      assertTrue(describedConfigs)
+      EasyMock.reset(describeResult)
+    }
+
+    testShouldDescribeIpConfig(Some("1.2.3.4"))
+    testShouldDescribeIpConfig(Some(null))
+    testShouldDescribeIpConfig(None)
+  }
+
+  @Test
+  def testAlterIpConfig(): Unit = {
+    def testShouldAlterIpConfig(ip: Option[String], remove: Boolean): Unit = {
+      val (ipArgs, ipEntity, ipComponent) = toValues(ip, ClientQuotaEntity.IP, "ips")
+      val alterOpts = if (remove)
+        Array("--delete-config", "connection_creation_rate")
+      else
+        Array("--add-config", "connection_creation_rate=100")
+      val createOpts = new ConfigCommandOptions(Array("--bootstrap-server", "localhost:9092",
+        "--alter") ++ ipArgs ++ alterOpts)
+
+      // Explicitly populate a HashMap to ensure nulls are recorded properly.
+      val entityMap = new java.util.HashMap[String, String]
+      ipEntity.foreach(u => entityMap.put(ClientQuotaEntity.IP, u))
+      val entity = new ClientQuotaEntity(entityMap)
+
+      var describedConfigs = false
+      val describeFuture = new KafkaFutureImpl[util.Map[ClientQuotaEntity, util.Map[String, java.lang.Double]]]
+      describeFuture.complete(Map(entity -> Map("connection_creation_rate" -> Double.box(50.0)).asJava).asJava)
+      val describeResult: DescribeClientQuotasResult = EasyMock.createNiceMock(classOf[DescribeClientQuotasResult])
+      EasyMock.expect(describeResult.entities()).andReturn(describeFuture)
+
+      var alteredConfigs = false
+      val alterFuture = new KafkaFutureImpl[Void]
+      alterFuture.complete(null)
+      val alterResult: AlterClientQuotasResult = EasyMock.createNiceMock(classOf[AlterClientQuotasResult])
+      EasyMock.expect(alterResult.all()).andReturn(alterFuture)
+
+      val node = new Node(1, "localhost", 9092)
+      val mockAdminClient = new MockAdminClient(util.Collections.singletonList(node), node) {

Review comment:
       ditto

##########
File path: core/src/test/scala/unit/kafka/server/ClientQuotasRequestTest.scala
##########
@@ -187,6 +190,59 @@ class ClientQuotasRequestTest extends BaseRequestTest {
     ))
   }
 
+  @Test
+  def testAlterIpQuotasRequest(): Unit = {

Review comment:
       We must add tests with invalid IPs.

##########
File path: core/src/test/scala/integration/kafka/network/DynamicConnectionQuotaTest.scala
##########
@@ -259,13 +264,19 @@ class DynamicConnectionQuotaTest extends BaseRequestTest {
   }
 
   private def updateIpConnectionRate(ip: Option[String], updatedRate: Int): Unit = {
-    adminZkClient.changeIpConfig(ip.getOrElse(ConfigEntityName.Default),
-      CoreUtils.propsWith(DynamicConfig.Ip.IpConnectionRateOverrideProp, updatedRate.toString))
+    val initialConnectionCount = connectionCount
+    val adminClient = createAdminClient()

Review comment:
       I second this.

##########
File path: core/src/test/scala/unit/kafka/admin/ConfigCommandTest.scala
##########
@@ -472,6 +512,134 @@ class ConfigCommandTest extends ZooKeeperTestHarness with Logging {
     EasyMock.reset(alterResult, describeResult)
   }
 
+  @Test
+  def shouldNotAlterNonQuotaIpConfigsUsingBootstrapServer(): Unit = {
+    // when using --bootstrap-server, it should be illegal to alter anything that is not a connection quota
+    // for ip entities
+    val node = new Node(1, "localhost", 9092)
+    val mockAdminClient = new MockAdminClient(util.Collections.singletonList(node), node)
+
+    def verifyCommand(entityType: String, alterOpts: String*): Unit = {
+      val opts = new ConfigCommandOptions(Array("--bootstrap-server", "localhost:9092",
+        "--entity-type", entityType, "--entity-name", "admin",
+        "--alter") ++ alterOpts)
+      val e = intercept[IllegalArgumentException] {
+        ConfigCommand.alterConfig(mockAdminClient, opts)
+      }
+      assertTrue(s"Unexpected exception: $e", e.getMessage.contains("some_config"))
+    }
+
+    verifyCommand("ips", "--add-config", "connection_creation_rate=10000,some_config=10")

Review comment:
       I do agree. It took me a while to grasp it. Moreover, the very same `verifyCommand` is defined in other places. I wonder if we could consolidate them into one. What do you think?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r536241597



##########
File path: core/src/test/scala/unit/kafka/server/ClientQuotasRequestTest.scala
##########
@@ -236,13 +310,22 @@ class ClientQuotasRequestTest extends BaseRequestTest {
     (Some("user-3"), None, 58.58),
     (Some(null), None, 59.59),
     (None, Some("client-id-2"), 60.60)
-  ).map { case (u, c, v) => (toEntity(u, c), v) }
+  ).map { case (u, c, v) => (toClientEntity(u, c), v) }
+
+  private val matchIpEntities = List(
+    (Some("1.2.3.4"), 10.0),
+    (Some("2.3.4.5"), 20.0)
+  ).map { case (ip, quota) => (toIpEntity(ip), quota)}
 
   private def setupDescribeClientQuotasMatchTest() = {
-    val result = alterClientQuotas(matchEntities.map { case (e, v) =>
-      (e -> Map((RequestPercentageProp, Some(v))))
-    }.toMap, validateOnly = false)
-    matchEntities.foreach(e => result.get(e._1).get.get(10, TimeUnit.SECONDS))
+    val userClientQuotas = matchUserClientEntities.map { case (e, v) =>
+      e -> Map((RequestPercentageProp, Some(v)))
+    }.toMap
+    val ipQuotas = matchIpEntities.map { case (e, v) =>
+      e -> Map((IpConnectionRateProp, Some(v)))
+    }.toMap
+    val result = alterClientQuotas(userClientQuotas ++ ipQuotas, validateOnly = false)
+    (matchUserClientEntities ++ matchIpEntities).foreach(e => result(e._1).get(10, TimeUnit.SECONDS))
 
     // Allow time for watch callbacks to be triggered.
     Thread.sleep(500)

Review comment:
       makes sense, will do.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r536241597



##########
File path: core/src/test/scala/unit/kafka/server/ClientQuotasRequestTest.scala
##########
@@ -236,13 +310,22 @@ class ClientQuotasRequestTest extends BaseRequestTest {
     (Some("user-3"), None, 58.58),
     (Some(null), None, 59.59),
     (None, Some("client-id-2"), 60.60)
-  ).map { case (u, c, v) => (toEntity(u, c), v) }
+  ).map { case (u, c, v) => (toClientEntity(u, c), v) }
+
+  private val matchIpEntities = List(
+    (Some("1.2.3.4"), 10.0),
+    (Some("2.3.4.5"), 20.0)
+  ).map { case (ip, quota) => (toIpEntity(ip), quota)}
 
   private def setupDescribeClientQuotasMatchTest() = {
-    val result = alterClientQuotas(matchEntities.map { case (e, v) =>
-      (e -> Map((RequestPercentageProp, Some(v))))
-    }.toMap, validateOnly = false)
-    matchEntities.foreach(e => result.get(e._1).get.get(10, TimeUnit.SECONDS))
+    val userClientQuotas = matchUserClientEntities.map { case (e, v) =>
+      e -> Map((RequestPercentageProp, Some(v)))
+    }.toMap
+    val ipQuotas = matchIpEntities.map { case (e, v) =>
+      e -> Map((IpConnectionRateProp, Some(v)))
+    }.toMap
+    val result = alterClientQuotas(userClientQuotas ++ ipQuotas, validateOnly = false)
+    (matchUserClientEntities ++ matchIpEntities).foreach(e => result(e._1).get(10, TimeUnit.SECONDS))
 
     // Allow time for watch callbacks to be triggered.
     Thread.sleep(500)

Review comment:
       makes sense, will do.
   Actually, taking a closer look, there is no need for the `Thread.sleep()` call. None of the tests that use the match setup rely on ZK watches to trigger. I'll just remove the call to `Thread.sleep()` instead.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#issuecomment-740266395


   Thanks for the feedback @dajac 
   I went ahead and also refactored some tests in `ConfigCommandTest` that were a bit repetitive.
   Let me know what you think


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r533600431



##########
File path: clients/src/main/java/org/apache/kafka/common/quota/ClientQuotaEntity.java
##########
@@ -32,6 +32,7 @@
      */
     public static final String USER = "user";
     public static final String CLIENT_ID = "client-id";
+    public static final String IP = "IP";

Review comment:
       I have no objection to editing the KIP to changing this (and other uses of IP) to lower case, I just made it upper case because the original iteration of IP quotas in KIP-612 had IP in upper-case.
   
   @apovzner @dajac for any thoughts on upper vs lower.
   FWIW, the ConfigCommand help output is better ordered when IP is in lower-case due to how it sorts flags.

##########
File path: clients/src/main/java/org/apache/kafka/common/quota/ClientQuotaEntity.java
##########
@@ -32,6 +32,7 @@
      */
     public static final String USER = "user";
     public static final String CLIENT_ID = "client-id";
+    public static final String IP = "IP";

Review comment:
       I have no objection to editing the KIP to changing this (and other uses of IP) to lower case, I just made it upper case because the original iteration of IP quotas in KIP-612 had IP in upper-case.
   
   @apovzner @dajac for any thoughts on upper vs lower.
   FWIW, the `ConfigCommand` help output is better ordered when IP is in lower-case due to how it sorts flags.

##########
File path: clients/src/main/java/org/apache/kafka/common/quota/ClientQuotaEntity.java
##########
@@ -32,6 +32,7 @@
      */
     public static final String USER = "user";
     public static final String CLIENT_ID = "client-id";
+    public static final String IP = "IP";

Review comment:
       I have no objection to editing the KIP to changing this (and other uses of IP) to lower case, I just made it upper case because the original iteration of IP quotas in KIP-612 had IP in upper-case.
   
   @apovzner @dajac for any thoughts on upper vs lower.
   FWIW, the `kafka-configs` help output is better ordered when IP is in lower-case due to how it sorts flags.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r536238405



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -778,6 +793,10 @@ object ConfigCommand extends Config {
     val brokerLogger = parser.accepts("broker-logger", "The broker's ID for its logger config.")
       .withRequiredArg
       .ofType(classOf[String])
+    val ipDefaults = parser.accepts("ip-defaults", "The config defaults for all IPs.")
+    val ip = parser.accepts("ip", "The IP address.")

Review comment:
       will do, along with the change to lower




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] dajac commented on pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
dajac commented on pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#issuecomment-742375037


   Failed test is not related:
   * JDK 11 - kafka.api.PlaintextAdminIntegrationTest.testAlterReplicaLogDirs
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] dajac commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r534287486



##########
File path: core/src/main/scala/kafka/server/AdminManager.scala
##########
@@ -920,32 +954,49 @@ class AdminManager(val config: KafkaConfig,
         !name.isDefined || !strict
     }
 
-    def fromProps(props: Map[String, String]): Map[String, Double] = {
-      props.map { case (key, value) =>
-        val doubleValue = try value.toDouble catch {
-          case _: NumberFormatException =>
-            throw new IllegalStateException(s"Unexpected client quota configuration value: $key -> $value")
-        }
-        key -> doubleValue
-      }
-    }
-
-    (userEntries ++ clientIdEntries ++ bothEntries).map { case ((u, c), p) =>
+    (userEntries ++ clientIdEntries ++ bothEntries).flatMap { case ((u, c), p) =>
       val quotaProps = p.asScala.filter { case (key, _) => QuotaConfigs.isQuotaConfig(key) }
       if (quotaProps.nonEmpty && matches(userComponent, u) && matches(clientIdComponent, c))
         Some(userClientIdToEntity(u, c) -> fromProps(quotaProps))
       else
         None
-    }.flatten.toMap
+    }.toMap
+  }
+
+  def handleDescribeIpQuotas(ipComponent: Option[ClientQuotaFilterComponent], strict: Boolean): Map[ClientQuotaEntity, Map[String, Double]] = {
+    val ip = ipComponent.flatMap(c => toOption(c.`match`))
+    val exactIp = wantExact(ipComponent)
+    val allIps = ipComponent.exists(_.`match` == null) || (ipComponent.isEmpty && !strict)
+    val ipEntries = if (exactIp)
+      Map(Some(ip.get) -> adminZkClient.fetchEntityConfig(ConfigType.Ip, sanitized(ip)))
+    else if (allIps)
+      adminZkClient.fetchAllEntityConfigs(ConfigType.Ip).map { case (name, props) =>
+        Some(desanitizeEntityName(name)) -> props
+      }
+    else
+      Map.empty
+
+    def ipToQuotaEntity(ip: Option[String]): ClientQuotaEntity = {
+      new ClientQuotaEntity(ip.map(ipName => ClientQuotaEntity.IP -> ipName).toMap.asJava)
+    }
+
+    ipEntries.flatMap { case (ip, props) =>
+      val ipQuotaProps = props.asScala.filter { case (key, _) => DynamicConfig.Ip.names.contains(key) }
+      if (ipQuotaProps.nonEmpty)
+        Some(ipToQuotaEntity(ip) -> fromProps(ipQuotaProps))
+      else
+        None
+    }
   }
 
   def alterClientQuotas(entries: Seq[ClientQuotaAlteration], validateOnly: Boolean): Map[ClientQuotaEntity, ApiError] = {
     def alterEntityQuotas(entity: ClientQuotaEntity, ops: Iterable[ClientQuotaAlteration.Op]): Unit = {
-      val (path, configType, configKeys) = entityToSanitizedUserClientId(entity) match {
-        case (Some(user), Some(clientId)) => (user + "/clients/" + clientId, ConfigType.User, DynamicConfig.User.configKeys)
-        case (Some(user), None) => (user, ConfigType.User, DynamicConfig.User.configKeys)
-        case (None, Some(clientId)) => (clientId, ConfigType.Client, DynamicConfig.Client.configKeys)
-        case _ => throw new InvalidRequestException("Invalid empty client quota entity")
+      val (path, configType, configKeys) = parseAndSanitizeQuotaEntity(entity) match {
+        case (Some(user), Some(clientId), None) => (user + "/clients/" + clientId, ConfigType.User, DynamicConfig.User.configKeys)
+        case (Some(user), None, None) => (user, ConfigType.User, DynamicConfig.User.configKeys)
+        case (None, Some(clientId), None) => (clientId, ConfigType.Client, DynamicConfig.Client.configKeys)
+        case (None, None, Some(ip)) => (ip, ConfigType.Ip, DynamicConfig.Ip.configKeys)
+        case _ => throw new InvalidRequestException("Invalid client quota entity")

Review comment:
       Should we also provide a better error message if the user mixes up IP and user/clientId?

##########
File path: core/src/main/scala/kafka/server/AdminManager.scala
##########
@@ -920,32 +954,49 @@ class AdminManager(val config: KafkaConfig,
         !name.isDefined || !strict
     }
 
-    def fromProps(props: Map[String, String]): Map[String, Double] = {
-      props.map { case (key, value) =>
-        val doubleValue = try value.toDouble catch {
-          case _: NumberFormatException =>
-            throw new IllegalStateException(s"Unexpected client quota configuration value: $key -> $value")
-        }
-        key -> doubleValue
-      }
-    }
-
-    (userEntries ++ clientIdEntries ++ bothEntries).map { case ((u, c), p) =>
+    (userEntries ++ clientIdEntries ++ bothEntries).flatMap { case ((u, c), p) =>
       val quotaProps = p.asScala.filter { case (key, _) => QuotaConfigs.isQuotaConfig(key) }
       if (quotaProps.nonEmpty && matches(userComponent, u) && matches(clientIdComponent, c))
         Some(userClientIdToEntity(u, c) -> fromProps(quotaProps))
       else
         None
-    }.flatten.toMap
+    }.toMap

Review comment:
       I am not sure to understand why you have removed `flatten` here. Could you elaborate a bit on this please?

##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -56,11 +56,12 @@ import scala.collection._
  *                          --entity-type users --entity-name <user-principal> --entity-type clients --entity-name <client-id>
  *     <li> broker: --broker <broker-id> OR --entity-type brokers --entity-name <broker-id>
  *     <li> broker-logger: --broker-logger <broker-id> OR --entity-type broker-loggers --entity-name <broker-id>
+ *     <li> ip: --IP <ip> OR --entity-type IPs --entity-name <ip>
  * </ul>
- * --entity-type <users|clients|brokers> --entity-default may be specified in place of --entity-type <users|clients|brokers> --entity-name <entityName>
- * when describing or altering default configuration for users, clients, or brokers, respectively.
- * Alternatively, --user-defaults, --client-defaults, or --broker-defaults may be specified in place of
- * --entity-type <users|clients|brokers> --entity-default, respectively.
+ * --entity-type <users|clients|brokers|IPs> --entity-default may be specified in place of --entity-type <users|clients|brokers|IPs> --entity-name <entityName>
+ * when describing or altering default configuration for users, clients, brokers, or IPs, respectively.
+ * Alternatively, --user-defaults, --client-defaults, --broker-defaults, or --IP-defaults may be specified in place of

Review comment:
       `--IP-defaults` looks pretty bad. I think that we should definitely use lower-case.

##########
File path: clients/src/main/java/org/apache/kafka/common/quota/ClientQuotaEntity.java
##########
@@ -32,6 +32,7 @@
      */
     public static final String USER = "user";
     public static final String CLIENT_ID = "client-id";
+    public static final String IP = "IP";

Review comment:
       I would also make it lower-case for consistency.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] dajac merged pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
dajac merged pull request #9628:
URL: https://github.com/apache/kafka/pull/9628


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] dajac commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r535387504



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -56,11 +56,12 @@ import scala.collection._
  *                          --entity-type users --entity-name <user-principal> --entity-type clients --entity-name <client-id>
  *     <li> broker: --broker <broker-id> OR --entity-type brokers --entity-name <broker-id>
  *     <li> broker-logger: --broker-logger <broker-id> OR --entity-type broker-loggers --entity-name <broker-id>
+ *     <li> ip: --IP <ip> OR --entity-type IPs --entity-name <ip>
  * </ul>
- * --entity-type <users|clients|brokers> --entity-default may be specified in place of --entity-type <users|clients|brokers> --entity-name <entityName>
- * when describing or altering default configuration for users, clients, or brokers, respectively.
- * Alternatively, --user-defaults, --client-defaults, or --broker-defaults may be specified in place of
- * --entity-type <users|clients|brokers> --entity-default, respectively.
+ * --entity-type <users|clients|brokers|IPs> --entity-default may be specified in place of --entity-type <users|clients|brokers|IPs> --entity-name <entityName>
+ * when describing or altering default configuration for users, clients, brokers, or IPs, respectively.
+ * Alternatively, --user-defaults, --client-defaults, --broker-defaults, or --IP-defaults may be specified in place of

Review comment:
       @splett2 Isn't the ZK path already lower-case?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] dajac commented on pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
dajac commented on pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#issuecomment-742378220


   @splett2 Thanks for your contribution!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r536391337



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -864,11 +885,21 @@ object ConfigCommand extends Config {
         }
       }
 
+      if (hasEntityName && entityTypeVals.contains(ConfigType.Ip)) {
+        Seq(entityName, ip).filter(options.has(_)).map(options.valueOf(_)).foreach { ipAddress =>
+          if (!Utils.validHostPattern(ipAddress))

Review comment:
       it looks like `max.connections.per.ip` permits hosts in addition to IPs.
   
   ```
       val invalidAddresses = maxConnectionsPerIpOverrides.keys.filterNot(address => Utils.validHostPattern(address))
       if (!invalidAddresses.isEmpty)
         throw new IllegalArgumentException(s"${KafkaConfig.MaxConnectionsPerIpOverridesProp} contains invalid addresses : ${invalidAddresses.mkString(",")}")
   ```
   Do you think we should prefer consistency with the existing IP quota prop (e.g., permit any resolvable host address) or go for more strict enforcement (only IPs permitted)?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r535397962



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -56,11 +56,12 @@ import scala.collection._
  *                          --entity-type users --entity-name <user-principal> --entity-type clients --entity-name <client-id>
  *     <li> broker: --broker <broker-id> OR --entity-type brokers --entity-name <broker-id>
  *     <li> broker-logger: --broker-logger <broker-id> OR --entity-type broker-loggers --entity-name <broker-id>
+ *     <li> ip: --IP <ip> OR --entity-type IPs --entity-name <ip>
  * </ul>
- * --entity-type <users|clients|brokers> --entity-default may be specified in place of --entity-type <users|clients|brokers> --entity-name <entityName>
- * when describing or altering default configuration for users, clients, or brokers, respectively.
- * Alternatively, --user-defaults, --client-defaults, or --broker-defaults may be specified in place of
- * --entity-type <users|clients|brokers> --entity-default, respectively.
+ * --entity-type <users|clients|brokers|IPs> --entity-default may be specified in place of --entity-type <users|clients|brokers|IPs> --entity-name <entityName>
+ * when describing or altering default configuration for users, clients, brokers, or IPs, respectively.
+ * Alternatively, --user-defaults, --client-defaults, --broker-defaults, or --IP-defaults may be specified in place of

Review comment:
       @dajac 
   it was originally lower-case in #9386
   after reading the KIP, i realized it was described as being upper case, e.g.
   ```
   Default quota for <IPs> will be stored in Zookeeper at /config/IPs/<default>
   Quota for a specific IP address for which quota override is defined will be stored in Zookeeper at /config/IPs/<ip-address>
   ```
   and changed it to upper in this PR with the change in
   ```
   core/src/main/scala/kafka/server/DynamicConfigManager.scala
   ```




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r536391337



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -864,11 +885,21 @@ object ConfigCommand extends Config {
         }
       }
 
+      if (hasEntityName && entityTypeVals.contains(ConfigType.Ip)) {
+        Seq(entityName, ip).filter(options.has(_)).map(options.valueOf(_)).foreach { ipAddress =>
+          if (!Utils.validHostPattern(ipAddress))

Review comment:
       it looks like `max.connections.per.ip` permits hosts in addition to IPs.
   
   ```
       val invalidAddresses = maxConnectionsPerIpOverrides.keys.filterNot(address => Utils.validHostPattern(address))
       if (!invalidAddresses.isEmpty)
         throw new IllegalArgumentException(s"${KafkaConfig.MaxConnectionsPerIpOverridesProp} contains invalid addresses : ${invalidAddresses.mkString(",")}")
   ```
   Do you think we should prefer consistency with the existing IP quota prop or go for more strict enforcement (only IPs permitted).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] dajac commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r537615920



##########
File path: core/src/main/scala/kafka/server/AdminManager.scala
##########
@@ -920,32 +954,51 @@ class AdminManager(val config: KafkaConfig,
         !name.isDefined || !strict
     }
 
-    def fromProps(props: Map[String, String]): Map[String, Double] = {
-      props.map { case (key, value) =>
-        val doubleValue = try value.toDouble catch {
-          case _: NumberFormatException =>
-            throw new IllegalStateException(s"Unexpected client quota configuration value: $key -> $value")
-        }
-        key -> doubleValue
-      }
-    }
-
-    (userEntries ++ clientIdEntries ++ bothEntries).map { case ((u, c), p) =>
+    (userEntries ++ clientIdEntries ++ bothEntries).flatMap { case ((u, c), p) =>
       val quotaProps = p.asScala.filter { case (key, _) => QuotaConfigs.isQuotaConfig(key) }
       if (quotaProps.nonEmpty && matches(userComponent, u) && matches(clientIdComponent, c))
         Some(userClientIdToEntity(u, c) -> fromProps(quotaProps))
       else
         None
-    }.flatten.toMap
+    }.toMap
+  }
+
+  def handleDescribeIpQuotas(ipComponent: Option[ClientQuotaFilterComponent], strict: Boolean): Map[ClientQuotaEntity, Map[String, Double]] = {
+    val ip = ipComponent.flatMap(c => toOption(c.`match`))
+    val exactIp = wantExact(ipComponent)
+    val allIps = ipComponent.exists(_.`match` == null) || (ipComponent.isEmpty && !strict)
+    val ipEntries = if (exactIp)
+      Map(Some(ip.get) -> adminZkClient.fetchEntityConfig(ConfigType.Ip, sanitized(ip)))
+    else if (allIps)
+      adminZkClient.fetchAllEntityConfigs(ConfigType.Ip).map { case (name, props) =>
+        Some(desanitizeEntityName(name)) -> props
+      }
+    else
+      Map.empty
+
+    def ipToQuotaEntity(ip: Option[String]): ClientQuotaEntity = {
+      new ClientQuotaEntity(ip.map(ipName => ClientQuotaEntity.IP -> ipName).toMap.asJava)
+    }
+
+    ipEntries.flatMap { case (ip, props) =>
+      val ipQuotaProps = props.asScala.filter { case (key, _) => DynamicConfig.Ip.names.contains(key) }
+      if (ipQuotaProps.nonEmpty)
+        Some(ipToQuotaEntity(ip) -> fromProps(ipQuotaProps))
+      else
+        None
+    }
   }
 
   def alterClientQuotas(entries: Seq[ClientQuotaAlteration], validateOnly: Boolean): Map[ClientQuotaEntity, ApiError] = {
     def alterEntityQuotas(entity: ClientQuotaEntity, ops: Iterable[ClientQuotaAlteration.Op]): Unit = {
-      val (path, configType, configKeys) = entityToSanitizedUserClientId(entity) match {
-        case (Some(user), Some(clientId)) => (user + "/clients/" + clientId, ConfigType.User, DynamicConfig.User.configKeys)
-        case (Some(user), None) => (user, ConfigType.User, DynamicConfig.User.configKeys)
-        case (None, Some(clientId)) => (clientId, ConfigType.Client, DynamicConfig.Client.configKeys)
-        case _ => throw new InvalidRequestException("Invalid empty client quota entity")
+      val (path, configType, configKeys) = parseAndSanitizeQuotaEntity(entity) match {
+        case (Some(user), Some(clientId), None) => (user + "/clients/" + clientId, ConfigType.User, DynamicConfig.User.configKeys)
+        case (Some(user), None, None) => (user, ConfigType.User, DynamicConfig.User.configKeys)
+        case (None, Some(clientId), None) => (clientId, ConfigType.Client, DynamicConfig.Client.configKeys)
+        case (None, None, Some(ip)) => (ip, ConfigType.Ip, DynamicConfig.Ip.configKeys)
+        case (_, _, Some(_)) => throw new InvalidRequestException(s"Invalid quota entity combination, " +

Review comment:
       Thanks for the explanation. That makes sense.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r536240414



##########
File path: core/src/main/scala/kafka/server/AdminManager.scala
##########
@@ -920,32 +954,51 @@ class AdminManager(val config: KafkaConfig,
         !name.isDefined || !strict
     }
 
-    def fromProps(props: Map[String, String]): Map[String, Double] = {
-      props.map { case (key, value) =>
-        val doubleValue = try value.toDouble catch {
-          case _: NumberFormatException =>
-            throw new IllegalStateException(s"Unexpected client quota configuration value: $key -> $value")
-        }
-        key -> doubleValue
-      }
-    }
-
-    (userEntries ++ clientIdEntries ++ bothEntries).map { case ((u, c), p) =>
+    (userEntries ++ clientIdEntries ++ bothEntries).flatMap { case ((u, c), p) =>
       val quotaProps = p.asScala.filter { case (key, _) => QuotaConfigs.isQuotaConfig(key) }
       if (quotaProps.nonEmpty && matches(userComponent, u) && matches(clientIdComponent, c))
         Some(userClientIdToEntity(u, c) -> fromProps(quotaProps))
       else
         None
-    }.flatten.toMap
+    }.toMap
+  }
+
+  def handleDescribeIpQuotas(ipComponent: Option[ClientQuotaFilterComponent], strict: Boolean): Map[ClientQuotaEntity, Map[String, Double]] = {
+    val ip = ipComponent.flatMap(c => toOption(c.`match`))
+    val exactIp = wantExact(ipComponent)
+    val allIps = ipComponent.exists(_.`match` == null) || (ipComponent.isEmpty && !strict)
+    val ipEntries = if (exactIp)
+      Map(Some(ip.get) -> adminZkClient.fetchEntityConfig(ConfigType.Ip, sanitized(ip)))
+    else if (allIps)
+      adminZkClient.fetchAllEntityConfigs(ConfigType.Ip).map { case (name, props) =>
+        Some(desanitizeEntityName(name)) -> props
+      }
+    else
+      Map.empty
+
+    def ipToQuotaEntity(ip: Option[String]): ClientQuotaEntity = {
+      new ClientQuotaEntity(ip.map(ipName => ClientQuotaEntity.IP -> ipName).toMap.asJava)
+    }
+
+    ipEntries.flatMap { case (ip, props) =>
+      val ipQuotaProps = props.asScala.filter { case (key, _) => DynamicConfig.Ip.names.contains(key) }
+      if (ipQuotaProps.nonEmpty)
+        Some(ipToQuotaEntity(ip) -> fromProps(ipQuotaProps))
+      else
+        None
+    }
   }
 
   def alterClientQuotas(entries: Seq[ClientQuotaAlteration], validateOnly: Boolean): Map[ClientQuotaEntity, ApiError] = {
     def alterEntityQuotas(entity: ClientQuotaEntity, ops: Iterable[ClientQuotaAlteration.Op]): Unit = {
-      val (path, configType, configKeys) = entityToSanitizedUserClientId(entity) match {
-        case (Some(user), Some(clientId)) => (user + "/clients/" + clientId, ConfigType.User, DynamicConfig.User.configKeys)
-        case (Some(user), None) => (user, ConfigType.User, DynamicConfig.User.configKeys)
-        case (None, Some(clientId)) => (clientId, ConfigType.Client, DynamicConfig.Client.configKeys)
-        case _ => throw new InvalidRequestException("Invalid empty client quota entity")
+      val (path, configType, configKeys) = parseAndSanitizeQuotaEntity(entity) match {
+        case (Some(user), Some(clientId), None) => (user + "/clients/" + clientId, ConfigType.User, DynamicConfig.User.configKeys)
+        case (Some(user), None, None) => (user, ConfigType.User, DynamicConfig.User.configKeys)
+        case (None, Some(clientId), None) => (clientId, ConfigType.Client, DynamicConfig.Client.configKeys)
+        case (None, None, Some(ip)) => (ip, ConfigType.Ip, DynamicConfig.Ip.configKeys)
+        case (_, _, Some(_)) => throw new InvalidRequestException(s"Invalid quota entity combination, " +

Review comment:
       i used `(_, _, Some(_))` for this case, because we want to throw the more verbose exception in all of these cases:
   
   `(Some(_), None, Some(_))` => specifying a user and IP, but no client
   `(None, Some, Some)` => specifying client and IP, but no user
   `(Some, Some, Some)` => specifying client, user and IP
   
   basically any time there is a client OR user specified along with IP. I felt like adding all three cases would be pretty verbose, so I relied on the ordering of the cases instead.
   




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] bdbyrne commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
bdbyrne commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r532909153



##########
File path: clients/src/main/java/org/apache/kafka/common/quota/ClientQuotaEntity.java
##########
@@ -32,6 +32,7 @@
      */
     public static final String USER = "user";
     public static final String CLIENT_ID = "client-id";
+    public static final String IP = "IP";

Review comment:
       Consider making lower-case for consistency, but otherwise not a big issue, as I see that's how it was proposed in the KIP.

##########
File path: core/src/main/scala/kafka/server/AdminManager.scala
##########
@@ -807,23 +807,25 @@ class AdminManager(val config: KafkaConfig,
       case name => Sanitizer.desanitize(name)
     }
 
-  private def entityToSanitizedUserClientId(entity: ClientQuotaEntity): (Option[String], Option[String]) = {
+  private def entityToSanitizedUserClientId(entity: ClientQuotaEntity): (Option[String], Option[String], Option[String]) = {

Review comment:
       May want to rename (generalize) since this now includes IP

##########
File path: core/src/main/scala/kafka/server/AdminManager.scala
##########
@@ -833,45 +835,76 @@ class AdminManager(val config: KafkaConfig,
   def describeClientQuotas(filter: ClientQuotaFilter): Map[ClientQuotaEntity, Map[String, Double]] = {
     var userComponent: Option[ClientQuotaFilterComponent] = None
     var clientIdComponent: Option[ClientQuotaFilterComponent] = None
+    var ipComponent: Option[ClientQuotaFilterComponent] = None
     filter.components.forEach { component =>
       component.entityType match {
         case ClientQuotaEntity.USER =>
           if (userComponent.isDefined)
-            throw new InvalidRequestException(s"Duplicate user filter component entity type");
+            throw new InvalidRequestException(s"Duplicate user filter component entity type")
           userComponent = Some(component)
         case ClientQuotaEntity.CLIENT_ID =>
           if (clientIdComponent.isDefined)
-            throw new InvalidRequestException(s"Duplicate client filter component entity type");
+            throw new InvalidRequestException(s"Duplicate client filter component entity type")
           clientIdComponent = Some(component)
+        case ClientQuotaEntity.IP =>
+          if (ipComponent.isDefined)
+            throw new InvalidRequestException(s"Duplicate ip filter component entity type")
+          ipComponent = Some(component)
         case "" =>
           throw new InvalidRequestException(s"Unexpected empty filter component entity type")
         case et =>
           // Supplying other entity types is not yet supported.
           throw new UnsupportedVersionException(s"Custom entity type '${et}' not supported")
       }
     }
-    handleDescribeClientQuotas(userComponent, clientIdComponent, filter.strict)
+    if ((userComponent.isDefined || clientIdComponent.isDefined) && ipComponent.isDefined)
+      throw new InvalidRequestException(s"Invalid entity filter component combination")

Review comment:
       May want to provide a little more detail in the response

##########
File path: core/src/test/scala/unit/kafka/utils/TestUtils.scala
##########
@@ -1529,6 +1530,16 @@ object TestUtils extends Logging {
     adminClient.incrementalAlterConfigs(configs)
   }
 
+  def alterClientQuotas(adminClient: Admin, request: Map[ClientQuotaEntity, Map[String, Option[Double]]]): AlterClientQuotasResult = {
+    val entries = request.map { case (entity, alter) =>
+      val ops = alter.map { case (key, value) =>
+        new ClientQuotaAlteration.Op(key, value.map(Double.box).getOrElse(null))
+      }.asJavaCollection
+        new ClientQuotaAlteration(entity, ops)

Review comment:
       nit: indenting




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r536241597



##########
File path: core/src/test/scala/unit/kafka/server/ClientQuotasRequestTest.scala
##########
@@ -236,13 +310,22 @@ class ClientQuotasRequestTest extends BaseRequestTest {
     (Some("user-3"), None, 58.58),
     (Some(null), None, 59.59),
     (None, Some("client-id-2"), 60.60)
-  ).map { case (u, c, v) => (toEntity(u, c), v) }
+  ).map { case (u, c, v) => (toClientEntity(u, c), v) }
+
+  private val matchIpEntities = List(
+    (Some("1.2.3.4"), 10.0),
+    (Some("2.3.4.5"), 20.0)
+  ).map { case (ip, quota) => (toIpEntity(ip), quota)}
 
   private def setupDescribeClientQuotasMatchTest() = {
-    val result = alterClientQuotas(matchEntities.map { case (e, v) =>
-      (e -> Map((RequestPercentageProp, Some(v))))
-    }.toMap, validateOnly = false)
-    matchEntities.foreach(e => result.get(e._1).get.get(10, TimeUnit.SECONDS))
+    val userClientQuotas = matchUserClientEntities.map { case (e, v) =>
+      e -> Map((RequestPercentageProp, Some(v)))
+    }.toMap
+    val ipQuotas = matchIpEntities.map { case (e, v) =>
+      e -> Map((IpConnectionRateProp, Some(v)))
+    }.toMap
+    val result = alterClientQuotas(userClientQuotas ++ ipQuotas, validateOnly = false)
+    (matchUserClientEntities ++ matchIpEntities).foreach(e => result(e._1).get(10, TimeUnit.SECONDS))
 
     // Allow time for watch callbacks to be triggered.
     Thread.sleep(500)

Review comment:
       makes sense, will do.
   Actually, taking a closer look, there is no need for the `Thread.sleep()` call. None of the tests that use the match setup rely on ZK watches to trigger.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] dajac commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r539138004



##########
File path: core/src/test/scala/unit/kafka/server/ClientQuotasRequestTest.scala
##########
@@ -223,8 +279,34 @@ class ClientQuotasRequestTest extends BaseRequestTest {
     alterEntityQuotas(entity, Map((ProducerByteRateProp -> Some(10000.5))), validateOnly = true)
   }
 
+  @Test
+  def testAlterClientQuotasInvalidEntityCombination(): Unit = {

Review comment:
       For these three tests, would it make sense to also verify that the error message? `InvalidRequestException` are thrown all over the place so we may get one but not for the expected cause. Your last commit suggests that you got caught by this already, is it? https://github.com/apache/kafka/pull/9628/commits/1b0ec2a3ca79dc8da5908c41afc4ef87d2288d84#diff-f02b619b5cd14e83f7e21dbe211b3f88336f825e1ee5c630fee32b8a0fbe3d20R294

##########
File path: core/src/main/scala/kafka/server/AdminManager.scala
##########
@@ -920,32 +954,57 @@ class AdminManager(val config: KafkaConfig,
         !name.isDefined || !strict
     }
 
-    def fromProps(props: Map[String, String]): Map[String, Double] = {
-      props.map { case (key, value) =>
-        val doubleValue = try value.toDouble catch {
-          case _: NumberFormatException =>
-            throw new IllegalStateException(s"Unexpected client quota configuration value: $key -> $value")
-        }
-        key -> doubleValue
-      }
-    }
-
-    (userEntries ++ clientIdEntries ++ bothEntries).map { case ((u, c), p) =>
+    (userEntries ++ clientIdEntries ++ bothEntries).flatMap { case ((u, c), p) =>
       val quotaProps = p.asScala.filter { case (key, _) => QuotaConfigs.isQuotaConfig(key) }
       if (quotaProps.nonEmpty && matches(userComponent, u) && matches(clientIdComponent, c))
         Some(userClientIdToEntity(u, c) -> fromProps(quotaProps))
       else
         None
-    }.flatten.toMap
+    }.toMap
+  }
+
+  def handleDescribeIpQuotas(ipComponent: Option[ClientQuotaFilterComponent], strict: Boolean): Map[ClientQuotaEntity, Map[String, Double]] = {
+    val ip = ipComponent.flatMap(c => toOption(c.`match`))
+    val exactIp = wantExact(ipComponent)
+    val allIps = ipComponent.exists(_.`match` == null) || (ipComponent.isEmpty && !strict)
+    val ipEntries = if (exactIp)
+      Map(Some(ip.get) -> adminZkClient.fetchEntityConfig(ConfigType.Ip, sanitized(ip)))
+    else if (allIps)
+      adminZkClient.fetchAllEntityConfigs(ConfigType.Ip).map { case (name, props) =>
+        Some(desanitizeEntityName(name)) -> props
+      }
+    else
+      Map.empty
+
+    def ipToQuotaEntity(ip: Option[String]): ClientQuotaEntity = {
+      new ClientQuotaEntity(ip.map(ipName => ClientQuotaEntity.IP -> ipName).toMap.asJava)
+    }
+
+    ipEntries.flatMap { case (ip, props) =>
+      val ipQuotaProps = props.asScala.filter { case (key, _) => DynamicConfig.Ip.names.contains(key) }
+      if (ipQuotaProps.nonEmpty)
+        Some(ipToQuotaEntity(ip) -> fromProps(ipQuotaProps))
+      else
+        None
+    }
   }
 
   def alterClientQuotas(entries: Seq[ClientQuotaAlteration], validateOnly: Boolean): Map[ClientQuotaEntity, ApiError] = {
     def alterEntityQuotas(entity: ClientQuotaEntity, ops: Iterable[ClientQuotaAlteration.Op]): Unit = {
-      val (path, configType, configKeys) = entityToSanitizedUserClientId(entity) match {
-        case (Some(user), Some(clientId)) => (user + "/clients/" + clientId, ConfigType.User, DynamicConfig.User.configKeys)
-        case (Some(user), None) => (user, ConfigType.User, DynamicConfig.User.configKeys)
-        case (None, Some(clientId)) => (clientId, ConfigType.Client, DynamicConfig.Client.configKeys)
-        case _ => throw new InvalidRequestException("Invalid empty client quota entity")
+      val (path, configType, configKeys) = parseAndSanitizeQuotaEntity(entity) match {
+        case (Some(user), Some(clientId), None) => (user + "/clients/" + clientId, ConfigType.User, DynamicConfig.User.configKeys)
+        case (Some(user), None, None) => (user, ConfigType.User, DynamicConfig.User.configKeys)
+        case (None, Some(clientId), None) => (clientId, ConfigType.Client, DynamicConfig.Client.configKeys)
+        case (None, None, Some(ip)) =>
+          try {
+            DynamicConfig.Ip.validateIpOrHost(ip)
+          } catch {
+            case e: IllegalArgumentException => throw new InvalidRequestException(e.getMessage)
+          }

Review comment:
       * For my understanding, have you chosen to do the validation here to ensure that requests with `validateOnly = true` are fully validated?
   * Initially, I thought that we could just catch the `IllegalArgumentException` here https://github.com/apache/kafka/blob/1b0ec2a3ca79dc8da5908c41afc4ef87d2288d84/core/src/main/scala/kafka/server/AdminManager.scala#L1043. Doing the conversion here seems reasonable as well. Alternatively, we could also change `validateIpOrHost` to return a boolean and let the caller decides which exception to throw.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r536391337



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -864,11 +885,21 @@ object ConfigCommand extends Config {
         }
       }
 
+      if (hasEntityName && entityTypeVals.contains(ConfigType.Ip)) {
+        Seq(entityName, ip).filter(options.has(_)).map(options.valueOf(_)).foreach { ipAddress =>
+          if (!Utils.validHostPattern(ipAddress))

Review comment:
       it looks like `max.connections.per.ip` permits hosts in addition to IPs.
   
   ```
       val invalidAddresses = maxConnectionsPerIpOverrides.keys.filterNot(address => Utils.validHostPattern(address))
       if (!invalidAddresses.isEmpty)
         throw new IllegalArgumentException(s"${KafkaConfig.MaxConnectionsPerIpOverridesProp} contains invalid addresses : ${invalidAddresses.mkString(",")}")
   ```
   Do you think we should go for consistency with the existing IP quota prop or go for more strict enforcement (only IPs permitted).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] apovzner commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
apovzner commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r534558407



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -56,11 +56,12 @@ import scala.collection._
  *                          --entity-type users --entity-name <user-principal> --entity-type clients --entity-name <client-id>
  *     <li> broker: --broker <broker-id> OR --entity-type brokers --entity-name <broker-id>
  *     <li> broker-logger: --broker-logger <broker-id> OR --entity-type broker-loggers --entity-name <broker-id>
+ *     <li> ip: --IP <ip> OR --entity-type IPs --entity-name <ip>
  * </ul>
- * --entity-type <users|clients|brokers> --entity-default may be specified in place of --entity-type <users|clients|brokers> --entity-name <entityName>
- * when describing or altering default configuration for users, clients, or brokers, respectively.
- * Alternatively, --user-defaults, --client-defaults, or --broker-defaults may be specified in place of
- * --entity-type <users|clients|brokers> --entity-default, respectively.
+ * --entity-type <users|clients|brokers|IPs> --entity-default may be specified in place of --entity-type <users|clients|brokers|IPs> --entity-name <entityName>
+ * when describing or altering default configuration for users, clients, brokers, or IPs, respectively.
+ * Alternatively, --user-defaults, --client-defaults, --broker-defaults, or --IP-defaults may be specified in place of

Review comment:
       yes, agreed about `--IP-defaults` looking bad. 
   
   The main reason I originally proposed upper case "IP" in the KIP is because `ips` as in `--entity-type ips` does not seem very recognizable as plural for IP (but that could be just me), i.e., not very user friendly. But if we do `--entity-type IPs` and then `--ip-defaults`, I think that would cause more issues later because people would assume it's consistent and use wrong case.
   
   Ok, I am convinced that we lower-case is better, since there are more advantages to this vs. upper-case.  




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r537945058



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -778,6 +793,10 @@ object ConfigCommand extends Config {
     val brokerLogger = parser.accepts("broker-logger", "The broker's ID for its logger config.")
       .withRequiredArg
       .ofType(classOf[String])
+    val ipDefaults = parser.accepts("ip-defaults", "The config defaults for all IPs.")
+    val ip = parser.accepts("ip", "The IP address.")

Review comment:
       this is done. while I was updating the KIP, i noticed that the previous config documentation was inconsistent with existing configs.
   
   ```
   Default connection rate quotas for an IP address can be configured by omitting entity name.
   bin/kafka-configs  --bootstrap-server localhost:9091 --alter --add-config 'connection_creation_rate=100' --entity-type IPs
   ```
   This isn't the case for any of the existing configs, you get an error:
   `an entity-name or default entity must be specified with --alter of users, clients, brokers`

##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -778,6 +793,10 @@ object ConfigCommand extends Config {
     val brokerLogger = parser.accepts("broker-logger", "The broker's ID for its logger config.")
       .withRequiredArg
       .ofType(classOf[String])
+    val ipDefaults = parser.accepts("ip-defaults", "The config defaults for all IPs.")
+    val ip = parser.accepts("ip", "The IP address.")

Review comment:
       done. while I was updating the KIP, i noticed that the previous config documentation was inconsistent with existing configs.
   
   ```
   Default connection rate quotas for an IP address can be configured by omitting entity name.
   bin/kafka-configs  --bootstrap-server localhost:9091 --alter --add-config 'connection_creation_rate=100' --entity-type IPs
   ```
   This isn't the case for any of the existing configs, you get an error:
   `an entity-name or default entity must be specified with --alter of users, clients, brokers`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] dajac commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r537615504



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -864,11 +885,21 @@ object ConfigCommand extends Config {
         }
       }
 
+      if (hasEntityName && entityTypeVals.contains(ConfigType.Ip)) {
+        Seq(entityName, ip).filter(options.has(_)).map(options.valueOf(_)).foreach { ipAddress =>
+          if (!Utils.validHostPattern(ipAddress))

Review comment:
       I think that we also permit hosts in our case as `InetAddress.getByName` resolves names that are not IP addresses. The issue is that `Utils.validHostPattern` is just a regex so it does not validate much.
   
   `max.connections.per.ip` is validated in two phases. The code that you outlines is from the `KafkaConfig`. I suppose that we do a lightweight verification there in order to avoid doing any DNS resolution while parsing the config. This is later complemented in the socket server.
   
   ```
     private[network] def updateMaxConnectionsPerIpOverride(overrideQuotas: Map[String, Int]): Unit = {
       maxConnectionsPerIpOverrides = overrideQuotas.map { case (host, count) => (InetAddress.getByName(host), count) }
     }
   ```
   
   In our case, it is important to remain consistent everywhere. 




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] dajac commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
dajac commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r535388027



##########
File path: core/src/main/scala/kafka/server/AdminManager.scala
##########
@@ -920,32 +954,49 @@ class AdminManager(val config: KafkaConfig,
         !name.isDefined || !strict
     }
 
-    def fromProps(props: Map[String, String]): Map[String, Double] = {
-      props.map { case (key, value) =>
-        val doubleValue = try value.toDouble catch {
-          case _: NumberFormatException =>
-            throw new IllegalStateException(s"Unexpected client quota configuration value: $key -> $value")
-        }
-        key -> doubleValue
-      }
-    }
-
-    (userEntries ++ clientIdEntries ++ bothEntries).map { case ((u, c), p) =>
+    (userEntries ++ clientIdEntries ++ bothEntries).flatMap { case ((u, c), p) =>
       val quotaProps = p.asScala.filter { case (key, _) => QuotaConfigs.isQuotaConfig(key) }
       if (quotaProps.nonEmpty && matches(userComponent, u) && matches(clientIdComponent, c))
         Some(userClientIdToEntity(u, c) -> fromProps(quotaProps))
       else
         None
-    }.flatten.toMap
+    }.toMap

Review comment:
       Ah. I missed the `flatMap`. That makes sense, thanks.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r537945058



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -778,6 +793,10 @@ object ConfigCommand extends Config {
     val brokerLogger = parser.accepts("broker-logger", "The broker's ID for its logger config.")
       .withRequiredArg
       .ofType(classOf[String])
+    val ipDefaults = parser.accepts("ip-defaults", "The config defaults for all IPs.")
+    val ip = parser.accepts("ip", "The IP address.")

Review comment:
       done. while I was updating the KIP, i noticed that the previous config documentation was inconsistent with existing configs.
   
   ```
   Default connection rate quotas for an IP address can be configured by omitting entity name.
   bin/kafka-configs  --bootstrap-server localhost:9091 --alter --add-config 'connection_creation_rate=100' --entity-type IPs
   ```
   This isn't the case for any of the existing configs, you get an error:
   `an entity-name or default entity must be specified with --alter of users, clients, brokers`
   so I also updated the KIP for default IP to specify --entity-default.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r536391337



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -864,11 +885,21 @@ object ConfigCommand extends Config {
         }
       }
 
+      if (hasEntityName && entityTypeVals.contains(ConfigType.Ip)) {
+        Seq(entityName, ip).filter(options.has(_)).map(options.valueOf(_)).foreach { ipAddress =>
+          if (!Utils.validHostPattern(ipAddress))

Review comment:
       it looks like `max.connections.per.ip` permits hosts in addition to IPs.
   
   ```
       val invalidAddresses = maxConnectionsPerIpOverrides.keys.filterNot(address => Utils.validHostPattern(address))
       if (!invalidAddresses.isEmpty)
         throw new IllegalArgumentException(s"${KafkaConfig.MaxConnectionsPerIpOverridesProp} contains invalid addresses : ${invalidAddresses.mkString(",")}")
   ```
   In this case, I'm not sure whether we should go for consistency with the existing IP quota prop or go for more strict enforcement (only IPs permitted).




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r539455602



##########
File path: core/src/main/scala/kafka/server/AdminManager.scala
##########
@@ -920,32 +954,57 @@ class AdminManager(val config: KafkaConfig,
         !name.isDefined || !strict
     }
 
-    def fromProps(props: Map[String, String]): Map[String, Double] = {
-      props.map { case (key, value) =>
-        val doubleValue = try value.toDouble catch {
-          case _: NumberFormatException =>
-            throw new IllegalStateException(s"Unexpected client quota configuration value: $key -> $value")
-        }
-        key -> doubleValue
-      }
-    }
-
-    (userEntries ++ clientIdEntries ++ bothEntries).map { case ((u, c), p) =>
+    (userEntries ++ clientIdEntries ++ bothEntries).flatMap { case ((u, c), p) =>
       val quotaProps = p.asScala.filter { case (key, _) => QuotaConfigs.isQuotaConfig(key) }
       if (quotaProps.nonEmpty && matches(userComponent, u) && matches(clientIdComponent, c))
         Some(userClientIdToEntity(u, c) -> fromProps(quotaProps))
       else
         None
-    }.flatten.toMap
+    }.toMap
+  }
+
+  def handleDescribeIpQuotas(ipComponent: Option[ClientQuotaFilterComponent], strict: Boolean): Map[ClientQuotaEntity, Map[String, Double]] = {
+    val ip = ipComponent.flatMap(c => toOption(c.`match`))
+    val exactIp = wantExact(ipComponent)
+    val allIps = ipComponent.exists(_.`match` == null) || (ipComponent.isEmpty && !strict)
+    val ipEntries = if (exactIp)
+      Map(Some(ip.get) -> adminZkClient.fetchEntityConfig(ConfigType.Ip, sanitized(ip)))
+    else if (allIps)
+      adminZkClient.fetchAllEntityConfigs(ConfigType.Ip).map { case (name, props) =>
+        Some(desanitizeEntityName(name)) -> props
+      }
+    else
+      Map.empty
+
+    def ipToQuotaEntity(ip: Option[String]): ClientQuotaEntity = {
+      new ClientQuotaEntity(ip.map(ipName => ClientQuotaEntity.IP -> ipName).toMap.asJava)
+    }
+
+    ipEntries.flatMap { case (ip, props) =>
+      val ipQuotaProps = props.asScala.filter { case (key, _) => DynamicConfig.Ip.names.contains(key) }
+      if (ipQuotaProps.nonEmpty)
+        Some(ipToQuotaEntity(ip) -> fromProps(ipQuotaProps))
+      else
+        None
+    }
   }
 
   def alterClientQuotas(entries: Seq[ClientQuotaAlteration], validateOnly: Boolean): Map[ClientQuotaEntity, ApiError] = {
     def alterEntityQuotas(entity: ClientQuotaEntity, ops: Iterable[ClientQuotaAlteration.Op]): Unit = {
-      val (path, configType, configKeys) = entityToSanitizedUserClientId(entity) match {
-        case (Some(user), Some(clientId)) => (user + "/clients/" + clientId, ConfigType.User, DynamicConfig.User.configKeys)
-        case (Some(user), None) => (user, ConfigType.User, DynamicConfig.User.configKeys)
-        case (None, Some(clientId)) => (clientId, ConfigType.Client, DynamicConfig.Client.configKeys)
-        case _ => throw new InvalidRequestException("Invalid empty client quota entity")
+      val (path, configType, configKeys) = parseAndSanitizeQuotaEntity(entity) match {
+        case (Some(user), Some(clientId), None) => (user + "/clients/" + clientId, ConfigType.User, DynamicConfig.User.configKeys)
+        case (Some(user), None, None) => (user, ConfigType.User, DynamicConfig.User.configKeys)
+        case (None, Some(clientId), None) => (clientId, ConfigType.Client, DynamicConfig.Client.configKeys)
+        case (None, None, Some(ip)) =>
+          try {
+            DynamicConfig.Ip.validateIpOrHost(ip)
+          } catch {
+            case e: IllegalArgumentException => throw new InvalidRequestException(e.getMessage)
+          }

Review comment:
       @dajac 
   yes, I thought that catching in that block would work, but after taking a closer look, it seems like entity validation is only done when altering, so we wouldn't hit validation in the `validateOnly = true` case.
   
   I think changing `validateIpOrHost` to return a boolean is reasonable. I originally wanted to have it throw exception instead of boolean since we had two validation steps (`validHostPattern` and then resolvable IP), but since we simplified that code block to only do host resolution, it makes more sense to have it just return a boolean.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] apovzner commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
apovzner commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r535829042



##########
File path: core/src/test/scala/integration/kafka/network/DynamicConnectionQuotaTest.scala
##########
@@ -259,13 +264,19 @@ class DynamicConnectionQuotaTest extends BaseRequestTest {
   }
 
   private def updateIpConnectionRate(ip: Option[String], updatedRate: Int): Unit = {
-    adminZkClient.changeIpConfig(ip.getOrElse(ConfigEntityName.Default),
-      CoreUtils.propsWith(DynamicConfig.Ip.IpConnectionRateOverrideProp, updatedRate.toString))
+    val initialConnectionCount = connectionCount
+    val adminClient = createAdminClient()

Review comment:
       `TestUtils.alterClientQuotas` called before `adminClient.close` can throw an exception. Admin extends AutoCloseable, so you can use try-with-resources.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [kafka] splett2 commented on a change in pull request #9628: KAFKA-10747: Implement APIs for altering and describing IP connection rate quotas

Posted by GitBox <gi...@apache.org>.
splett2 commented on a change in pull request #9628:
URL: https://github.com/apache/kafka/pull/9628#discussion_r536250843



##########
File path: core/src/main/scala/kafka/admin/ConfigCommand.scala
##########
@@ -864,11 +885,21 @@ object ConfigCommand extends Config {
         }
       }
 
+      if (hasEntityName && entityTypeVals.contains(ConfigType.Ip)) {
+        Seq(entityName, ip).filter(options.has(_)).map(options.valueOf(_)).foreach { ipAddress =>

Review comment:
       unexpectedly, I get a compile error if I remove `(_)` in `map`.
   
   ```
   [Error]~/kafka/core/src/main/scala/kafka/admin/ConfigCommand.scala:889: type mismatch;
    found   : [V](x$1: joptsimple.OptionSpec[V]): V <and> (x$1: String): Object
    required: joptsimple.ArgumentAcceptingOptionSpec[String] => ?
   ```
   I will keep it as is for both cases to avoid confusion, e.g. why do we use a function reference in `filter`, but not `map`




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org