[jira] [Resolved] (KAFKA-18281) Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses

["José Armando García Sancio (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-18281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

José Armando García Sancio resolved KAFKA-18281.
------------------------------------------------
    Resolution: Fixed

> Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses
> -------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-18281
>                 URL: https://issues.apache.org/jira/browse/KAFKA-18281
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 3.9.0
>            Reporter: Justin Lee
>            Assignee: PoAn Yang
>            Priority: Blocker
>             Fix For: 4.0.0, 3.9.1
>
>
> Kafka 3.9.0 is currently validating whether all listeners (including non-advertised controller listeners) are routable. Non-advertised controller listeners don't need to be routable.
> *Background:*
> [KIP-853 (KRaft Controller Membership Changes)|https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Controller+Membership+Changes] introduced dynamic controller quorums.
> When a cluster is configured with a dynamic controller quorum, KRaft clients request (via a metadata request) the currently-valid set of controller endpoints, which are configured on a per-controller basis via `advertised.listeners` (previously, controller listeners were prohibited in `advertised.listeners`).
> As part of [this work|https://github.com/confluentinc/ce-kafka/blob/78066ca74e7964bc7f9dfb0f7660814ee91ef149/core/src/main/scala/kafka/server/KafkaConfig.scala#L3445], Kafka added two things:
>  * When a controller listener is not in `advertised.listeners`, the (non-advertised) controller listener from `listeners` is added to `effectiveAdvertisedBrokerListeners`
>  * We validate that all effective advertised broker listeners are routable (i.e. not `0.0.0.0`)
> This is incorrect logic, because the listeners property indicates what IPs a listener binds on, not how clients should connect to the listener (which may be a static configuration that is transparent to the controller)
>  
> In 3.8, both of these config combinations are valid:
> _Implicit "bind on all IP addresses"_
> {code:java}
> controller.quorum.voters=9991@localhost:9094
> listeners=BROKER://:9093,CONTROLLER://:9094
> advertised.listeners=BROKER://hostname:9093{code}
> _Explicit "bind on all addresses" (specifically, IPv4)_
> {code:java}
> controller.quorum.voters=9991@localhost:9094
> listeners=BROKER://0.0.0.0:9093,CONTROLLER://0.0.0.0:9094
> advertised.listeners=BROKER://<hostname>:9093{code}
> Formatting a controller with this configuration fails with an error like this:
> {code:java}
> broker | Exception in thread "main" java.lang.IllegalArgumentException: requirement failed: advertised.listeners cannot use the nonroutable meta-address 0.0.0.0. Use a routable IP address. at scala.Predef$.require(Predef.scala:337) at kafka.server.KafkaConfig.validateValues(KafkaConfig.scala:1022) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:852) at kafka.server.KafkaConfig.<init>(KafkaConfig.scala:184) at kafka.tools.StorageTool$.$anonfun$execute$1(StorageTool.scala:79) at scala.Option.flatMap(Option.scala:283) at kafka.tools.StorageTool$.execute(StorageTool.scala:79) at kafka.tools.StorageTool$.main(StorageTool.scala:46) at kafka.docker.KafkaDockerWrapper$.main(KafkaDockerWrapper.scala:48) at kafka.docker.KafkaDockerWrapper.main(KafkaDockerWrapper.scala) at java.base@21.0.2/java.lang.invoke.LambdaForm$DMH/sa346b79c.invokeStaticInit(LambdaForm$DMH) {code}
> In 3.9.0, the implicit configuration works, but the explicit configuration does not. This is a breaking change because a configuration that previously worked no longer works.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)