You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "FrankYang0529 (via GitHub)" <gi...@apache.org> on 2025/01/04 13:41:26 UTC

[PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

FrankYang0529 opened a new pull request, #18387:
URL: https://github.com/apache/kafka/pull/18387

   
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "jsancio (via GitHub)" <gi...@apache.org>.
jsancio commented on code in PR #18387:
URL: https://github.com/apache/kafka/pull/18387#discussion_r1942018528


##########
core/src/main/scala/kafka/server/KafkaConfig.scala:
##########
@@ -560,7 +560,17 @@ class KafkaConfig private(doLog: Boolean, val props: util.Map[_, _])
     controllerListenerNames.flatMap { name =>
       controllerAdvertisedListeners
         .find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name)))
-        .orElse(controllerListenersValue.find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name))))
+        .orElse(
+          // If users don't define advertised.listeners, the advertised controller listeners inherit from listeners configuration
+          // which match listener names in controller.listener.names. Also, removing "0.0.0.0" host to avoid validation errors.
+          controllerListenersValue
+            .find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name)))
+            .map(endpoint => if (endpoint.host == "0.0.0.0") {
+              new EndPoint(null, endpoint.port, endpoint.listenerName, endpoint.securityProtocol)

Review Comment:
   I think that `null` or `""` works in some network because in the `ControllerServer` code we do a reverse lookup.
   ```scala
           val listenerInfo = ListenerInfo
             .create(config.effectiveAdvertisedControllerListeners.map(_.toJava).asJava)
             .withWildcardHostnamesResolved()
             .withEphemeralPortsCorrected(name => socketServer.boundPort(new ListenerName(name)))
   ```
   Notice the use of `withWildcardHostnamesResolved`. Can you document this in the comment above?
   
   I should also mentioned that `0.0.0.0` is IPv4. Kafka also supports IPv6 which is encoded as `::`.



##########
core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala:
##########
@@ -1278,6 +1278,44 @@ class KafkaConfigTest {
     KafkaConfig.fromProps(props)
   }
 
+  @Test
+  def testImplicitAllBindingControllerListenersCanBeAdvertisedForKRaftCombined(): Unit = {
+    val props = new Properties()
+    props.setProperty(KRaftConfigs.PROCESS_ROLES_CONFIG, "controller")

Review Comment:
   The name of the method doesn't look correct since the process role is just controller.



##########
core/src/main/scala/kafka/server/KafkaConfig.scala:
##########
@@ -575,7 +585,8 @@ class KafkaConfig private(doLog: Boolean, val props: util.Map[_, _])
     if (advertisedListenersProp != null) {
       CoreUtils.listenerListToEndPoints(advertisedListenersProp, effectiveListenerSecurityProtocolMap, requireDistinctPorts=false)
     } else {
-      listeners
+      // Only use implicit broker listeners here. Implicit controller listeners are handled in effectiveAdvertisedControllerListeners.
+      listeners.filterNot(l => controllerListenerNames.contains(l.listenerName.value()))

Review Comment:
   Do you need this? `advertisedListeners` is a private method that contains all of the advertise listeners if it is set or listeners if it is not set. `effectiveAdvertisedBrokerListeners` and `effectiveAdvertisedControllerListeners` do the necessary filtering.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "ahuang98 (via GitHub)" <gi...@apache.org>.
ahuang98 commented on code in PR #18387:
URL: https://github.com/apache/kafka/pull/18387#discussion_r1913714254


##########
core/src/main/scala/kafka/server/KafkaConfig.scala:
##########
@@ -559,7 +559,16 @@ class KafkaConfig private(doLog: Boolean, val props: util.Map[_, _])
     controllerListenerNames.flatMap { name =>
       controllerAdvertisedListeners
         .find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name)))
-        .orElse(controllerListenersValue.find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name))))
+        .orElse(
+          // implicit advertised controller listener

Review Comment:
   nit: how do you feel about making the comment more verbose to explain why this conversion is needed?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "ahuang98 (via GitHub)" <gi...@apache.org>.
ahuang98 commented on code in PR #18387:
URL: https://github.com/apache/kafka/pull/18387#discussion_r1924087587


##########
core/src/main/scala/kafka/server/KafkaConfig.scala:
##########
@@ -559,7 +559,16 @@ class KafkaConfig private(doLog: Boolean, val props: util.Map[_, _])
     controllerListenerNames.flatMap { name =>
       controllerAdvertisedListeners
         .find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name)))
-        .orElse(controllerListenersValue.find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name))))
+        .orElse(
+          // implicit advertised controller listener

Review Comment:
   That's a bit what my question was (why can't we relax that rule), is it because we can't/shouldn't change the documentation for 3.9 or it's actually incorrect to do so for some reason I'm missing



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #18387:
URL: https://github.com/apache/kafka/pull/18387#discussion_r1948343697


##########
core/src/main/scala/kafka/server/KafkaConfig.scala:
##########
@@ -575,7 +585,8 @@ class KafkaConfig private(doLog: Boolean, val props: util.Map[_, _])
     if (advertisedListenersProp != null) {
       CoreUtils.listenerListToEndPoints(advertisedListenersProp, effectiveListenerSecurityProtocolMap, requireDistinctPorts=false)
     } else {
-      listeners
+      // Only use implicit broker listeners here. Implicit controller listeners are handled in effectiveAdvertisedControllerListeners.
+      listeners.filterNot(l => controllerListenerNames.contains(l.listenerName.value()))

Review Comment:
   Yes, removed `advertisedListeners`. Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "jsancio (via GitHub)" <gi...@apache.org>.
jsancio commented on PR #18387:
URL: https://github.com/apache/kafka/pull/18387#issuecomment-2680243271

   Build looks green here: https://github.com/apache/kafka/actions/runs/13500445410


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #18387:
URL: https://github.com/apache/kafka/pull/18387#discussion_r1919474587


##########
core/src/main/scala/kafka/server/KafkaConfig.scala:
##########
@@ -559,7 +559,16 @@ class KafkaConfig private(doLog: Boolean, val props: util.Map[_, _])
     controllerListenerNames.flatMap { name =>
       controllerAdvertisedListeners
         .find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name)))
-        .orElse(controllerListenersValue.find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name))))
+        .orElse(
+          // implicit advertised controller listener

Review Comment:
   > would it have made sense to remove the condition that effectiveAdvertisedControllerListeners cannot be 0.0.0.0
   
   Hi @ahuang98, thanks for the review. From the [document](https://kafka.apache.org/documentation/#brokerconfigs_advertised.listeners), it says `advertised.listeners` can't have `0.0.0.0` host name. I think we still need to keep the check to avoid users define `advertised.listeners` with `0.0.0.0` explicitly. 
   
   > have withWildcardHostnamesResolved account for the 0.0.0.0 case?
   
   My change here to remove `0.0.0.0` from implicit advertised controller listeners make `withWildcardHostnamesResolved` work for them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #18387:
URL: https://github.com/apache/kafka/pull/18387#discussion_r1944479557


##########
core/src/main/scala/kafka/server/KafkaConfig.scala:
##########
@@ -575,7 +585,8 @@ class KafkaConfig private(doLog: Boolean, val props: util.Map[_, _])
     if (advertisedListenersProp != null) {
       CoreUtils.listenerListToEndPoints(advertisedListenersProp, effectiveListenerSecurityProtocolMap, requireDistinctPorts=false)
     } else {
-      listeners
+      // Only use implicit broker listeners here. Implicit controller listeners are handled in effectiveAdvertisedControllerListeners.
+      listeners.filterNot(l => controllerListenerNames.contains(l.listenerName.value()))

Review Comment:
   Yes, if we return `listeners` directly, we can't know whether `advertisedListeners` is defined by user or implicit inherit from `listeners` in `effectiveAdvertisedControllerListeners`. If it's from user definition, we should return error. This case is included in `testExplicitAllBindingControllerListenersCanBeAdvertisedForKRaftController`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "jsancio (via GitHub)" <gi...@apache.org>.
jsancio merged PR #18387:
URL: https://github.com/apache/kafka/pull/18387


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "jsancio (via GitHub)" <gi...@apache.org>.
jsancio commented on code in PR #18387:
URL: https://github.com/apache/kafka/pull/18387#discussion_r1966025315


##########
core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala:
##########
@@ -1230,6 +1230,44 @@ class KafkaConfigTest {
     KafkaConfig.fromProps(props)
   }
 
+  @Test
+  def testImplicitAllBindingControllerListenersCanBeAdvertisedForKRaftController(): Unit = {
+    val props = new Properties()
+    props.setProperty(KRaftConfigs.PROCESS_ROLES_CONFIG, "controller")
+    val listeners = "CONTROLLER://:9093"
+    props.setProperty(SocketServerConfigs.LISTENERS_CONFIG, listeners)
+    props.setProperty(SocketServerConfigs.ADVERTISED_LISTENERS_CONFIG, listeners) // explicitly setting it in KRaft

Review Comment:
   Okay. Can you confirm that this is valid for the broker's advertise listener? Ideally advertised listener should not contain unroutable hosts.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #18387:
URL: https://github.com/apache/kafka/pull/18387#discussion_r1944480684


##########
core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala:
##########
@@ -1278,6 +1278,44 @@ class KafkaConfigTest {
     KafkaConfig.fromProps(props)
   }
 
+  @Test
+  def testImplicitAllBindingControllerListenersCanBeAdvertisedForKRaftCombined(): Unit = {
+    val props = new Properties()
+    props.setProperty(KRaftConfigs.PROCESS_ROLES_CONFIG, "controller")

Review Comment:
   Updated case name. Thanks.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on PR #18387:
URL: https://github.com/apache/kafka/pull/18387#issuecomment-2665046858

   Hi @jsancio, I addressed all comments. Could you help me review again when you have time? Thank you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #18387:
URL: https://github.com/apache/kafka/pull/18387#discussion_r1967862812


##########
core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala:
##########
@@ -1230,6 +1230,44 @@ class KafkaConfigTest {
     KafkaConfig.fromProps(props)
   }
 
+  @Test
+  def testImplicitAllBindingControllerListenersCanBeAdvertisedForKRaftController(): Unit = {
+    val props = new Properties()
+    props.setProperty(KRaftConfigs.PROCESS_ROLES_CONFIG, "controller")
+    val listeners = "CONTROLLER://:9093"
+    props.setProperty(SocketServerConfigs.LISTENERS_CONFIG, listeners)
+    props.setProperty(SocketServerConfigs.ADVERTISED_LISTENERS_CONFIG, listeners) // explicitly setting it in KRaft

Review Comment:
   @jsancio, thanks for the review. I add two cases for broker:
   
   * testImplicitAllBindingListenersCanBeAdvertisedForBroker
   * testExplicitAllBindingListenersCannotBeUsedForBroker



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "FrankYang0529 (via GitHub)" <gi...@apache.org>.
FrankYang0529 commented on code in PR #18387:
URL: https://github.com/apache/kafka/pull/18387#discussion_r1944480119


##########
core/src/main/scala/kafka/server/KafkaConfig.scala:
##########
@@ -560,7 +560,17 @@ class KafkaConfig private(doLog: Boolean, val props: util.Map[_, _])
     controllerListenerNames.flatMap { name =>
       controllerAdvertisedListeners
         .find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name)))
-        .orElse(controllerListenersValue.find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name))))
+        .orElse(
+          // If users don't define advertised.listeners, the advertised controller listeners inherit from listeners configuration
+          // which match listener names in controller.listener.names. Also, removing "0.0.0.0" host to avoid validation errors.
+          controllerListenersValue
+            .find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name)))
+            .map(endpoint => if (endpoint.host == "0.0.0.0") {
+              new EndPoint(null, endpoint.port, endpoint.listenerName, endpoint.securityProtocol)

Review Comment:
   Add related comments. Thanks for the suggestion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "jsancio (via GitHub)" <gi...@apache.org>.
jsancio commented on code in PR #18387:
URL: https://github.com/apache/kafka/pull/18387#discussion_r1945545749


##########
core/src/main/scala/kafka/server/KafkaConfig.scala:
##########
@@ -575,7 +585,8 @@ class KafkaConfig private(doLog: Boolean, val props: util.Map[_, _])
     if (advertisedListenersProp != null) {
       CoreUtils.listenerListToEndPoints(advertisedListenersProp, effectiveListenerSecurityProtocolMap, requireDistinctPorts=false)
     } else {
-      listeners
+      // Only use implicit broker listeners here. Implicit controller listeners are handled in effectiveAdvertisedControllerListeners.
+      listeners.filterNot(l => controllerListenerNames.contains(l.listenerName.value()))

Review Comment:
   Okay. It looks like the `advertisedListeners` method is not as useful as I originally thought when I implemented it. If you agree, let's just remove the method and inline the different implementations in `effectiveAdvertisedBrokerListeners` and `effectiveAdvertisedControllerListeners`. What do you think?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-18281: Kafka (3.9.0) is improperly validating non-advertised listeners for routable controller addresses [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #18387:
URL: https://github.com/apache/kafka/pull/18387#discussion_r2011523341


##########
core/src/main/scala/kafka/server/KafkaConfig.scala:
##########
@@ -519,29 +519,44 @@ class KafkaConfig private(doLog: Boolean, val props: util.Map[_, _])
   }
 
   def effectiveAdvertisedControllerListeners: Seq[EndPoint] = {
-    val controllerAdvertisedListeners = advertisedListeners.filter(l => controllerListenerNames.contains(l.listenerName.value()))
+    val advertisedListenersProp = getString(SocketServerConfigs.ADVERTISED_LISTENERS_CONFIG)
+    val controllerAdvertisedListeners = if (advertisedListenersProp != null) {
+      CoreUtils.listenerListToEndPoints(advertisedListenersProp, effectiveListenerSecurityProtocolMap, requireDistinctPorts=false)
+        .filter(l => controllerListenerNames.contains(l.listenerName.value()))
+    } else {
+      Seq.empty
+    }
     val controllerListenersValue = controllerListeners
 
     controllerListenerNames.flatMap { name =>
       controllerAdvertisedListeners
         .find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name)))
-        .orElse(controllerListenersValue.find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name))))
+        .orElse(
+          // If users don't define advertised.listeners, the advertised controller listeners inherit from listeners configuration
+          // which match listener names in controller.listener.names.
+          // Removing "0.0.0.0" host to avoid validation errors. This is to be compatible with the old behavior before 3.9.
+          // The null or "" host does a reverse lookup in ListenerInfo#withWildcardHostnamesResolved.
+          controllerListenersValue
+            .find(endpoint => endpoint.listenerName.equals(ListenerName.normalised(name)))
+            .map(endpoint => if (endpoint.host == "0.0.0.0") {

Review Comment:
   @jsancio @FrankYang0529 Pardon me, should we apply this to broker listener (`effectiveAdvertisedBrokerListeners`) for consistency? 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org