You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Pavel Zeger (Jira)" <ji...@apache.org> on 2026/06/05 17:00:00 UTC
[jira] [Updated] (KAFKA-20644) Connect’s RestServer assumes admin.listeners differ from listeners, silently orphaning admin resources when they match
[ https://issues.apache.org/jira/browse/KAFKA-20644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pavel Zeger updated KAFKA-20644:
--------------------------------
Description:
The Kafka Connect REST server lets operators expose admin endpoints on a separate listener via admin.listeners. RestServer handles
three intended cases:
1. admin.listeners unset (null) → admin resources are served on the main listeners.
2. admin.listeners set to an empty list → admin resources are disabled.
3. admin.listeners set to values distinct from listeners → admin resources are served on a separate Jetty connector.
It does not handle a fourth case: when admin.listeners overlaps listeners (shares a host:port). The code explicitly assumes the two
are different — see the TODOs in RestServer.initializeResources:
// connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java:245-246
// TODO: we need to check if these listeners are same as 'listeners'
// TODO: the following code assumes that they are different
There is no cross-validation of the two configs. admin.listeners is validated only by anyNonDuplicateValues
(RestServerConfig.java:148-153, checks for duplicates within itself), and listeners uses ListenersValidator
(RestServerConfig.java:166, validates URL format only). Neither compares the two lists.
What happens with overlapping config
Example:
listeners=http://0.0.0.0:8083
admin.listeners=http://0.0.0.0:8083
RestServer.createConnectors (RestServer.java:120-138) unconditionally creates a Jetty ServerConnector per main listener and a
separate admin ServerConnector per admin listener. With the config above this produces two connectors bound to the identical
host:port — a main connector named http_0.0.0.08083 (RestServer.java:180) and an admin connector named Admin (RestServer.java:185).
initializeResources then binds the admin context to the @Admin connector via virtual host (RestServer.java:270).
Because nothing enables SO_REUSEPORT, two listening sockets on the same host:port collide. The expected result is that the second
bind() fails at jettyServer.start() (RestServer.java:215) with BindException: Address already in use, which initializeServer wraps
as ConnectException("Unable to initialize REST server") — i.e. the worker fails to start, with an error message that gives no hint
the real cause is admin.listeners overlapping listeners.
(Note: the precise runtime manifestation — startup bind failure vs. a running server with broken admin endpoints — should be
confirmed with a reproduction test as part of the fix. Code analysis points to a startup BindException. The earlier "404 / orphaned
resources" framing appears inaccurate: the admin connector is created; it's the bind that conflicts.)
Either way, a plausible misconfiguration (a user wanting admin endpoints on the main port may set the two equal, not realizing that
leaving admin.listeners unset already does exactly that) yields a confusing failure with no actionable message.
Proposed solutions
- Option A: treat an overlapping/identical admin.listeners as unset (inherit admin resources onto the main listeners).
- Option B (recommended): fail fast. Validate early (before any connector is built) that no admin.listeners entry shares a host:port
with any listeners entry, and throw a ConfigException directing the user to use distinct host:port(s) or to leave admin.listeners
unset to share the main listener.
Option B is recommended because it is explicit, naturally covers partial overlap (e.g. listeners=...:8083,...:9083 with
admin.listeners=...:8083), and avoids silently reinterpreting a non-null config as null. The unset/null path already provides Option
A's intended "admin on the main listener" behavior.
was:
The Connect REST server supports separating admin endpoints (/loggers, etc.) onto a different listener via admin.listeners. The intended cases:
> admin.listeners not set (null) → admin resources go on the main listeners.
> admin.listeners set to empty list → admin resources disabled.
> admin.listeners set to different values from listeners → admin resources go on a separate connector.
The TODOs flag the missing case:
> admin.listeners set to the same values as listeners.
Today the code enters the else \{ else{ ... }} branch, creates a brand-new ResourceConfig for admin resources, and proceeds - but the new ResourceConfig is bound to a Jetty connector that won’t be created (because listeners and admin.listeners resolve to the same Jetty connector). The admin resources become orphaned. Operators see /loggers return 404 on the listener they configured for both regular and admin traffic.
Reproduction:
{code:java}
# connect-distributed.properties
listeners=http://0.0.0.0:8083
admin.listeners=http://0.0.0.0:8083 {code}
It’s a low-stakes but deeply confusing failure mode.
The operator: Reads docs that say admin.listeners exposes admin resources on a separate listener. Configures them to the same value (e.g. because they want the admin features but don’t want a separate port). Hits 404 with no clear error.
Either case should be handled gracefully: If the user wants admin resources on the same port, set admin.listeners to null (or to the same string and have the code recognise it). The current code’s silent orphaning is the worst possible outcome.
*Proposed fix*
# Option A: treat matching listeners as null
# Option B: fail fast
*KIP needed?*
No - edge case for already broken flow by a design
> Connect’s RestServer assumes admin.listeners differ from listeners, silently orphaning admin resources when they match
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-20644
> URL: https://issues.apache.org/jira/browse/KAFKA-20644
> Project: Kafka
> Issue Type: Bug
> Components: connect
> Reporter: Pavel Zeger
> Assignee: Pavel Zeger
> Priority: Major
>
> The Kafka Connect REST server lets operators expose admin endpoints on a separate listener via admin.listeners. RestServer handles
> three intended cases:
> 1. admin.listeners unset (null) → admin resources are served on the main listeners.
> 2. admin.listeners set to an empty list → admin resources are disabled.
> 3. admin.listeners set to values distinct from listeners → admin resources are served on a separate Jetty connector.
> It does not handle a fourth case: when admin.listeners overlaps listeners (shares a host:port). The code explicitly assumes the two
> are different — see the TODOs in RestServer.initializeResources:
> // connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/RestServer.java:245-246
> // TODO: we need to check if these listeners are same as 'listeners'
> // TODO: the following code assumes that they are different
> There is no cross-validation of the two configs. admin.listeners is validated only by anyNonDuplicateValues
> (RestServerConfig.java:148-153, checks for duplicates within itself), and listeners uses ListenersValidator
> (RestServerConfig.java:166, validates URL format only). Neither compares the two lists.
>
> What happens with overlapping config
> Example:
> listeners=http://0.0.0.0:8083
> admin.listeners=http://0.0.0.0:8083
>
> RestServer.createConnectors (RestServer.java:120-138) unconditionally creates a Jetty ServerConnector per main listener and a
> separate admin ServerConnector per admin listener. With the config above this produces two connectors bound to the identical
> host:port — a main connector named http_0.0.0.08083 (RestServer.java:180) and an admin connector named Admin (RestServer.java:185).
> initializeResources then binds the admin context to the @Admin connector via virtual host (RestServer.java:270).
>
> Because nothing enables SO_REUSEPORT, two listening sockets on the same host:port collide. The expected result is that the second
> bind() fails at jettyServer.start() (RestServer.java:215) with BindException: Address already in use, which initializeServer wraps
> as ConnectException("Unable to initialize REST server") — i.e. the worker fails to start, with an error message that gives no hint
> the real cause is admin.listeners overlapping listeners.
> (Note: the precise runtime manifestation — startup bind failure vs. a running server with broken admin endpoints — should be
> confirmed with a reproduction test as part of the fix. Code analysis points to a startup BindException. The earlier "404 / orphaned
> resources" framing appears inaccurate: the admin connector is created; it's the bind that conflicts.)
>
> Either way, a plausible misconfiguration (a user wanting admin endpoints on the main port may set the two equal, not realizing that
> leaving admin.listeners unset already does exactly that) yields a confusing failure with no actionable message.
> Proposed solutions
> - Option A: treat an overlapping/identical admin.listeners as unset (inherit admin resources onto the main listeners).
> - Option B (recommended): fail fast. Validate early (before any connector is built) that no admin.listeners entry shares a host:port
> with any listeners entry, and throw a ConfigException directing the user to use distinct host:port(s) or to leave admin.listeners
> unset to share the main listener.
> Option B is recommended because it is explicit, naturally covers partial overlap (e.g. listeners=...:8083,...:9083 with
> admin.listeners=...:8083), and avoids silently reinterpreting a non-null config as null. The unset/null path already provides Option
> A's intended "admin on the main listener" behavior.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)