You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/09/23 22:22:29 UTC
[GitHub] [kafka] hachikuji opened a new pull request, #12682: MINOR: Add section on listener configuration (including kraft) to security docs
hachikuji opened a new pull request, #12682:
URL: https://github.com/apache/kafka/pull/12682
This patch adds a section in security.html about basic listener configuration. This includes the basics of how to define the security mapping of each listener as well as the configurations to control inter-cluster traffic.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on a diff in pull request #12682: MINOR: Add section on listener configuration (including kraft) to security docs
Posted by GitBox <gi...@apache.org>.
hachikuji commented on code in PR #12682:
URL: https://github.com/apache/kafka/pull/12682#discussion_r981453929
##########
docs/security.html:
##########
@@ -36,7 +36,136 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each server must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka servers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each server. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage of each listener clearer.</p>
+
+ <p>Among the listeners in this list, it is possible to declare the listener to be used for
+ inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration
+ to the name of the listener. The primary purpose of the inter-broker listener is
+ partition replication. If not defined, then the inter-broker listener is determined
+ by the security protocol defined by <code>security.inter.broker.protocol</code>, which
+ defaults to <code>PLAINTEXT</code>.</p>
+
+ <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to
+ declare a separate listener to be used for metadata propagation from the active controller
+ to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller
+ will use this listener when it needs to push metadata updates to the brokers in the cluster.
+ The benefit of using a control plane listener is that it uses a separate processing thread,
+ which makes it less likely for application traffic to impede timely propagation of metadata changes
+ (such as partition leader and ISR updates). Note that the default value is null, which
+ means that the controller will use the same listener defined by <code>inter.broker.listener</code></p>
+
+ <p>In a KRaft cluster, a broker is any server which has the <code>broker</code> role enabled
+ in <code>process.roles</code> and a controller is any server which has the <code>controller</code>
+ role enabled. Listener configuration depends on the role. The listener defined by
+ <code>inter.broker.listener.name</code> is used exclusively for requests between brokers.
+ Controllers, on the other hand, must use separate listener which is defined by the
+ <code>controller.listener.names</code> configuration. This cannot be set to the same
+ value as the inter-broker listener.</p>
+
+ <p>Controllers receive requests both from other controllers and from brokers. For
+ this reason, even if a server does not have the <code>controller</code> role enabled
+ (i.e. it is just a broker), it must still define the controller listener along with
+ any security properties that are needed to configure it. For example, we might
+ use the following configuration on a standalone broker:</p>
+
+ <pre class="line-numbers"><code class="language-text">process.roles=broker
+listeners=BROKER://localhost:9092
+inter.broker.listener.name=BROKER
+controller.quorum.voters=0@localhost:9093
+controller.listener.names=CONTROLLER
+listener.security.protocol.map=BROKER:SASL_SSL,CONTROLLER:SASL_SSL</code></pre>
+
+ <p>The controller listener is still configured in this example to use the <code>SASL_SSL</code>
+ security protocol, but it is not included in <code>listeners</code> since the broker
+ does not expose the controller listener itself. The port that will be used in this case
+ comes from the <code>controller.quorum.voters</code> configuration, which defines
+ the complete list of controllers.</p>
+
+ <p>For KRaft servers which have both the broker and controller role enabled, the configuration
+ is similar. The only difference is that the controller listener must be included in
+ <code>listeners</code>:</p>
+
+ <pre class="line-numbers"><code class="language-text">process.roles=broker,controller
+listeners=BROKER://localhost:9092,CONTROLLER://localhost:9093
+inter.broker.listener.name=BROKER
+controller.quorum.voters=0@localhost:9093
+controller.listener.names=CONTROLLER
+listener.security.protocol.map=BROKER:SASL_SSL,CONTROLLER:SASL_SSL</code></pre>
+
+ <p>It is a requirement for the port defined in <code>controller.quorum.voters</code> to
+ exactly match one of the exposed controller listeners. For example, here the
+ <code>CONTROLLER</code> listener is bound to port 9093. The connection string
+ defined by <code>controller.quorum.voters</code> must then also use port 9093,
+ as it does here.</p>
+
+ <p>The controller will accept requests on all listeners defined by <code>controller.listener.names</code>.
+ Typically there would be just one controller listener, but it is possible to have more.
+ For example, this provides a way to change the active listener from one port or security
+ protocol to another through a roll of the cluster (one roll to expose the new listener,
+ and one roll to remove the old listener). When multiple controller listeners are defined,
+ the first one in the list will be used for outbound requests.</p>
+
+ <p>It is conventional in Kafka to use a separate listener for clients. This allows the
+ inter-cluster listeners to be isolated at the network level. In the case of the controller
+ listener in KRaft, the listener should be isolated since clients do not work with it
+ anyway. Clients are expected to connect to any other listener configured on a broker.
+ Any requests that are bound for the controller will be forwarded as described
+ <a href="#kraft_principal_forwarding">below</a></p>
Review Comment:
I added the principal forwarding section in a previous patch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on pull request #12682: MINOR: Add section on listener configuration (including kraft) to security docs
Posted by GitBox <gi...@apache.org>.
hachikuji commented on PR #12682:
URL: https://github.com/apache/kafka/pull/12682#issuecomment-1258878768
@showuon I'd appreciate one more look before we merge if you have time. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon merged pull request #12682: MINOR: Add section on listener configuration (including kraft) to security docs
Posted by GitBox <gi...@apache.org>.
showuon merged PR #12682:
URL: https://github.com/apache/kafka/pull/12682
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jsancio commented on a diff in pull request #12682: MINOR: Add section on listener configuration (including kraft) to security docs
Posted by GitBox <gi...@apache.org>.
jsancio commented on code in PR #12682:
URL: https://github.com/apache/kafka/pull/12682#discussion_r980449119
##########
docs/security.html:
##########
@@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
Review Comment:
```suggestion
<p>Kafka servers support listening for connections on multiple ports. This is configured through
the <code>listeners</code> property in the server configuration, which accepts a comma-separated
list of the listeners to enable. At least one listener must be defined on each server. The format
of each listener defined in <code>listeners</code> is given below:</p>
```
##########
docs/security.html:
##########
@@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage clearer.</p>
+
+ <p>Among the listeners in this list, it is possible to declare the listener to be used for
+ inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration
+ to the name of the listener. If not defined, then the inter-broker listener is determined
+ by the security protocol defined by <code>security.inter.broker.protocol</code>, which
+ defaults to <code>PLAINTEXT</code>.</p>
+
+ <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to
+ declare a separate listener to be used for metadata propagation from the active controller
+ to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller
+ will use this listener when it needs to push metadata updates to the brokers in the cluster.
+ The benefit of using a control plane listener is that it uses a separate processing thread,
+ which makes it less likely for application traffic to impede timely propagation of metadata changes
+ (such as partition leader and ISR updates). Note that the default value is null, which
+ means that the controller will use the same listener as defined by <code>inter.broker.listener</code></p>
+
+ <p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used
+ exclusively for requests sent to nodes which have the "broker" role enabled in <code>process.roles</code>.
+ For legacy clusters, the active controller will connect to all brokers in the cluster
+ to send metadata updates. In KRaft, it is the other way around. The broker is responsible
+ for finding the current controller and connecting to it in order to receive metadata updates.
+ Hence the configuration <code>control.plane.listener.name</code> cannot be used in KRaft clusters.</p>
+
+ <p>Instead, nodes with the "controller" role must define separate listeners through the
+ <code>controller.listener.names</code> configuration. Any node running with both the
+ "broker" and "controller" roles must use separate listeners for the broker and controller.
+ For example, a minimal configuration for a combined node might look like the following:</p>
+
+ <pre class="line-numbers"><code class="language-text">process.roles=broker,controller
+listeners=BROKER://localhost:9092,CONTROLLER://localhost:9093
+inter.broker.listener.name=BROKER
+controller.listener.names=CONTROLLER
+listener.security.protocol.map=BROKER:SASL_SSL,CONTROLLER:SASL_SSL</code></pre>
+
+ <p>This defines two listeners, one for the controller and one for the broker. It is conventional
+ to use a separate listener for client traffic as well.</p>
+
+ <p>The controller will accept requests on all listeners defined by <code>controller.listener.names</code>.
+ For outbound requests to other controller nodes, the first listener listed in <code>controller.listener.names</code>
+ will be used. The list of listeners provides a way to change the active listener from one port
+ to another through a roll of the cluster. Basically do one roll to expose the new listener,
+ and one roll to remove the old listener.</p>
+
+ <p>In the following section, this document covers how to enable SSL on a listener for encryption
+ as well as authentication. The subsequent section will then cover additional authentication
+ mechanism using SASL.</p>
Review Comment:
Should we have these relative references "In the following section" and "The subsequent section"? Should we instead use the anchor id to provide a direct reference to them.
Feel free to ignore this comment if it is not possible with the current HTML documentation implementation.
##########
docs/security.html:
##########
@@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage clearer.</p>
+
+ <p>Among the listeners in this list, it is possible to declare the listener to be used for
+ inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration
+ to the name of the listener. If not defined, then the inter-broker listener is determined
+ by the security protocol defined by <code>security.inter.broker.protocol</code>, which
+ defaults to <code>PLAINTEXT</code>.</p>
+
+ <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to
+ declare a separate listener to be used for metadata propagation from the active controller
+ to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller
+ will use this listener when it needs to push metadata updates to the brokers in the cluster.
+ The benefit of using a control plane listener is that it uses a separate processing thread,
+ which makes it less likely for application traffic to impede timely propagation of metadata changes
+ (such as partition leader and ISR updates). Note that the default value is null, which
+ means that the controller will use the same listener as defined by <code>inter.broker.listener</code></p>
+
+ <p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used
+ exclusively for requests sent to nodes which have the "broker" role enabled in <code>process.roles</code>.
+ For legacy clusters, the active controller will connect to all brokers in the cluster
+ to send metadata updates. In KRaft, it is the other way around. The broker is responsible
+ for finding the current controller and connecting to it in order to receive metadata updates.
+ Hence the configuration <code>control.plane.listener.name</code> cannot be used in KRaft clusters.</p>
Review Comment:
```suggestion
<p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used
exclusively for requests sent by brokers to brokers. A Kafka server is a broker if the <code>broker</code> role enabled in <code>process.roles</code>.
In KRaft, the broker is responsible
for finding the current controller and connecting to it in order to receive metadata updates.
The broker determines the listener and security protocol based the configuration in <code>controller.listener.names</code> and <code>listener.security.protocol.map</code></p>
```
I think it is easier to read if we document how legacy cluster are configure in one paragraph/section and how kraft clusters are configured in another paragraph/section. I think that comparing the two makes it harder to read and understand.
I would argue that at this point the user it trying to configure their cluster correctly. They are less interest in the reason of why the configuration are different. I think they can read the KIPs and discussion for that kind of detail. What do you think?
##########
docs/security.html:
##########
@@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
Review Comment:
```suggestion
communicate with the servers. Each server must define the set of listeners that are used to
```
##########
docs/security.html:
##########
@@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage clearer.</p>
+
+ <p>Among the listeners in this list, it is possible to declare the listener to be used for
+ inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration
+ to the name of the listener. If not defined, then the inter-broker listener is determined
+ by the security protocol defined by <code>security.inter.broker.protocol</code>, which
+ defaults to <code>PLAINTEXT</code>.</p>
Review Comment:
Should we mentioned that inter-broker communication is mainly used for the replication of topic partitions?
##########
docs/security.html:
##########
@@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage clearer.</p>
Review Comment:
I say we remove this if we don't want to recommend this type of configuration.
If a user is ready this section to understand an existing configuration maybe we can move this to the end of the section as a note.
##########
docs/security.html:
##########
@@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage clearer.</p>
+
+ <p>Among the listeners in this list, it is possible to declare the listener to be used for
+ inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration
+ to the name of the listener. If not defined, then the inter-broker listener is determined
+ by the security protocol defined by <code>security.inter.broker.protocol</code>, which
+ defaults to <code>PLAINTEXT</code>.</p>
+
+ <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to
+ declare a separate listener to be used for metadata propagation from the active controller
+ to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller
+ will use this listener when it needs to push metadata updates to the brokers in the cluster.
+ The benefit of using a control plane listener is that it uses a separate processing thread,
+ which makes it less likely for application traffic to impede timely propagation of metadata changes
+ (such as partition leader and ISR updates). Note that the default value is null, which
+ means that the controller will use the same listener as defined by <code>inter.broker.listener</code></p>
+
+ <p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used
+ exclusively for requests sent to nodes which have the "broker" role enabled in <code>process.roles</code>.
+ For legacy clusters, the active controller will connect to all brokers in the cluster
+ to send metadata updates. In KRaft, it is the other way around. The broker is responsible
+ for finding the current controller and connecting to it in order to receive metadata updates.
+ Hence the configuration <code>control.plane.listener.name</code> cannot be used in KRaft clusters.</p>
+
+ <p>Instead, nodes with the "controller" role must define separate listeners through the
+ <code>controller.listener.names</code> configuration. Any node running with both the
+ "broker" and "controller" roles must use separate listeners for the broker and controller.
+ For example, a minimal configuration for a combined node might look like the following:</p>
Review Comment:
```suggestion
<p>Instead, servers with the <code>controlle</code> role must define separate listeners through the
<code>controller.listener.names</code> configuration. Any server running with both the
<code>broker</code> and <code>controller</code> roles must use separate listeners for the broker and controller.
For example, a minimal configuration for a combined server might look like the following:</p>
```
The broker also needs to the controller listener name so that it knows how which security protocol to use.
##########
docs/security.html:
##########
@@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage clearer.</p>
+
+ <p>Among the listeners in this list, it is possible to declare the listener to be used for
+ inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration
+ to the name of the listener. If not defined, then the inter-broker listener is determined
+ by the security protocol defined by <code>security.inter.broker.protocol</code>, which
+ defaults to <code>PLAINTEXT</code>.</p>
+
+ <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to
+ declare a separate listener to be used for metadata propagation from the active controller
+ to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller
+ will use this listener when it needs to push metadata updates to the brokers in the cluster.
+ The benefit of using a control plane listener is that it uses a separate processing thread,
+ which makes it less likely for application traffic to impede timely propagation of metadata changes
+ (such as partition leader and ISR updates). Note that the default value is null, which
+ means that the controller will use the same listener as defined by <code>inter.broker.listener</code></p>
+
+ <p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used
+ exclusively for requests sent to nodes which have the "broker" role enabled in <code>process.roles</code>.
+ For legacy clusters, the active controller will connect to all brokers in the cluster
+ to send metadata updates. In KRaft, it is the other way around. The broker is responsible
+ for finding the current controller and connecting to it in order to receive metadata updates.
+ Hence the configuration <code>control.plane.listener.name</code> cannot be used in KRaft clusters.</p>
+
+ <p>Instead, nodes with the "controller" role must define separate listeners through the
+ <code>controller.listener.names</code> configuration. Any node running with both the
+ "broker" and "controller" roles must use separate listeners for the broker and controller.
+ For example, a minimal configuration for a combined node might look like the following:</p>
+
+ <pre class="line-numbers"><code class="language-text">process.roles=broker,controller
+listeners=BROKER://localhost:9092,CONTROLLER://localhost:9093
+inter.broker.listener.name=BROKER
+controller.listener.names=CONTROLLER
+listener.security.protocol.map=BROKER:SASL_SSL,CONTROLLER:SASL_SSL</code></pre>
+
+ <p>This defines two listeners, one for the controller and one for the broker. It is conventional
+ to use a separate listener for client traffic as well.</p>
+
+ <p>The controller will accept requests on all listeners defined by <code>controller.listener.names</code>.
+ For outbound requests to other controller nodes, the first listener listed in <code>controller.listener.names</code>
+ will be used. The list of listeners provides a way to change the active listener from one port
+ to another through a roll of the cluster. Basically do one roll to expose the new listener,
+ and one roll to remove the old listener.</p>
Review Comment:
```suggestion
<p>The controller will accept requests on all listeners defined by <code>controller.listener.names</code>.
For outbound requests to other controller servers, the first listener listed in <code>controller.listener.names</code>
will be used. The list of listeners provides a way to change the active listener from one port
to another through a roll of the cluster. Basically do one roll to expose the new listener,
and one roll to remove the old listener.</p>
```
Hmm. Regarding the last sentence, maybe we can add a section at the end describing the process of changing the controller listener. What do you think? I think that last sentence is hiding a lot of details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on a diff in pull request #12682: MINOR: Add section on listener configuration (including kraft) to security docs
Posted by GitBox <gi...@apache.org>.
hachikuji commented on code in PR #12682:
URL: https://github.com/apache/kafka/pull/12682#discussion_r980271965
##########
docs/security.html:
##########
@@ -36,7 +36,101 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage clearer.</p>
+
+ <p>Among the listeners in this list, it is possible to declare the listener to be used for
+ inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration
+ to the name of the listener. If not defined, then the inter-broker listener is determined
+ by the security protocol defined by <code>security.inter.broker.protocol</code>, which
+ defaults to <code>PLAINTEXT</code>.</p>
+
+ <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to
+ declare a separate listener to be used for metadata propagation from the active controller
+ to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller
+ will use this listener when it needs to push metadata updates to the brokers in the cluster.
+ The benefit of using a control plane listener is that it uses a separate processing thread,
+ which makes it less likely for application traffic to impede timely propagation of metadata changes
+ (such as partition leader and ISR updates).</p>
+
+ <p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used
+ exclusively for requests sent to nodes which have the "broker" role enabled in <code>process.roles</code>.
+ Unlike with legacy clusters, brokers connect to controllers (i.e. nodes with the "controller" role enabled)
+ in order to receive metadata updates; not the other way around. Hence the configuration
Review Comment:
Hmm, "in order to" seems right? Let me rephrase without the parenthetical to try and make it clearer.
##########
docs/security.html:
##########
@@ -36,7 +36,101 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage clearer.</p>
+
+ <p>Among the listeners in this list, it is possible to declare the listener to be used for
+ inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration
+ to the name of the listener. If not defined, then the inter-broker listener is determined
+ by the security protocol defined by <code>security.inter.broker.protocol</code>, which
+ defaults to <code>PLAINTEXT</code>.</p>
+
+ <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to
+ declare a separate listener to be used for metadata propagation from the active controller
+ to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller
+ will use this listener when it needs to push metadata updates to the brokers in the cluster.
+ The benefit of using a control plane listener is that it uses a separate processing thread,
+ which makes it less likely for application traffic to impede timely propagation of metadata changes
+ (such as partition leader and ISR updates).</p>
Review Comment:
Good idea.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a diff in pull request #12682: MINOR: Add section on listener configuration (including kraft) to security docs
Posted by GitBox <gi...@apache.org>.
showuon commented on code in PR #12682:
URL: https://github.com/apache/kafka/pull/12682#discussion_r979513912
##########
docs/security.html:
##########
@@ -36,7 +36,101 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage clearer.</p>
+
+ <p>Among the listeners in this list, it is possible to declare the listener to be used for
+ inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration
+ to the name of the listener. If not defined, then the inter-broker listener is determined
+ by the security protocol defined by <code>security.inter.broker.protocol</code>, which
+ defaults to <code>PLAINTEXT</code>.</p>
+
+ <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to
+ declare a separate listener to be used for metadata propagation from the active controller
+ to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller
+ will use this listener when it needs to push metadata updates to the brokers in the cluster.
+ The benefit of using a control plane listener is that it uses a separate processing thread,
+ which makes it less likely for application traffic to impede timely propagation of metadata changes
+ (such as partition leader and ISR updates).</p>
+
+ <p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used
+ exclusively for requests sent to nodes which have the "broker" role enabled in <code>process.roles</code>.
+ Unlike with legacy clusters, brokers connect to controllers (i.e. nodes with the "controller" role enabled)
+ in order to receive metadata updates; not the other way around. Hence the configuration
Review Comment:
in order to -> is order to ?
##########
docs/security.html:
##########
@@ -36,7 +36,101 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage clearer.</p>
+
+ <p>Among the listeners in this list, it is possible to declare the listener to be used for
+ inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration
+ to the name of the listener. If not defined, then the inter-broker listener is determined
+ by the security protocol defined by <code>security.inter.broker.protocol</code>, which
+ defaults to <code>PLAINTEXT</code>.</p>
+
+ <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to
+ declare a separate listener to be used for metadata propagation from the active controller
+ to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller
+ will use this listener when it needs to push metadata updates to the brokers in the cluster.
+ The benefit of using a control plane listener is that it uses a separate processing thread,
+ which makes it less likely for application traffic to impede timely propagation of metadata changes
+ (such as partition leader and ISR updates).</p>
+
+ <p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used
+ exclusively for requests sent to nodes which have the "broker" role enabled in <code>process.roles</code>.
+ Unlike with legacy clusters, brokers connect to controllers (i.e. nodes with the "controller" role enabled)
+ in order to receive metadata updates; not the other way around. Hence the configuration
+ <code>control.plane.listener.name</code> cannot be used in KRaft clusters.</p>
+
+ <p>Instead, nodes with the "controller" role must define <code>controller.listener.names</code>.
+ Any node running with both the "broker" and "controller" roles must have use separate listeners
Review Comment:
must have use -> must use?
##########
docs/security.html:
##########
@@ -36,7 +36,101 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage clearer.</p>
+
+ <p>Among the listeners in this list, it is possible to declare the listener to be used for
+ inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration
+ to the name of the listener. If not defined, then the inter-broker listener is determined
+ by the security protocol defined by <code>security.inter.broker.protocol</code>, which
+ defaults to <code>PLAINTEXT</code>.</p>
+
+ <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to
+ declare a separate listener to be used for metadata propagation from the active controller
+ to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller
+ will use this listener when it needs to push metadata updates to the brokers in the cluster.
+ The benefit of using a control plane listener is that it uses a separate processing thread,
+ which makes it less likely for application traffic to impede timely propagation of metadata changes
+ (such as partition leader and ISR updates).</p>
Review Comment:
Should we mention the `control.plane.listener.name` default value is null and there will be no dedicated endpoints for controller connections?
##########
docs/security.html:
##########
@@ -36,7 +36,101 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage clearer.</p>
+
+ <p>Among the listeners in this list, it is possible to declare the listener to be used for
+ inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration
+ to the name of the listener. If not defined, then the inter-broker listener is determined
+ by the security protocol defined by <code>security.inter.broker.protocol</code>, which
+ defaults to <code>PLAINTEXT</code>.</p>
+
+ <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to
+ declare a separate listener to be used for metadata propagation from the active controller
+ to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller
+ will use this listener when it needs to push metadata updates to the brokers in the cluster.
+ The benefit of using a control plane listener is that it uses a separate processing thread,
+ which makes it less likely for application traffic to impede timely propagation of metadata changes
+ (such as partition leader and ISR updates).</p>
+
+ <p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used
+ exclusively for requests sent to nodes which have the "broker" role enabled in <code>process.roles</code>.
+ Unlike with legacy clusters, brokers connect to controllers (i.e. nodes with the "controller" role enabled)
+ in order to receive metadata updates; not the other way around. Hence the configuration
+ <code>control.plane.listener.name</code> cannot be used in KRaft clusters.</p>
+
+ <p>Instead, nodes with the "controller" role must define <code>controller.listener.names</code>.
+ Any node running with both the "broker" and "controller" roles must have use separate listeners
+ for the broker an controller. For example, a minimal configuration for a combined node
Review Comment:
for the broker an[d] controller
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #12682: MINOR: Add section on listener configuration (including kraft) to security docs
Posted by GitBox <gi...@apache.org>.
showuon commented on PR #12682:
URL: https://github.com/apache/kafka/pull/12682#issuecomment-1258882134
> @showuon I'd appreciate one more look before we merge if you have time. Thanks!
Will merge it soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on pull request #12682: MINOR: Add section on listener configuration (including kraft) to security docs
Posted by GitBox <gi...@apache.org>.
showuon commented on PR #12682:
URL: https://github.com/apache/kafka/pull/12682#issuecomment-1258888447
backport to 3.3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on a diff in pull request #12682: MINOR: Add section on listener configuration (including kraft) to security docs
Posted by GitBox <gi...@apache.org>.
hachikuji commented on code in PR #12682:
URL: https://github.com/apache/kafka/pull/12682#discussion_r980495338
##########
docs/security.html:
##########
@@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage clearer.</p>
+
+ <p>Among the listeners in this list, it is possible to declare the listener to be used for
+ inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration
+ to the name of the listener. If not defined, then the inter-broker listener is determined
+ by the security protocol defined by <code>security.inter.broker.protocol</code>, which
+ defaults to <code>PLAINTEXT</code>.</p>
+
+ <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to
+ declare a separate listener to be used for metadata propagation from the active controller
+ to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller
+ will use this listener when it needs to push metadata updates to the brokers in the cluster.
+ The benefit of using a control plane listener is that it uses a separate processing thread,
+ which makes it less likely for application traffic to impede timely propagation of metadata changes
+ (such as partition leader and ISR updates). Note that the default value is null, which
+ means that the controller will use the same listener as defined by <code>inter.broker.listener</code></p>
+
+ <p>In a KRaft cluster, the listener defined by <code>inter.broker.listener.name</code> is used
+ exclusively for requests sent to nodes which have the "broker" role enabled in <code>process.roles</code>.
+ For legacy clusters, the active controller will connect to all brokers in the cluster
+ to send metadata updates. In KRaft, it is the other way around. The broker is responsible
+ for finding the current controller and connecting to it in order to receive metadata updates.
+ Hence the configuration <code>control.plane.listener.name</code> cannot be used in KRaft clusters.</p>
Review Comment:
Let me try to make the comment more concise. I just wanted a brief explanation for why the control plane listener doesn't exist in KRaft clusters.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] hachikuji commented on a diff in pull request #12682: MINOR: Add section on listener configuration (including kraft) to security docs
Posted by GitBox <gi...@apache.org>.
hachikuji commented on code in PR #12682:
URL: https://github.com/apache/kafka/pull/12682#discussion_r980518429
##########
docs/security.html:
##########
@@ -36,7 +36,104 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each node must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka brokers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each node. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage clearer.</p>
Review Comment:
Below we still have references to this approach, so I wanted to mention it. In a subsequent patch, I was going to clean up those sections as well, so maybe we can remove this then as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] showuon commented on a diff in pull request #12682: MINOR: Add section on listener configuration (including kraft) to security docs
Posted by GitBox <gi...@apache.org>.
showuon commented on code in PR #12682:
URL: https://github.com/apache/kafka/pull/12682#discussion_r980670293
##########
docs/security.html:
##########
@@ -36,7 +36,136 @@ <h3 class="anchor-heading"><a id="security_overview" class="anchor-link"></a><a
The guides below explain how to configure and use the security features in both clients and brokers.
- <h3 class="anchor-heading"><a id="security_ssl" class="anchor-link"></a><a href="#security_ssl">7.2 Encryption and Authentication using SSL</a></h3>
+ <h3 class="anchor-heading"><a id="listener_configuration" class="anchor-link"></a><a href="#listener_configuration">7.2 Listener Configuration</a></h3>
+
+ <p>In order to secure a Kafka cluster, it is necessary to secure the channels that are used to
+ communicate with the servers. Each server must define the set of listeners that are used to
+ receive requests from clients as well as other servers. Each listener may be configured
+ to authenticate clients using various mechanisms and to ensure traffic between the
+ server and the client is encrypted. This section provides a primer for the configuration
+ of listeners.</p>
+
+ <p>Kafka servers support listening for connections on multiple ports. This is configured through
+ the <code>listeners</code> property in the server configuration, which accepts a comma-separated
+ list of the listeners to enable. At least one listener must be defined on each server. The format
+ of each listener defined in <code>listeners</code> is given below:</p>
+
+ <pre class="line-numbers"><code class="language-text">{LISTENER_NAME}://{hostname}:{port}</code></pre>
+
+ <p>The <code>LISTENER_NAME</code> is usually a descriptive name which defines the purpose of
+ the listener. For example, many configurations use a separate listener for client traffic,
+ so they might refer to the corresponding listener as <code>CLIENT</code> in the configuration:</p
+
+ <pre class="line-numbers"><code class="language-text">listeners=CLIENT://localhost:9092</code></pre>
+
+ <p>The security protocol of each listener is defined in a separate configuration:
+ <code>listener.security.protocol.map</code>. The value is a comma-separated list
+ of each listener mapped to its security protocol. For example, the follow value
+ configuration specifies that the <code>CLIENT</code> listener will use SSL while the
+ <code>BROKER</code> listener will use plaintext.</p>
+
+ <pre class="line-numbers"><code class="language-text">listener.security.protocol.map=CLIENT:SSL,BROKER:PLAINTEXT</code></pre>
+
+ <p>Possible options for the security protocol are given below:</p>
+ <ol>
+ <li>PLAINTEXT</li>
+ <li>SSL</li>
+ <li>SASL_PLAINTEXT</li>
+ <li>SASL_SSL</li>
+ </ol>
+
+ <p>The plaintext protocol provides no security and does not require any additional configuration.
+ In the following sections, this document covers how to configure the remaining protocols.</p>
+
+ <p>If each required listener uses a separate security protocol, it is also possible to use the
+ security protocol name as the listener name in <code>listeners</code>. Using the example above,
+ we could skip the definition of the <code>CLIENT</code> and <code>BROKER</code> listeners
+ using the following definition:</p>
+
+ <pre class="line-numbers"><code class="language-text">listeners=SSL://localhost:9092,PLAINTEXT://localhost:9093</code></pre>
+
+ <p>However, we recommend users to provide explicit names for the listeners since it
+ makes the intended usage of each listener clearer.</p>
+
+ <p>Among the listeners in this list, it is possible to declare the listener to be used for
+ inter-broker communication by setting the <code>inter.broker.listener.name</code> configuration
+ to the name of the listener. The primary purpose of the inter-broker listener is
+ partition replication. If not defined, then the inter-broker listener is determined
+ by the security protocol defined by <code>security.inter.broker.protocol</code>, which
+ defaults to <code>PLAINTEXT</code>.</p>
+
+ <p>For legacy clusters which rely on Zookeeper to store cluster metadata, it is possible to
+ declare a separate listener to be used for metadata propagation from the active controller
+ to the brokers. This is defined by <code>control.plane.listener.name</code>. The active controller
+ will use this listener when it needs to push metadata updates to the brokers in the cluster.
+ The benefit of using a control plane listener is that it uses a separate processing thread,
+ which makes it less likely for application traffic to impede timely propagation of metadata changes
+ (such as partition leader and ISR updates). Note that the default value is null, which
+ means that the controller will use the same listener defined by <code>inter.broker.listener</code></p>
+
+ <p>In a KRaft cluster, a broker is any server which has the <code>broker</code> role enabled
+ in <code>process.roles</code> and a controller is any server which has the <code>controller</code>
+ role enabled. Listener configuration depends on the role. The listener defined by
+ <code>inter.broker.listener.name</code> is used exclusively for requests between brokers.
+ Controllers, on the other hand, must use separate listener which is defined by the
+ <code>controller.listener.names</code> configuration. This cannot be set to the same
+ value as the inter-broker listener.</p>
+
+ <p>Controllers receive requests both from other controllers and from brokers. For
+ this reason, even if a server does not have the <code>controller</code> role enabled
+ (i.e. it is just a broker), it must still define the controller listener along with
+ any security properties that are needed to configure it. For example, we might
+ use the following configuration on a standalone broker:</p>
+
+ <pre class="line-numbers"><code class="language-text">process.roles=broker
+listeners=BROKER://localhost:9092
+inter.broker.listener.name=BROKER
+controller.quorum.voters=0@localhost:9093
+controller.listener.names=CONTROLLER
+listener.security.protocol.map=BROKER:SASL_SSL,CONTROLLER:SASL_SSL</code></pre>
+
+ <p>The controller listener is still configured in this example to use the <code>SASL_SSL</code>
+ security protocol, but it is not included in <code>listeners</code> since the broker
+ does not expose the controller listener itself. The port that will be used in this case
+ comes from the <code>controller.quorum.voters</code> configuration, which defines
+ the complete list of controllers.</p>
+
+ <p>For KRaft servers which have both the broker and controller role enabled, the configuration
+ is similar. The only difference is that the controller listener must be included in
+ <code>listeners</code>:</p>
+
+ <pre class="line-numbers"><code class="language-text">process.roles=broker,controller
+listeners=BROKER://localhost:9092,CONTROLLER://localhost:9093
+inter.broker.listener.name=BROKER
+controller.quorum.voters=0@localhost:9093
+controller.listener.names=CONTROLLER
+listener.security.protocol.map=BROKER:SASL_SSL,CONTROLLER:SASL_SSL</code></pre>
+
+ <p>It is a requirement for the port defined in <code>controller.quorum.voters</code> to
+ exactly match one of the exposed controller listeners. For example, here the
+ <code>CONTROLLER</code> listener is bound to port 9093. The connection string
+ defined by <code>controller.quorum.voters</code> must then also use port 9093,
+ as it does here.</p>
+
+ <p>The controller will accept requests on all listeners defined by <code>controller.listener.names</code>.
+ Typically there would be just one controller listener, but it is possible to have more.
+ For example, this provides a way to change the active listener from one port or security
+ protocol to another through a roll of the cluster (one roll to expose the new listener,
+ and one roll to remove the old listener). When multiple controller listeners are defined,
+ the first one in the list will be used for outbound requests.</p>
+
+ <p>It is conventional in Kafka to use a separate listener for clients. This allows the
+ inter-cluster listeners to be isolated at the network level. In the case of the controller
+ listener in KRaft, the listener should be isolated since clients do not work with it
+ anyway. Clients are expected to connect to any other listener configured on a broker.
+ Any requests that are bound for the controller will be forwarded as described
+ <a href="#kraft_principal_forwarding">below</a></p>
Review Comment:
I guess this is in a separate PR, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org