[DISCUSS] KIP-649: Dynamic Client Configuration

[Ryan Dielhenn <rd...@confluent.io>]

Hi everyone,

I would like to start a discussion on KIP 649:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-649%3A+Dynamic+Client+Configuration

This proposal specifies the mechanisms that will enable dynamic configuration of producers and consumers. We have cherry picked a few producer and consumer configurations that we would like to provide dynamic support for and they are outlined in this KIP. Please reply with any comments or suggestions.

Thank you!
Ryan Dielhenn

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

[Ryan Dielhenn <rd...@confluent.io>]

Hi David,

Thank you for the comments, they were helpful. Here are the changes I made to the KIP: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=12&selectedPageVersions=11

> 1. Once dynamic configs have been loaded and resolved, how can a client
> know what values are selected? Will we log the actual resolved values once
> they are loaded?

Yes, they will be logged if the client determines that they are valid.

> 2. Do we want to support the case that we load dynamic configs but also
> allow client overrides?

I think this would mean that enable.dynamic.config would need to be able to take three values instead of just true or false. Since there are so few configurations that this KIP provides support for it may make more sense to just disable the feature if the user wants the user-provided client configs to take precedence. 

> 3. Are there other configs we'd like to allow the broker to push up to the
> clients? Did we consider making this mechanism generic so the broker could
> push any consumer/producer config up to the clients via dynamic configs?

Yes, the mechanism is generic. All that needs to be done to add support for an additional dynamic client config is to add that config to this ConfigDef https://github.com/apache/kafka/blob/b9cd0fd52741f99fd8f5912435bd2deaca5e5bab/core/src/main/scala/kafka/server/DynamicConfig.scala#L121

Once that is done, kafka-configs.sh will accept that new configuration, the broker will store it, and the client(s) will receive it when they pull their dynamic configs. Logic will also have to be added to the client to actually use this value. 



On 2020/08/03 17:47:19, David Arthur <mu...@gmail.com> wrote: 
> Hey Ryan, thanks for the KIP. This will be a really useful feature. Few
> questions
> 
> 1. Once dynamic configs have been loaded and resolved, how can a client
> know what values are selected? Will we log the actual resolved values once
> they are loaded?
> 
> 2. Do we want to support the case that we load dynamic configs but also
> allow client overrides?
> 
> 3. Are there other configs we'd like to allow the broker to push up to the
> clients? Did we consider making this mechanism generic so the broker could
> push any consumer/producer config up to the clients via dynamic configs?
> 
> Also, minor note, in ConfigResource#Type, I think the byte values are used
> for bit masking somewhere, so 9 won't work -- you'd need to bump up to 16
> 
> -David
> 
> On Fri, Jul 31, 2020 at 6:06 PM Ryan Dielhenn <rd...@confluent.io>
> wrote:
> 
> > Thanks Jose,
> > Here are the changes to the KIP:
> > https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=10&selectedPageVersions=9
> >
> > 1. Done.
> > 2. Done.
> > 3. Done.
> > 4. Yes metadata.max.age.ms will be use, I updated the KIP with this.
> >
> > > 5.
> > > The Consumer Changes section mentions that the consumer would ask for
> > > the dynamic configuration from the broker before joining the group
> > > coordinator. This makes sense to me. How about the producer? Should
> > > the producer also describe the dynamic configuration before sending
> > > acks for the "produce" messages?
> >
> >
> > I do think that the producer should do one synchronous
> > DescribeConfigsRequest before producing messages, just for different
> > reasons than the consumer. The decision to do this in the consumer was to
> > avoid an expensive rebalance operation associated with sending an extra
> > join group request.
> >
> > If this were done for the producer it would be to maintain durability
> > semantics by ensuring that the correct dynamic acks value is being used
> > before producing messages, however it would not improve performance like it
> > does for consumer groups.
> >
> > > 6.
> > > For the Admin Client Changes section, how are DescribeConfigs and
> > > IncrementalAlterConfig requests going to get routed by the client to
> > > the different brokers in the cluster?
> >
> > They will all be routed to the least loaded node for the CLIENT resource
> > type. I update the KIP with this.
> >
> > > 7.
> > > You mentioned that the producer and the consumer will validate the
> > > keys and values received from the broker through DescribeConfigs. Will
> > > the ConfigCommand validate any of the keys or values specified in
> > > --add-config and --delete-config? Will the broker validate any of the
> > > keys or values received in the IncrementalAlterConfigs?
> >
> > Yes, ConfigCommand will only allow keys that we are providing support for
> > to be sent to the broker. The broker will validate the keys as well as the
> > values and return an InvalidRequest error code if any key-value pairs are
> > invalid.
> >
> > 8. I removed this line.
> >
> > On 2020/07/31 18:27:38, Jose Garcia Sancio <js...@confluent.io> wrote:
> > > Thanks for the KIP Ryan. Here are some of my observations.
> > >
> > > 1.
> > > > We will also be adding a new client configuration
> > enable.dynamic.config to both the producer and consumer that will be true
> > by default so that the user has the option to disable this feature.
> > >
> > > How about?
> > > "The Java producer and consumer clients will have a new configuration
> > > property `enable.dynamic.config` with a default value of `true`. When
> > > this configuration property is true the proposed producer and consumer
> > > changes in this KIP are enabled."
> > >
> > > You can go into more details in the producer and consumer changes
> > section.
> > >
> > > 2.
> > > > ... the following order of precedence:
> > >
> > > I would be a bit more specific here. For example, when the broker
> > > handles `DescribeConfigRequest`, it will first use the client config
> > > key-values store in `/config/clients/<default>`. If there is a
> > > matching `config/clients/<client-id>` then any client config key-value
> > > in the matching znode will override the key-values found in
> > > `config/clients/<default>`.
> > >
> > > 3.
> > > > These dynamic configs will be stored in zookeeper as the children of
> > the ZNode /config/clients
> > >
> > > How are the client dynamic config key-values stored here? I assume
> > > that they are updated based on the content of
> > > `IncrementalAlterConfigsRequest`. When is `.../<default>` updated?
> > > When is `.../<client-id>` updated?
> > >
> > > 4.
> > > > The interval on which dynamic configs are fetched will be the same
> > amount of time as the interval for MetadataRequest which is currently five
> > minutes.
> > >
> > > Will this be hard-coded to 5 minutes? Or is this KIP going to use the
> > > same frequency as the producer config `metadata.max.age.ms`? Same
> > > question for the "Consumer Changes" section.
> > >
> > > 5.
> > > The Consumer Changes section mentions that the consumer would ask for
> > > the dynamic configuration from the broker before joining the group
> > > coordinator. This makes sense to me. How about the producer? Should
> > > the producer also describe the dynamic configuration before sending
> > > acks for the "produce" messages?
> > >
> > > 6.
> > > For the Admin Client Changes section, how are DescribeConfigs and
> > > IncrementalAlterConfig requests going to get routed by the client to
> > > the different brokers in the cluster?
> > >
> > > 7.
> > > You mentioned that the producer and the consumer will validate the
> > > keys and values received from the broker through DescribeConfigs. Will
> > > the ConfigCommand validate any of the keys or values specified in
> > > --add-config and --delete-config? Will the broker validate any of the
> > > keys or values received in the IncrementalAlterConfigs?
> > >
> > > 8.
> > > In rejected ideas the KIP says:
> > > > This might make sense for certain configurations such as acks, but
> > does not for others such as timeouts.
> > >
> > > I don't think it makes sense even for acks since the clients of the
> > > Java Producer assume that all of the produce messages are sent with
> > > the same ack value.
> > >
> > > --
> > > -Jose
> > >
> >
> 
> 
> -- 
> David Arthur
> 

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

[Ryan Dielhenn <rd...@confluent.io>]

Hi Jason,

I hope you're having a good start to your week! I made these changes
<https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=16&selectedPageVersions=14>
to
the KIP and they are reflected in the PR.

These changes include

1. Scoping dynamic configs by user-principal to address security issues.
2. Allowing default overrides only at the user level.
3. Changed enable.dynamic.config to default to false. This is so that the
user knows what applications support the capability since they had to set
the config on the client.
4. Updated the compatibility, deprecation, and migration plan to address
the issue of adding support for dynamic configs over time.

I am still working on changing the behavior of JoinGroup to avoid a
rebalance when updating the timeout.

Best,
Ryan Dielhenn





On Wed, Aug 5, 2020 at 5:23 PM Jason Gustafson <ja...@confluent.io> wrote:

> Hi Ryan,
>
> Thanks for the proposal. Just a few quick questions:
>
> 1. I wonder if we need to bother with `enable.dynamic.config`, especially
> if the default is going to be true anyway. I think users who don't want to
> use this capability can just not set dynamic configs. The only case I can
> see an explicit opt-out being useful is when users are trying to avoid
> getting affected by dynamic defaults. And on that note, is there a strong
> case for supporting default overrides? Many client configs are tied closely
> to application behavior, so it feels a bit dangerous to give users the
> ability to override the configuration for all applications.
>
> 2. Tying dynamic configurations to clientId has some downsides. It is
> common for users to use a different clientId for every application in a
> consumer group so that it is easier to tie group members back to where
> the client is running. This makes setting configurations at an application
> level cumbersome. The alternative is to use the default, but that means
> hitting /all/ applications which I think is probably not a good idea. A
> convenient alternative for consumers would be to use group.id, but we
> don't
> have anything similar for the producer. I am wondering if we need to give
> the clients a separate config label of some kind so that there is a
> convenient way to group configurations. For example `config.group`. Note
> that this would be another way to opt into dynamic config support.
>
> 3. I'm trying to understand the contract between brokers and clients to
> support dynamic configurations. I imagine that once this is available,
> users will have a hard time telling which applications support the
> capability and which do not. Also, we would likely add new dynamic config
> support over time which would make this even harder since we cannot
> retroactively change clients to add support for new dynamic configs. I'm
> wondering if there is anything we can do to make it easier for users to
> tell which dynamic configs are available for each application.
>
> 4. In the case of `session.timeout.ms`, even if the config is updated, the
> group will need to be rebalanced for it to take effect. This is because the
> session timeout is sent to the group coordinator in the JoinGroup request.
> I'm wondering if we need to change the JoinGroup behavior so that it can be
> used to update the session timeout without triggering a rebalance.
>
> Thanks,
> Jason
>
>
>
>
> On Mon, Aug 3, 2020 at 3:10 PM Ryan Dielhenn <rd...@confluent.io>
> wrote:
>
> > Hi David,
> >
> > Here are some additional thoughts...
> >
> > > 1. Once dynamic configs have been loaded and resolved, how can a client
> > > know what values are selected?
> >
> > A copy of the original user-provided configs is kept by the client.
> > Currently these are used to revert to the user-provided config if a
> dynamic
> > config is deleted. However, they can also be used to distinguish between
> > dynamic and user-provided configs.
> >
> > > 3. Are there other configs we'd like to allow the broker to push up to
> > the
> > > clients? Did we consider making this mechanism generic so the broker
> > could
> > > push any consumer/producer config up to the clients via dynamic
> configs?
> >
> > Rephrasing my answer to this question:
> >
> > The mechanism for sending and altering configs is rather generic.
> However,
> > the client-side handling of these configs is not. The reason for this is
> > that configs affect the behavior of the clients in specific ways, so the
> > client must reconfigure itself in a specific way for each different
> config.
> >
> > An example of this is that when session.timeout.ms is dynamically
> > configured, the consumer must rejoin the group by sending a
> > JoinGroupRequest. This is because the session timeout is sent in the
> > initial JoinGroupRequest to the coordinator and stored with the rest of
> the
> > group member's metadata. To reconfigure the client, the value in the
> > coordinator must also be changed. This does not need to be done for
> > heartbeat.interval.ms.
> >
> >
> > On 2020/08/03 17:47:19, David Arthur <mu...@gmail.com> wrote:
> > > Hey Ryan, thanks for the KIP. This will be a really useful feature. Few
> > > questions
> > >
> > > 1. Once dynamic configs have been loaded and resolved, how can a client
> > > know what values are selected? Will we log the actual resolved values
> > once
> > > they are loaded?
> > >
> > > 2. Do we want to support the case that we load dynamic configs but also
> > > allow client overrides?
> > >
> > > 3. Are there other configs we'd like to allow the broker to push up to
> > the
> > > clients? Did we consider making this mechanism generic so the broker
> > could
> > > push any consumer/producer config up to the clients via dynamic
> configs?
> > >
> > > Also, minor note, in ConfigResource#Type, I think the byte values are
> > used
> > > for bit masking somewhere, so 9 won't work -- you'd need to bump up to
> 16
> > >
> > > -David
> > >
> > > On Fri, Jul 31, 2020 at 6:06 PM Ryan Dielhenn <rd...@confluent.io>
> > > wrote:
> > >
> > > > Thanks Jose,
> > > > Here are the changes to the KIP:
> > > >
> >
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=10&selectedPageVersions=9
> > > >
> > > > 1. Done.
> > > > 2. Done.
> > > > 3. Done.
> > > > 4. Yes metadata.max.age.ms will be use, I updated the KIP with this.
> > > >
> > > > > 5.
> > > > > The Consumer Changes section mentions that the consumer would ask
> for
> > > > > the dynamic configuration from the broker before joining the group
> > > > > coordinator. This makes sense to me. How about the producer? Should
> > > > > the producer also describe the dynamic configuration before sending
> > > > > acks for the "produce" messages?
> > > >
> > > >
> > > > I do think that the producer should do one synchronous
> > > > DescribeConfigsRequest before producing messages, just for different
> > > > reasons than the consumer. The decision to do this in the consumer
> was
> > to
> > > > avoid an expensive rebalance operation associated with sending an
> extra
> > > > join group request.
> > > >
> > > > If this were done for the producer it would be to maintain durability
> > > > semantics by ensuring that the correct dynamic acks value is being
> used
> > > > before producing messages, however it would not improve performance
> > like it
> > > > does for consumer groups.
> > > >
> > > > > 6.
> > > > > For the Admin Client Changes section, how are DescribeConfigs and
> > > > > IncrementalAlterConfig requests going to get routed by the client
> to
> > > > > the different brokers in the cluster?
> > > >
> > > > They will all be routed to the least loaded node for the CLIENT
> > resource
> > > > type. I update the KIP with this.
> > > >
> > > > > 7.
> > > > > You mentioned that the producer and the consumer will validate the
> > > > > keys and values received from the broker through DescribeConfigs.
> > Will
> > > > > the ConfigCommand validate any of the keys or values specified in
> > > > > --add-config and --delete-config? Will the broker validate any of
> the
> > > > > keys or values received in the IncrementalAlterConfigs?
> > > >
> > > > Yes, ConfigCommand will only allow keys that we are providing support
> > for
> > > > to be sent to the broker. The broker will validate the keys as well
> as
> > the
> > > > values and return an InvalidRequest error code if any key-value pairs
> > are
> > > > invalid.
> > > >
> > > > 8. I removed this line.
> > > >
> > > > On 2020/07/31 18:27:38, Jose Garcia Sancio <js...@confluent.io>
> > wrote:
> > > > > Thanks for the KIP Ryan. Here are some of my observations.
> > > > >
> > > > > 1.
> > > > > > We will also be adding a new client configuration
> > > > enable.dynamic.config to both the producer and consumer that will be
> > true
> > > > by default so that the user has the option to disable this feature.
> > > > >
> > > > > How about?
> > > > > "The Java producer and consumer clients will have a new
> configuration
> > > > > property `enable.dynamic.config` with a default value of `true`.
> When
> > > > > this configuration property is true the proposed producer and
> > consumer
> > > > > changes in this KIP are enabled."
> > > > >
> > > > > You can go into more details in the producer and consumer changes
> > > > section.
> > > > >
> > > > > 2.
> > > > > > ... the following order of precedence:
> > > > >
> > > > > I would be a bit more specific here. For example, when the broker
> > > > > handles `DescribeConfigRequest`, it will first use the client
> config
> > > > > key-values store in `/config/clients/<default>`. If there is a
> > > > > matching `config/clients/<client-id>` then any client config
> > key-value
> > > > > in the matching znode will override the key-values found in
> > > > > `config/clients/<default>`.
> > > > >
> > > > > 3.
> > > > > > These dynamic configs will be stored in zookeeper as the children
> > of
> > > > the ZNode /config/clients
> > > > >
> > > > > How are the client dynamic config key-values stored here? I assume
> > > > > that they are updated based on the content of
> > > > > `IncrementalAlterConfigsRequest`. When is `.../<default>` updated?
> > > > > When is `.../<client-id>` updated?
> > > > >
> > > > > 4.
> > > > > > The interval on which dynamic configs are fetched will be the
> same
> > > > amount of time as the interval for MetadataRequest which is currently
> > five
> > > > minutes.
> > > > >
> > > > > Will this be hard-coded to 5 minutes? Or is this KIP going to use
> the
> > > > > same frequency as the producer config `metadata.max.age.ms`? Same
> > > > > question for the "Consumer Changes" section.
> > > > >
> > > > > 5.
> > > > > The Consumer Changes section mentions that the consumer would ask
> for
> > > > > the dynamic configuration from the broker before joining the group
> > > > > coordinator. This makes sense to me. How about the producer? Should
> > > > > the producer also describe the dynamic configuration before sending
> > > > > acks for the "produce" messages?
> > > > >
> > > > > 6.
> > > > > For the Admin Client Changes section, how are DescribeConfigs and
> > > > > IncrementalAlterConfig requests going to get routed by the client
> to
> > > > > the different brokers in the cluster?
> > > > >
> > > > > 7.
> > > > > You mentioned that the producer and the consumer will validate the
> > > > > keys and values received from the broker through DescribeConfigs.
> > Will
> > > > > the ConfigCommand validate any of the keys or values specified in
> > > > > --add-config and --delete-config? Will the broker validate any of
> the
> > > > > keys or values received in the IncrementalAlterConfigs?
> > > > >
> > > > > 8.
> > > > > In rejected ideas the KIP says:
> > > > > > This might make sense for certain configurations such as acks,
> but
> > > > does not for others such as timeouts.
> > > > >
> > > > > I don't think it makes sense even for acks since the clients of the
> > > > > Java Producer assume that all of the produce messages are sent with
> > > > > the same ack value.
> > > > >
> > > > > --
> > > > > -Jose
> > > > >
> > > >
> > >
> > >
> > > --
> > > David Arthur
> > >
> >
>

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

[Ryan Dielhenn <rd...@confluent.io>]

Hi Jason,

Thank you for this feedback it was very insightful and helpful.

> 1. I wonder if we need to bother with `enable.dynamic.config`, especially
> if the default is going to be true anyway. I think users who don't want to
> use this capability can just not set dynamic configs. The only case I can
> see an explicit opt-out being useful is when users are trying to avoid
> getting affected by dynamic defaults. And on that note, is there a strong
> case for supporting default overrides? Many client configs are tied closely
> to application behavior, so it feels a bit dangerous to give users the
> ability to override the configuration for all applications.

I agree that dynamic defaults hitting all applications seems very dangerous and probably not a good idea. I also agree that there are better alternatives to `enable.dynamic.config`.

> 2. Tying dynamic configurations to clientId has some downsides. It is
> common for users to use a different clientId for every application in a
> consumer group so that it is easier to tie group members back to where
> the client is running. This makes setting configurations at an application
> level cumbersome. The alternative is to use the default, but that means
> hitting /all/ applications which I think is probably not a good idea. A
> convenient alternative for consumers would be to use group.id, but we don't
> have anything similar for the producer. I am wondering if we need to give
> the clients a separate config label of some kind so that there is a
> convenient way to group configurations. For example `config.group`. Note
> that this would be another way to opt into dynamic config support.

A label `config.group` might be the best way to allow users to arbitrarily group configs and opt into configs set up for specific workloads. This also makes the contract for dynamic config support explicit. The user must set this value in the client’s config file, so they should know that the client supports dynamic configs. Furthermore, it seems best to scope this `config.group` by the user principal. This will prevent users from altering and describing other user’s important configs. I’m wondering if new APIs would need to be created to scope by principal or if user/config.group could be encoded together in the `ResourceName` of `{Describe,IncrementalAlter}Configs`. The configs could be stored in:

/config/users/<user>/groups/<group>

I’m wondering if introducing dynamic defaults that are not scoped by principal for client configs is worth it e.g. /config/groups/<default>. This would create a complicated hierarchy, make it very confusing for the user to know where dynamic configs are coming from, and also has the potential for all applications to match against the default which was already determined to not be a good idea.

A default for a specific user also has issues e.g. /config/users/<user> since the user would no longer need to set `config.group` on the client. In this case the contract for dynamic config support is not explicit and this would not allow the user to opt out of dynamic config support. An alternative is that this default is only used if a config group was set on the client but not found by the broker. 

> 3. I'm trying to understand the contract between brokers and clients to
> support dynamic configurations. I imagine that once this is available,
> users will have a hard time telling which applications support the
> capability and which do not. Also, we would likely add new dynamic config
> support over time which would make this even harder since we cannot
> retroactively change clients to add support for new dynamic configs. I'm
> wondering if there is anything we can do to make it easier for users to
> tell which dynamic configs are available for each application.

Listing dynamic configs that are supported for each running application could be helpful, although this would require storage and APIs for each client to register the configs that are supported. I’m wondering if it is most practical to make the contract for dynamic config support more explicit to the user by introducing `config.group` and then just logging rejected dynamic configs on the client. This would include dynamic configs that the client does not support.

> I'm wondering if we need to change the JoinGroup behavior so that it can be
> used to update the session timeout without triggering a rebalance.

I agree. Could a flag be added to the JoinGroup API specifying that just the session timeout needs updating? Without this it seems like it would be hard to differentiate between a regular JoinGroupRequest and a JoinGroupRequest that was sent to update the timeout.

On 2020/08/06 00:23:28, Jason Gustafson <ja...@confluent.io> wrote: 
> Hi Ryan,
> 
> Thanks for the proposal. Just a few quick questions:
> 
> 1. I wonder if we need to bother with `enable.dynamic.config`, especially
> if the default is going to be true anyway. I think users who don't want to
> use this capability can just not set dynamic configs. The only case I can
> see an explicit opt-out being useful is when users are trying to avoid
> getting affected by dynamic defaults. And on that note, is there a strong
> case for supporting default overrides? Many client configs are tied closely
> to application behavior, so it feels a bit dangerous to give users the
> ability to override the configuration for all applications.
> 
> 2. Tying dynamic configurations to clientId has some downsides. It is
> common for users to use a different clientId for every application in a
> consumer group so that it is easier to tie group members back to where
> the client is running. This makes setting configurations at an application
> level cumbersome. The alternative is to use the default, but that means
> hitting /all/ applications which I think is probably not a good idea. A
> convenient alternative for consumers would be to use group.id, but we don't
> have anything similar for the producer. I am wondering if we need to give
> the clients a separate config label of some kind so that there is a
> convenient way to group configurations. For example `config.group`. Note
> that this would be another way to opt into dynamic config support.
> 
> 3. I'm trying to understand the contract between brokers and clients to
> support dynamic configurations. I imagine that once this is available,
> users will have a hard time telling which applications support the
> capability and which do not. Also, we would likely add new dynamic config
> support over time which would make this even harder since we cannot
> retroactively change clients to add support for new dynamic configs. I'm
> wondering if there is anything we can do to make it easier for users to
> tell which dynamic configs are available for each application.
> 
> 4. In the case of `session.timeout.ms`, even if the config is updated, the
> group will need to be rebalanced for it to take effect. This is because the
> session timeout is sent to the group coordinator in the JoinGroup request.
> I'm wondering if we need to change the JoinGroup behavior so that it can be
> used to update the session timeout without triggering a rebalance.
> 
> Thanks,
> Jason
> 
> 
> 
> 
> On Mon, Aug 3, 2020 at 3:10 PM Ryan Dielhenn <rd...@confluent.io> wrote:
> 
> > Hi David,
> >
> > Here are some additional thoughts...
> >
> > > 1. Once dynamic configs have been loaded and resolved, how can a client
> > > know what values are selected?
> >
> > A copy of the original user-provided configs is kept by the client.
> > Currently these are used to revert to the user-provided config if a dynamic
> > config is deleted. However, they can also be used to distinguish between
> > dynamic and user-provided configs.
> >
> > > 3. Are there other configs we'd like to allow the broker to push up to
> > the
> > > clients? Did we consider making this mechanism generic so the broker
> > could
> > > push any consumer/producer config up to the clients via dynamic configs?
> >
> > Rephrasing my answer to this question:
> >
> > The mechanism for sending and altering configs is rather generic. However,
> > the client-side handling of these configs is not. The reason for this is
> > that configs affect the behavior of the clients in specific ways, so the
> > client must reconfigure itself in a specific way for each different config.
> >
> > An example of this is that when session.timeout.ms is dynamically
> > configured, the consumer must rejoin the group by sending a
> > JoinGroupRequest. This is because the session timeout is sent in the
> > initial JoinGroupRequest to the coordinator and stored with the rest of the
> > group member's metadata. To reconfigure the client, the value in the
> > coordinator must also be changed. This does not need to be done for
> > heartbeat.interval.ms.
> >
> >
> > On 2020/08/03 17:47:19, David Arthur <mu...@gmail.com> wrote:
> > > Hey Ryan, thanks for the KIP. This will be a really useful feature. Few
> > > questions
> > >
> > > 1. Once dynamic configs have been loaded and resolved, how can a client
> > > know what values are selected? Will we log the actual resolved values
> > once
> > > they are loaded?
> > >
> > > 2. Do we want to support the case that we load dynamic configs but also
> > > allow client overrides?
> > >
> > > 3. Are there other configs we'd like to allow the broker to push up to
> > the
> > > clients? Did we consider making this mechanism generic so the broker
> > could
> > > push any consumer/producer config up to the clients via dynamic configs?
> > >
> > > Also, minor note, in ConfigResource#Type, I think the byte values are
> > used
> > > for bit masking somewhere, so 9 won't work -- you'd need to bump up to 16
> > >
> > > -David
> > >
> > > On Fri, Jul 31, 2020 at 6:06 PM Ryan Dielhenn <rd...@confluent.io>
> > > wrote:
> > >
> > > > Thanks Jose,
> > > > Here are the changes to the KIP:
> > > >
> > https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=10&selectedPageVersions=9
> > > >
> > > > 1. Done.
> > > > 2. Done.
> > > > 3. Done.
> > > > 4. Yes metadata.max.age.ms will be use, I updated the KIP with this.
> > > >
> > > > > 5.
> > > > > The Consumer Changes section mentions that the consumer would ask for
> > > > > the dynamic configuration from the broker before joining the group
> > > > > coordinator. This makes sense to me. How about the producer? Should
> > > > > the producer also describe the dynamic configuration before sending
> > > > > acks for the "produce" messages?
> > > >
> > > >
> > > > I do think that the producer should do one synchronous
> > > > DescribeConfigsRequest before producing messages, just for different
> > > > reasons than the consumer. The decision to do this in the consumer was
> > to
> > > > avoid an expensive rebalance operation associated with sending an extra
> > > > join group request.
> > > >
> > > > If this were done for the producer it would be to maintain durability
> > > > semantics by ensuring that the correct dynamic acks value is being used
> > > > before producing messages, however it would not improve performance
> > like it
> > > > does for consumer groups.
> > > >
> > > > > 6.
> > > > > For the Admin Client Changes section, how are DescribeConfigs and
> > > > > IncrementalAlterConfig requests going to get routed by the client to
> > > > > the different brokers in the cluster?
> > > >
> > > > They will all be routed to the least loaded node for the CLIENT
> > resource
> > > > type. I update the KIP with this.
> > > >
> > > > > 7.
> > > > > You mentioned that the producer and the consumer will validate the
> > > > > keys and values received from the broker through DescribeConfigs.
> > Will
> > > > > the ConfigCommand validate any of the keys or values specified in
> > > > > --add-config and --delete-config? Will the broker validate any of the
> > > > > keys or values received in the IncrementalAlterConfigs?
> > > >
> > > > Yes, ConfigCommand will only allow keys that we are providing support
> > for
> > > > to be sent to the broker. The broker will validate the keys as well as
> > the
> > > > values and return an InvalidRequest error code if any key-value pairs
> > are
> > > > invalid.
> > > >
> > > > 8. I removed this line.
> > > >
> > > > On 2020/07/31 18:27:38, Jose Garcia Sancio <js...@confluent.io>
> > wrote:
> > > > > Thanks for the KIP Ryan. Here are some of my observations.
> > > > >
> > > > > 1.
> > > > > > We will also be adding a new client configuration
> > > > enable.dynamic.config to both the producer and consumer that will be
> > true
> > > > by default so that the user has the option to disable this feature.
> > > > >
> > > > > How about?
> > > > > "The Java producer and consumer clients will have a new configuration
> > > > > property `enable.dynamic.config` with a default value of `true`. When
> > > > > this configuration property is true the proposed producer and
> > consumer
> > > > > changes in this KIP are enabled."
> > > > >
> > > > > You can go into more details in the producer and consumer changes
> > > > section.
> > > > >
> > > > > 2.
> > > > > > ... the following order of precedence:
> > > > >
> > > > > I would be a bit more specific here. For example, when the broker
> > > > > handles `DescribeConfigRequest`, it will first use the client config
> > > > > key-values store in `/config/clients/<default>`. If there is a
> > > > > matching `config/clients/<client-id>` then any client config
> > key-value
> > > > > in the matching znode will override the key-values found in
> > > > > `config/clients/<default>`.
> > > > >
> > > > > 3.
> > > > > > These dynamic configs will be stored in zookeeper as the children
> > of
> > > > the ZNode /config/clients
> > > > >
> > > > > How are the client dynamic config key-values stored here? I assume
> > > > > that they are updated based on the content of
> > > > > `IncrementalAlterConfigsRequest`. When is `.../<default>` updated?
> > > > > When is `.../<client-id>` updated?
> > > > >
> > > > > 4.
> > > > > > The interval on which dynamic configs are fetched will be the same
> > > > amount of time as the interval for MetadataRequest which is currently
> > five
> > > > minutes.
> > > > >
> > > > > Will this be hard-coded to 5 minutes? Or is this KIP going to use the
> > > > > same frequency as the producer config `metadata.max.age.ms`? Same
> > > > > question for the "Consumer Changes" section.
> > > > >
> > > > > 5.
> > > > > The Consumer Changes section mentions that the consumer would ask for
> > > > > the dynamic configuration from the broker before joining the group
> > > > > coordinator. This makes sense to me. How about the producer? Should
> > > > > the producer also describe the dynamic configuration before sending
> > > > > acks for the "produce" messages?
> > > > >
> > > > > 6.
> > > > > For the Admin Client Changes section, how are DescribeConfigs and
> > > > > IncrementalAlterConfig requests going to get routed by the client to
> > > > > the different brokers in the cluster?
> > > > >
> > > > > 7.
> > > > > You mentioned that the producer and the consumer will validate the
> > > > > keys and values received from the broker through DescribeConfigs.
> > Will
> > > > > the ConfigCommand validate any of the keys or values specified in
> > > > > --add-config and --delete-config? Will the broker validate any of the
> > > > > keys or values received in the IncrementalAlterConfigs?
> > > > >
> > > > > 8.
> > > > > In rejected ideas the KIP says:
> > > > > > This might make sense for certain configurations such as acks, but
> > > > does not for others such as timeouts.
> > > > >
> > > > > I don't think it makes sense even for acks since the clients of the
> > > > > Java Producer assume that all of the produce messages are sent with
> > > > > the same ack value.
> > > > >
> > > > > --
> > > > > -Jose
> > > > >
> > > >
> > >
> > >
> > > --
> > > David Arthur
> > >
> >
> 

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

[Ryan Dielher <di...@gmail.com>]

Hi again,

First of all, I apologize for the duplicate email I did not mean to send 2.

> I added the / dynamic / config `supported.configs` to the KIP. This is a map of client software name and version to a list of supported configs registered with that software name and version. 
> 
> e.g.
> {
> 'ClientInformation(softwareName=apache-kafka-java, softwareVersion=x.y.a-SNAPSHOT)': 'acks',
> 'ClientInformation(softwareName=apache-kafka-java, softwareVersion=x.y.b-SNAPSHOT)': 'acks, enable.idempotence'
> }  

There could be an internal topic for this instead of a dynamic config so that ad hoc aggregations can be performed on the config registrations. Would it make sense to key config registration by <user, client-id>? This would allow compatibility information and the current config values for each client that registered with a particular <user, client-id> entity to be aggregated and returned to the user in the DescribeClientConfigsResponse. Could clients register their configs every time that they update their configs? This way a log retention time could be set on the internal topic so that only data from active clients is kept.

Best,
Ryan

On 2020/09/08 07:10:52, Ryan Dielher <di...@gmail.com> wrote: 
> Hi Jason,
> 
> Thank you again for all of your feedback, it is greatly appreciated.
> Here are some more changes to the KIP: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=21&selectedPageVersions=17
> 
> > 1. I wonder if we need to bother with `enable.dynamic.config`, especially
> > if the default is going to be true anyway.
> 
> I removed this / static / config in favor of a / dynamic / config `supported.configs`. This config is explained in more detail below.
> 
> > 3. I'm trying to understand the contract between brokers and clients to
> > support dynamic configurations. I imagine that once this is available,
> > users will have a hard time telling which applications support the
> > capability and which do not. Also, we would likely add new dynamic config
> > support over time which would make this even harder since we cannot
> > retroactively change clients to add support for new dynamic configs. I'm
> > wondering if there is anything we can do to make it easier for users to
> > tell which dynamic configs are available for each application.
> 
> 
> I added the / dynamic / config `supported.configs` to the KIP. This is a map of client software name and version to a list of supported configs registered with that software name and version. 
> 
> e.g.
> {
> 'ClientInformation(softwareName=apache-kafka-java, softwareVersion=x.y.a-SNAPSHOT)': 'acks',
> 'ClientInformation(softwareName=apache-kafka-java, softwareVersion=x.y.b-SNAPSHOT)': 'acks, enable.idempotence'
> }  
> 
> The changes propose that `supported.configs` be serialized as a json and stored in an entity config alongside dynamic configs and quotas for an entity. This config is updated for an entity when producers and consumers register the dynamic configurations that they support with the entity. This registration happens every time that they request configs.
> 
> This makes the information refreshable since `supported.configs` can be deleted from an entity. Clients register configs every time they request configs, so the compatibility information will eventually be added again and continuously updated as new clients associate themselves with the entity.
> 
> > 2. Tying dynamic configurations to clientId has some downsides.
> 
> The new protocols in the changes to the KIP are based on the protocols for client quotas. 
> They allow dynamic configs to be tied to a user principal and optionally to a client-id without trying to fit multiple entity names and types into the {Describe, IncrementalAlter}Configs APIs. They also provide a more expressive and extensible interface for dynamic client config entity names and types.
> 
> > It is common for users to use a different clientId for every application in a
> > consumer group so that it is easier to tie group members back to where
> > the client is running. This makes setting configurations at an application
> > level cumbersome. The alternative is to use the default, but that means
> > hitting /all/ applications which I think is probably not a good idea. A
> > convenient alternative for consumers would be to use group.id, but we don't
> > have anything similar for the producer. I am wondering if we need to give
> > the clients a separate config label of some kind so that there is a
> > convenient way to group configurations. For example `config.group`. Note
> > that this would be another way to opt into dynamic config support.
> 
> Would it be reasonable to treat consumer entities slightly differently than producer entities since they are different types of clients with different needs?
> Since the design pattern of this new protocol is built in part for extensibility of entity names and types, could consumers be associated with an additional entity name and type? 
> 
> e.g. Consumer configs could be set at the user, group-id and client-id levels. They could be resolved with the following order of precedence from most precedent to least precedent.
> 
> /config/users/<user>/groups/<group-id>/clients/<client-id>
> /config/users/<user>/groups/<group-id>
> /config/users/<user>
> 
> This doesn’t introduce a new label but it would instead allow dynamic consumer configs to be set at the group-id level.
> 
> > 4. In the case of `session.timeout.ms`, even if the config is updated, the
> > group will need to be rebalanced for it to take effect. This is because the
> > session timeout is sent to the group coordinator in the JoinGroup request.
> > I'm wondering if we need to change the JoinGroup behavior so that it can be
> > used to update the session timeout without triggering a rebalance.
> 
> I updated the KIP and PR so that a JoinGroupRequest will not rebalance the group if the session timeout in the JoinGroupRequest is different from the session timeout in the GroupCoordinator’s metadata and if the group member that is rejoining is in a stable state. In practice this just changes the behavior of the leader of the group when the timeout is updated since the leader is the only one that will trigger a rebalance when rejoining while in a stable state. 
> 
> I would greatly appreciate any more feedback from you about these ideas.
> 
> Best,
> Ryan
> 
> On 2020/08/06 00:23:28, Jason Gustafson <ja...@confluent.io> wrote: 
> > Hi Ryan,
> > 
> > Thanks for the proposal. Just a few quick questions:
> > 
> > 1. I wonder if we need to bother with `enable.dynamic.config`, especially
> > if the default is going to be true anyway. I think users who don't want to
> > use this capability can just not set dynamic configs. The only case I can
> > see an explicit opt-out being useful is when users are trying to avoid
> > getting affected by dynamic defaults. And on that note, is there a strong
> > case for supporting default overrides? Many client configs are tied closely
> > to application behavior, so it feels a bit dangerous to give users the
> > ability to override the configuration for all applications.
> > 
> > 2. Tying dynamic configurations to clientId has some downsides. It is
> > common for users to use a different clientId for every application in a
> > consumer group so that it is easier to tie group members back to where
> > the client is running. This makes setting configurations at an application
> > level cumbersome. The alternative is to use the default, but that means
> > hitting /all/ applications which I think is probably not a good idea. A
> > convenient alternative for consumers would be to use group.id, but we don't
> > have anything similar for the producer. I am wondering if we need to give
> > the clients a separate config label of some kind so that there is a
> > convenient way to group configurations. For example `config.group`. Note
> > that this would be another way to opt into dynamic config support.
> > 
> > 3. I'm trying to understand the contract between brokers and clients to
> > support dynamic configurations. I imagine that once this is available,
> > users will have a hard time telling which applications support the
> > capability and which do not. Also, we would likely add new dynamic config
> > support over time which would make this even harder since we cannot
> > retroactively change clients to add support for new dynamic configs. I'm
> > wondering if there is anything we can do to make it easier for users to
> > tell which dynamic configs are available for each application.
> > 
> > 4. In the case of `session.timeout.ms`, even if the config is updated, the
> > group will need to be rebalanced for it to take effect. This is because the
> > session timeout is sent to the group coordinator in the JoinGroup request.
> > I'm wondering if we need to change the JoinGroup behavior so that it can be
> > used to update the session timeout without triggering a rebalance.
> > 
> > Thanks,
> > Jason
> > 
> > 
> > 
> > 
> > On Mon, Aug 3, 2020 at 3:10 PM Ryan Dielhenn <rd...@confluent.io> wrote:
> > 
> > > Hi David,
> > >
> > > Here are some additional thoughts...
> > >
> > > > 1. Once dynamic configs have been loaded and resolved, how can a client
> > > > know what values are selected?
> > >
> > > A copy of the original user-provided configs is kept by the client.
> > > Currently these are used to revert to the user-provided config if a dynamic
> > > config is deleted. However, they can also be used to distinguish between
> > > dynamic and user-provided configs.
> > >
> > > > 3. Are there other configs we'd like to allow the broker to push up to
> > > the
> > > > clients? Did we consider making this mechanism generic so the broker
> > > could
> > > > push any consumer/producer config up to the clients via dynamic configs?
> > >
> > > Rephrasing my answer to this question:
> > >
> > > The mechanism for sending and altering configs is rather generic. However,
> > > the client-side handling of these configs is not. The reason for this is
> > > that configs affect the behavior of the clients in specific ways, so the
> > > client must reconfigure itself in a specific way for each different config.
> > >
> > > An example of this is that when session.timeout.ms is dynamically
> > > configured, the consumer must rejoin the group by sending a
> > > JoinGroupRequest. This is because the session timeout is sent in the
> > > initial JoinGroupRequest to the coordinator and stored with the rest of the
> > > group member's metadata. To reconfigure the client, the value in the
> > > coordinator must also be changed. This does not need to be done for
> > > heartbeat.interval.ms.
> > >
> > >
> > > On 2020/08/03 17:47:19, David Arthur <mu...@gmail.com> wrote:
> > > > Hey Ryan, thanks for the KIP. This will be a really useful feature. Few
> > > > questions
> > > >
> > > > 1. Once dynamic configs have been loaded and resolved, how can a client
> > > > know what values are selected? Will we log the actual resolved values
> > > once
> > > > they are loaded?
> > > >
> > > > 2. Do we want to support the case that we load dynamic configs but also
> > > > allow client overrides?
> > > >
> > > > 3. Are there other configs we'd like to allow the broker to push up to
> > > the
> > > > clients? Did we consider making this mechanism generic so the broker
> > > could
> > > > push any consumer/producer config up to the clients via dynamic configs?
> > > >
> > > > Also, minor note, in ConfigResource#Type, I think the byte values are
> > > used
> > > > for bit masking somewhere, so 9 won't work -- you'd need to bump up to 16
> > > >
> > > > -David
> > > >
> > > > On Fri, Jul 31, 2020 at 6:06 PM Ryan Dielhenn <rd...@confluent.io>
> > > > wrote:
> > > >
> > > > > Thanks Jose,
> > > > > Here are the changes to the KIP:
> > > > >
> > > https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=10&selectedPageVersions=9
> > > > >
> > > > > 1. Done.
> > > > > 2. Done.
> > > > > 3. Done.
> > > > > 4. Yes metadata.max.age.ms will be use, I updated the KIP with this.
> > > > >
> > > > > > 5.
> > > > > > The Consumer Changes section mentions that the consumer would ask for
> > > > > > the dynamic configuration from the broker before joining the group
> > > > > > coordinator. This makes sense to me. How about the producer? Should
> > > > > > the producer also describe the dynamic configuration before sending
> > > > > > acks for the "produce" messages?
> > > > >
> > > > >
> > > > > I do think that the producer should do one synchronous
> > > > > DescribeConfigsRequest before producing messages, just for different
> > > > > reasons than the consumer. The decision to do this in the consumer was
> > > to
> > > > > avoid an expensive rebalance operation associated with sending an extra
> > > > > join group request.
> > > > >
> > > > > If this were done for the producer it would be to maintain durability
> > > > > semantics by ensuring that the correct dynamic acks value is being used
> > > > > before producing messages, however it would not improve performance
> > > like it
> > > > > does for consumer groups.
> > > > >
> > > > > > 6.
> > > > > > For the Admin Client Changes section, how are DescribeConfigs and
> > > > > > IncrementalAlterConfig requests going to get routed by the client to
> > > > > > the different brokers in the cluster?
> > > > >
> > > > > They will all be routed to the least loaded node for the CLIENT
> > > resource
> > > > > type. I update the KIP with this.
> > > > >
> > > > > > 7.
> > > > > > You mentioned that the producer and the consumer will validate the
> > > > > > keys and values received from the broker through DescribeConfigs.
> > > Will
> > > > > > the ConfigCommand validate any of the keys or values specified in
> > > > > > --add-config and --delete-config? Will the broker validate any of the
> > > > > > keys or values received in the IncrementalAlterConfigs?
> > > > >
> > > > > Yes, ConfigCommand will only allow keys that we are providing support
> > > for
> > > > > to be sent to the broker. The broker will validate the keys as well as
> > > the
> > > > > values and return an InvalidRequest error code if any key-value pairs
> > > are
> > > > > invalid.
> > > > >
> > > > > 8. I removed this line.
> > > > >
> > > > > On 2020/07/31 18:27:38, Jose Garcia Sancio <js...@confluent.io>
> > > wrote:
> > > > > > Thanks for the KIP Ryan. Here are some of my observations.
> > > > > >
> > > > > > 1.
> > > > > > > We will also be adding a new client configuration
> > > > > enable.dynamic.config to both the producer and consumer that will be
> > > true
> > > > > by default so that the user has the option to disable this feature.
> > > > > >
> > > > > > How about?
> > > > > > "The Java producer and consumer clients will have a new configuration
> > > > > > property `enable.dynamic.config` with a default value of `true`. When
> > > > > > this configuration property is true the proposed producer and
> > > consumer
> > > > > > changes in this KIP are enabled."
> > > > > >
> > > > > > You can go into more details in the producer and consumer changes
> > > > > section.
> > > > > >
> > > > > > 2.
> > > > > > > ... the following order of precedence:
> > > > > >
> > > > > > I would be a bit more specific here. For example, when the broker
> > > > > > handles `DescribeConfigRequest`, it will first use the client config
> > > > > > key-values store in `/config/clients/<default>`. If there is a
> > > > > > matching `config/clients/<client-id>` then any client config
> > > key-value
> > > > > > in the matching znode will override the key-values found in
> > > > > > `config/clients/<default>`.
> > > > > >
> > > > > > 3.
> > > > > > > These dynamic configs will be stored in zookeeper as the children
> > > of
> > > > > the ZNode /config/clients
> > > > > >
> > > > > > How are the client dynamic config key-values stored here? I assume
> > > > > > that they are updated based on the content of
> > > > > > `IncrementalAlterConfigsRequest`. When is `.../<default>` updated?
> > > > > > When is `.../<client-id>` updated?
> > > > > >
> > > > > > 4.
> > > > > > > The interval on which dynamic configs are fetched will be the same
> > > > > amount of time as the interval for MetadataRequest which is currently
> > > five
> > > > > minutes.
> > > > > >
> > > > > > Will this be hard-coded to 5 minutes? Or is this KIP going to use the
> > > > > > same frequency as the producer config `metadata.max.age.ms`? Same
> > > > > > question for the "Consumer Changes" section.
> > > > > >
> > > > > > 5.
> > > > > > The Consumer Changes section mentions that the consumer would ask for
> > > > > > the dynamic configuration from the broker before joining the group
> > > > > > coordinator. This makes sense to me. How about the producer? Should
> > > > > > the producer also describe the dynamic configuration before sending
> > > > > > acks for the "produce" messages?
> > > > > >
> > > > > > 6.
> > > > > > For the Admin Client Changes section, how are DescribeConfigs and
> > > > > > IncrementalAlterConfig requests going to get routed by the client to
> > > > > > the different brokers in the cluster?
> > > > > >
> > > > > > 7.
> > > > > > You mentioned that the producer and the consumer will validate the
> > > > > > keys and values received from the broker through DescribeConfigs.
> > > Will
> > > > > > the ConfigCommand validate any of the keys or values specified in
> > > > > > --add-config and --delete-config? Will the broker validate any of the
> > > > > > keys or values received in the IncrementalAlterConfigs?
> > > > > >
> > > > > > 8.
> > > > > > In rejected ideas the KIP says:
> > > > > > > This might make sense for certain configurations such as acks, but
> > > > > does not for others such as timeouts.
> > > > > >
> > > > > > I don't think it makes sense even for acks since the clients of the
> > > > > > Java Producer assume that all of the produce messages are sent with
> > > > > > the same ack value.
> > > > > >
> > > > > > --
> > > > > > -Jose
> > > > > >
> > > > >
> > > >
> > > >
> > > > --
> > > > David Arthur
> > > >
> > >
> > 
> 

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

[Ryan Dielher <di...@gmail.com>]

Hi Jason,

Thank you again for all of your feedback, it is greatly appreciated.
Here are some more changes to the KIP: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=21&selectedPageVersions=17

> 1. I wonder if we need to bother with `enable.dynamic.config`, especially
> if the default is going to be true anyway.

I removed this / static / config in favor of a / dynamic / config `supported.configs`. This config is explained in more detail below.

> 3. I'm trying to understand the contract between brokers and clients to
> support dynamic configurations. I imagine that once this is available,
> users will have a hard time telling which applications support the
> capability and which do not. Also, we would likely add new dynamic config
> support over time which would make this even harder since we cannot
> retroactively change clients to add support for new dynamic configs. I'm
> wondering if there is anything we can do to make it easier for users to
> tell which dynamic configs are available for each application.


I added the / dynamic / config `supported.configs` to the KIP. This is a map of client software name and version to a list of supported configs registered with that software name and version. 

e.g.
{
'ClientInformation(softwareName=apache-kafka-java, softwareVersion=x.y.a-SNAPSHOT)': 'acks',
'ClientInformation(softwareName=apache-kafka-java, softwareVersion=x.y.b-SNAPSHOT)': 'acks, enable.idempotence'
}  

The changes propose that `supported.configs` be serialized as a json and stored in an entity config alongside dynamic configs and quotas for an entity. This config is updated for an entity when producers and consumers register the dynamic configurations that they support with the entity. This registration happens every time that they request configs.

This makes the information refreshable since `supported.configs` can be deleted from an entity. Clients register configs every time they request configs, so the compatibility information will eventually be added again and continuously updated as new clients associate themselves with the entity.

> 2. Tying dynamic configurations to clientId has some downsides.

The new protocols in the changes to the KIP are based on the protocols for client quotas. 
They allow dynamic configs to be tied to a user principal and optionally to a client-id without trying to fit multiple entity names and types into the {Describe, IncrementalAlter}Configs APIs. They also provide a more expressive and extensible interface for dynamic client config entity names and types.

> It is common for users to use a different clientId for every application in a
> consumer group so that it is easier to tie group members back to where
> the client is running. This makes setting configurations at an application
> level cumbersome. The alternative is to use the default, but that means
> hitting /all/ applications which I think is probably not a good idea. A
> convenient alternative for consumers would be to use group.id, but we don't
> have anything similar for the producer. I am wondering if we need to give
> the clients a separate config label of some kind so that there is a
> convenient way to group configurations. For example `config.group`. Note
> that this would be another way to opt into dynamic config support.

Would it be reasonable to treat consumer entities slightly differently than producer entities since they are different types of clients with different needs?
Since the design pattern of this new protocol is built in part for extensibility of entity names and types, could consumers be associated with an additional entity name and type? 

e.g. Consumer configs could be set at the user, group-id and client-id levels. They could be resolved with the following order of precedence from most precedent to least precedent.

/config/users/<user>/groups/<group-id>/clients/<client-id>
/config/users/<user>/groups/<group-id>
/config/users/<user>

This doesn’t introduce a new label but it would instead allow dynamic consumer configs to be set at the group-id level.

> 4. In the case of `session.timeout.ms`, even if the config is updated, the
> group will need to be rebalanced for it to take effect. This is because the
> session timeout is sent to the group coordinator in the JoinGroup request.
> I'm wondering if we need to change the JoinGroup behavior so that it can be
> used to update the session timeout without triggering a rebalance.

I updated the KIP and PR so that a JoinGroupRequest will not rebalance the group if the session timeout in the JoinGroupRequest is different from the session timeout in the GroupCoordinator’s metadata and if the group member that is rejoining is in a stable state. In practice this just changes the behavior of the leader of the group when the timeout is updated since the leader is the only one that will trigger a rebalance when rejoining while in a stable state. 

I would greatly appreciate any more feedback from you about these ideas.

Best,
Ryan

On 2020/08/06 00:23:28, Jason Gustafson <ja...@confluent.io> wrote: 
> Hi Ryan,
> 
> Thanks for the proposal. Just a few quick questions:
> 
> 1. I wonder if we need to bother with `enable.dynamic.config`, especially
> if the default is going to be true anyway. I think users who don't want to
> use this capability can just not set dynamic configs. The only case I can
> see an explicit opt-out being useful is when users are trying to avoid
> getting affected by dynamic defaults. And on that note, is there a strong
> case for supporting default overrides? Many client configs are tied closely
> to application behavior, so it feels a bit dangerous to give users the
> ability to override the configuration for all applications.
> 
> 2. Tying dynamic configurations to clientId has some downsides. It is
> common for users to use a different clientId for every application in a
> consumer group so that it is easier to tie group members back to where
> the client is running. This makes setting configurations at an application
> level cumbersome. The alternative is to use the default, but that means
> hitting /all/ applications which I think is probably not a good idea. A
> convenient alternative for consumers would be to use group.id, but we don't
> have anything similar for the producer. I am wondering if we need to give
> the clients a separate config label of some kind so that there is a
> convenient way to group configurations. For example `config.group`. Note
> that this would be another way to opt into dynamic config support.
> 
> 3. I'm trying to understand the contract between brokers and clients to
> support dynamic configurations. I imagine that once this is available,
> users will have a hard time telling which applications support the
> capability and which do not. Also, we would likely add new dynamic config
> support over time which would make this even harder since we cannot
> retroactively change clients to add support for new dynamic configs. I'm
> wondering if there is anything we can do to make it easier for users to
> tell which dynamic configs are available for each application.
> 
> 4. In the case of `session.timeout.ms`, even if the config is updated, the
> group will need to be rebalanced for it to take effect. This is because the
> session timeout is sent to the group coordinator in the JoinGroup request.
> I'm wondering if we need to change the JoinGroup behavior so that it can be
> used to update the session timeout without triggering a rebalance.
> 
> Thanks,
> Jason
> 
> 
> 
> 
> On Mon, Aug 3, 2020 at 3:10 PM Ryan Dielhenn <rd...@confluent.io> wrote:
> 
> > Hi David,
> >
> > Here are some additional thoughts...
> >
> > > 1. Once dynamic configs have been loaded and resolved, how can a client
> > > know what values are selected?
> >
> > A copy of the original user-provided configs is kept by the client.
> > Currently these are used to revert to the user-provided config if a dynamic
> > config is deleted. However, they can also be used to distinguish between
> > dynamic and user-provided configs.
> >
> > > 3. Are there other configs we'd like to allow the broker to push up to
> > the
> > > clients? Did we consider making this mechanism generic so the broker
> > could
> > > push any consumer/producer config up to the clients via dynamic configs?
> >
> > Rephrasing my answer to this question:
> >
> > The mechanism for sending and altering configs is rather generic. However,
> > the client-side handling of these configs is not. The reason for this is
> > that configs affect the behavior of the clients in specific ways, so the
> > client must reconfigure itself in a specific way for each different config.
> >
> > An example of this is that when session.timeout.ms is dynamically
> > configured, the consumer must rejoin the group by sending a
> > JoinGroupRequest. This is because the session timeout is sent in the
> > initial JoinGroupRequest to the coordinator and stored with the rest of the
> > group member's metadata. To reconfigure the client, the value in the
> > coordinator must also be changed. This does not need to be done for
> > heartbeat.interval.ms.
> >
> >
> > On 2020/08/03 17:47:19, David Arthur <mu...@gmail.com> wrote:
> > > Hey Ryan, thanks for the KIP. This will be a really useful feature. Few
> > > questions
> > >
> > > 1. Once dynamic configs have been loaded and resolved, how can a client
> > > know what values are selected? Will we log the actual resolved values
> > once
> > > they are loaded?
> > >
> > > 2. Do we want to support the case that we load dynamic configs but also
> > > allow client overrides?
> > >
> > > 3. Are there other configs we'd like to allow the broker to push up to
> > the
> > > clients? Did we consider making this mechanism generic so the broker
> > could
> > > push any consumer/producer config up to the clients via dynamic configs?
> > >
> > > Also, minor note, in ConfigResource#Type, I think the byte values are
> > used
> > > for bit masking somewhere, so 9 won't work -- you'd need to bump up to 16
> > >
> > > -David
> > >
> > > On Fri, Jul 31, 2020 at 6:06 PM Ryan Dielhenn <rd...@confluent.io>
> > > wrote:
> > >
> > > > Thanks Jose,
> > > > Here are the changes to the KIP:
> > > >
> > https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=10&selectedPageVersions=9
> > > >
> > > > 1. Done.
> > > > 2. Done.
> > > > 3. Done.
> > > > 4. Yes metadata.max.age.ms will be use, I updated the KIP with this.
> > > >
> > > > > 5.
> > > > > The Consumer Changes section mentions that the consumer would ask for
> > > > > the dynamic configuration from the broker before joining the group
> > > > > coordinator. This makes sense to me. How about the producer? Should
> > > > > the producer also describe the dynamic configuration before sending
> > > > > acks for the "produce" messages?
> > > >
> > > >
> > > > I do think that the producer should do one synchronous
> > > > DescribeConfigsRequest before producing messages, just for different
> > > > reasons than the consumer. The decision to do this in the consumer was
> > to
> > > > avoid an expensive rebalance operation associated with sending an extra
> > > > join group request.
> > > >
> > > > If this were done for the producer it would be to maintain durability
> > > > semantics by ensuring that the correct dynamic acks value is being used
> > > > before producing messages, however it would not improve performance
> > like it
> > > > does for consumer groups.
> > > >
> > > > > 6.
> > > > > For the Admin Client Changes section, how are DescribeConfigs and
> > > > > IncrementalAlterConfig requests going to get routed by the client to
> > > > > the different brokers in the cluster?
> > > >
> > > > They will all be routed to the least loaded node for the CLIENT
> > resource
> > > > type. I update the KIP with this.
> > > >
> > > > > 7.
> > > > > You mentioned that the producer and the consumer will validate the
> > > > > keys and values received from the broker through DescribeConfigs.
> > Will
> > > > > the ConfigCommand validate any of the keys or values specified in
> > > > > --add-config and --delete-config? Will the broker validate any of the
> > > > > keys or values received in the IncrementalAlterConfigs?
> > > >
> > > > Yes, ConfigCommand will only allow keys that we are providing support
> > for
> > > > to be sent to the broker. The broker will validate the keys as well as
> > the
> > > > values and return an InvalidRequest error code if any key-value pairs
> > are
> > > > invalid.
> > > >
> > > > 8. I removed this line.
> > > >
> > > > On 2020/07/31 18:27:38, Jose Garcia Sancio <js...@confluent.io>
> > wrote:
> > > > > Thanks for the KIP Ryan. Here are some of my observations.
> > > > >
> > > > > 1.
> > > > > > We will also be adding a new client configuration
> > > > enable.dynamic.config to both the producer and consumer that will be
> > true
> > > > by default so that the user has the option to disable this feature.
> > > > >
> > > > > How about?
> > > > > "The Java producer and consumer clients will have a new configuration
> > > > > property `enable.dynamic.config` with a default value of `true`. When
> > > > > this configuration property is true the proposed producer and
> > consumer
> > > > > changes in this KIP are enabled."
> > > > >
> > > > > You can go into more details in the producer and consumer changes
> > > > section.
> > > > >
> > > > > 2.
> > > > > > ... the following order of precedence:
> > > > >
> > > > > I would be a bit more specific here. For example, when the broker
> > > > > handles `DescribeConfigRequest`, it will first use the client config
> > > > > key-values store in `/config/clients/<default>`. If there is a
> > > > > matching `config/clients/<client-id>` then any client config
> > key-value
> > > > > in the matching znode will override the key-values found in
> > > > > `config/clients/<default>`.
> > > > >
> > > > > 3.
> > > > > > These dynamic configs will be stored in zookeeper as the children
> > of
> > > > the ZNode /config/clients
> > > > >
> > > > > How are the client dynamic config key-values stored here? I assume
> > > > > that they are updated based on the content of
> > > > > `IncrementalAlterConfigsRequest`. When is `.../<default>` updated?
> > > > > When is `.../<client-id>` updated?
> > > > >
> > > > > 4.
> > > > > > The interval on which dynamic configs are fetched will be the same
> > > > amount of time as the interval for MetadataRequest which is currently
> > five
> > > > minutes.
> > > > >
> > > > > Will this be hard-coded to 5 minutes? Or is this KIP going to use the
> > > > > same frequency as the producer config `metadata.max.age.ms`? Same
> > > > > question for the "Consumer Changes" section.
> > > > >
> > > > > 5.
> > > > > The Consumer Changes section mentions that the consumer would ask for
> > > > > the dynamic configuration from the broker before joining the group
> > > > > coordinator. This makes sense to me. How about the producer? Should
> > > > > the producer also describe the dynamic configuration before sending
> > > > > acks for the "produce" messages?
> > > > >
> > > > > 6.
> > > > > For the Admin Client Changes section, how are DescribeConfigs and
> > > > > IncrementalAlterConfig requests going to get routed by the client to
> > > > > the different brokers in the cluster?
> > > > >
> > > > > 7.
> > > > > You mentioned that the producer and the consumer will validate the
> > > > > keys and values received from the broker through DescribeConfigs.
> > Will
> > > > > the ConfigCommand validate any of the keys or values specified in
> > > > > --add-config and --delete-config? Will the broker validate any of the
> > > > > keys or values received in the IncrementalAlterConfigs?
> > > > >
> > > > > 8.
> > > > > In rejected ideas the KIP says:
> > > > > > This might make sense for certain configurations such as acks, but
> > > > does not for others such as timeouts.
> > > > >
> > > > > I don't think it makes sense even for acks since the clients of the
> > > > > Java Producer assume that all of the produce messages are sent with
> > > > > the same ack value.
> > > > >
> > > > > --
> > > > > -Jose
> > > > >
> > > >
> > >
> > >
> > > --
> > > David Arthur
> > >
> >
> 

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

[Ryan Dielher <di...@gmail.com>]

Hi Jason,

Thank you again for all of your feedback, it is greatly appreciated.
Here are some more changes to the KIP: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=21&selectedPageVersions=17

> 1. I wonder if we need to bother with `enable.dynamic.config`, especially
> if the default is going to be true anyway.

I removed this / static / config in favor of a / dynamic / config `supported.configs`. This config is explained in more detail below.

> 3. I'm trying to understand the contract between brokers and clients to
> support dynamic configurations. I imagine that once this is available,
> users will have a hard time telling which applications support the
> capability and which do not. Also, we would likely add new dynamic config
> support over time which would make this even harder since we cannot
> retroactively change clients to add support for new dynamic configs. I'm
> wondering if there is anything we can do to make it easier for users to
> tell which dynamic configs are available for each application.


I added the / dynamic / config `supported.configs` to the KIP. This is a map of client software name and version to a list of supported configs registered with that software name and version. 

e.g.
{
'ClientInformation(softwareName=apache-kafka-java, softwareVersion=x.y.a-SNAPSHOT)': 'acks',
'ClientInformation(softwareName=apache-kafka-java, softwareVersion=x.y.b-SNAPSHOT)': 'acks, enable.idempotence'
}  

The changes propose that `supported.configs` be serialized as a json and stored in an entity config alongside dynamic configs and quotas for an entity. This config is updated for an entity when producers and consumers register the dynamic configurations that they support with the entity. This registration happens every time that they request configs.

This makes the information refreshable since `supported.configs` can be deleted from an entity. Clients register configs every time they request configs, so the compatibility information will eventually be added again and continuously updated as new clients associate themselves with the entity.

> 2. Tying dynamic configurations to clientId has some downsides.

The new protocols in the changes to the KIP are based on the protocols for client quotas. 
They allow dynamic configs to be tied to a user principal and optionally to a client-id without trying to fit multiple entity names and types into the {Describe, IncrementalAlter}Configs APIs. They also provide a more expressive and extensible interface for dynamic client config entity names and types.

> It is common for users to use a different clientId for every application in a
> consumer group so that it is easier to tie group members back to where
> the client is running. This makes setting configurations at an application
> level cumbersome. The alternative is to use the default, but that means
> hitting /all/ applications which I think is probably not a good idea. A
> convenient alternative for consumers would be to use group.id, but we don't
> have anything similar for the producer. I am wondering if we need to give
> the clients a separate config label of some kind so that there is a
> convenient way to group configurations. For example `config.group`. Note
> that this would be another way to opt into dynamic config support.

Would it be reasonable to treat consumer entities slightly differently than producer entities since they are different types of clients with different needs?
Since the design pattern of this new protocol is built in part for extensibility of entity names and types, could consumers be associated with an additional entity name and type? 

e.g. Consumer configs could be set at the user, group-id and client-id levels. They could be resolved with the following order of precedence from most precedent to least precedent.

/config/users/<user>/groups/<group-id>/clients/<client-id>
/config/users/<user>/groups/<group-id>
/config/users/<user>

This doesn’t introduce a new label but it would instead allow dynamic consumer configs to be set at the group-id level.

> 4. In the case of `session.timeout.ms`, even if the config is updated, the
> group will need to be rebalanced for it to take effect. This is because the
> session timeout is sent to the group coordinator in the JoinGroup request.
> I'm wondering if we need to change the JoinGroup behavior so that it can be
> used to update the session timeout without triggering a rebalance.

I updated the KIP and PR so that a JoinGroupRequest will not rebalance the group if the session timeout in the JoinGroupRequest is different from the session timeout in the GroupCoordinator’s metadata and if the group member that is rejoining is in a stable state. In practice this just changes the behavior of the leader of the group when the timeout is updated since the leader is the only one that will trigger a rebalance when rejoining while in a stable state. 

I would greatly appreciate any more feedback from you about these ideas.

Best,
Ryan

On 2020/08/06 00:23:28, Jason Gustafson <ja...@confluent.io> wrote: 
> Hi Ryan,
> 
> Thanks for the proposal. Just a few quick questions:
> 
> 1. I wonder if we need to bother with `enable.dynamic.config`, especially
> if the default is going to be true anyway. I think users who don't want to
> use this capability can just not set dynamic configs. The only case I can
> see an explicit opt-out being useful is when users are trying to avoid
> getting affected by dynamic defaults. And on that note, is there a strong
> case for supporting default overrides? Many client configs are tied closely
> to application behavior, so it feels a bit dangerous to give users the
> ability to override the configuration for all applications.
> 
> 2. Tying dynamic configurations to clientId has some downsides. It is
> common for users to use a different clientId for every application in a
> consumer group so that it is easier to tie group members back to where
> the client is running. This makes setting configurations at an application
> level cumbersome. The alternative is to use the default, but that means
> hitting /all/ applications which I think is probably not a good idea. A
> convenient alternative for consumers would be to use group.id, but we don't
> have anything similar for the producer. I am wondering if we need to give
> the clients a separate config label of some kind so that there is a
> convenient way to group configurations. For example `config.group`. Note
> that this would be another way to opt into dynamic config support.
> 
> 3. I'm trying to understand the contract between brokers and clients to
> support dynamic configurations. I imagine that once this is available,
> users will have a hard time telling which applications support the
> capability and which do not. Also, we would likely add new dynamic config
> support over time which would make this even harder since we cannot
> retroactively change clients to add support for new dynamic configs. I'm
> wondering if there is anything we can do to make it easier for users to
> tell which dynamic configs are available for each application.
> 
> 4. In the case of `session.timeout.ms`, even if the config is updated, the
> group will need to be rebalanced for it to take effect. This is because the
> session timeout is sent to the group coordinator in the JoinGroup request.
> I'm wondering if we need to change the JoinGroup behavior so that it can be
> used to update the session timeout without triggering a rebalance.
> 
> Thanks,
> Jason
> 
> 
> 
> 
> On Mon, Aug 3, 2020 at 3:10 PM Ryan Dielhenn <rd...@confluent.io> wrote:
> 
> > Hi David,
> >
> > Here are some additional thoughts...
> >
> > > 1. Once dynamic configs have been loaded and resolved, how can a client
> > > know what values are selected?
> >
> > A copy of the original user-provided configs is kept by the client.
> > Currently these are used to revert to the user-provided config if a dynamic
> > config is deleted. However, they can also be used to distinguish between
> > dynamic and user-provided configs.
> >
> > > 3. Are there other configs we'd like to allow the broker to push up to
> > the
> > > clients? Did we consider making this mechanism generic so the broker
> > could
> > > push any consumer/producer config up to the clients via dynamic configs?
> >
> > Rephrasing my answer to this question:
> >
> > The mechanism for sending and altering configs is rather generic. However,
> > the client-side handling of these configs is not. The reason for this is
> > that configs affect the behavior of the clients in specific ways, so the
> > client must reconfigure itself in a specific way for each different config.
> >
> > An example of this is that when session.timeout.ms is dynamically
> > configured, the consumer must rejoin the group by sending a
> > JoinGroupRequest. This is because the session timeout is sent in the
> > initial JoinGroupRequest to the coordinator and stored with the rest of the
> > group member's metadata. To reconfigure the client, the value in the
> > coordinator must also be changed. This does not need to be done for
> > heartbeat.interval.ms.
> >
> >
> > On 2020/08/03 17:47:19, David Arthur <mu...@gmail.com> wrote:
> > > Hey Ryan, thanks for the KIP. This will be a really useful feature. Few
> > > questions
> > >
> > > 1. Once dynamic configs have been loaded and resolved, how can a client
> > > know what values are selected? Will we log the actual resolved values
> > once
> > > they are loaded?
> > >
> > > 2. Do we want to support the case that we load dynamic configs but also
> > > allow client overrides?
> > >
> > > 3. Are there other configs we'd like to allow the broker to push up to
> > the
> > > clients? Did we consider making this mechanism generic so the broker
> > could
> > > push any consumer/producer config up to the clients via dynamic configs?
> > >
> > > Also, minor note, in ConfigResource#Type, I think the byte values are
> > used
> > > for bit masking somewhere, so 9 won't work -- you'd need to bump up to 16
> > >
> > > -David
> > >
> > > On Fri, Jul 31, 2020 at 6:06 PM Ryan Dielhenn <rd...@confluent.io>
> > > wrote:
> > >
> > > > Thanks Jose,
> > > > Here are the changes to the KIP:
> > > >
> > https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=10&selectedPageVersions=9
> > > >
> > > > 1. Done.
> > > > 2. Done.
> > > > 3. Done.
> > > > 4. Yes metadata.max.age.ms will be use, I updated the KIP with this.
> > > >
> > > > > 5.
> > > > > The Consumer Changes section mentions that the consumer would ask for
> > > > > the dynamic configuration from the broker before joining the group
> > > > > coordinator. This makes sense to me. How about the producer? Should
> > > > > the producer also describe the dynamic configuration before sending
> > > > > acks for the "produce" messages?
> > > >
> > > >
> > > > I do think that the producer should do one synchronous
> > > > DescribeConfigsRequest before producing messages, just for different
> > > > reasons than the consumer. The decision to do this in the consumer was
> > to
> > > > avoid an expensive rebalance operation associated with sending an extra
> > > > join group request.
> > > >
> > > > If this were done for the producer it would be to maintain durability
> > > > semantics by ensuring that the correct dynamic acks value is being used
> > > > before producing messages, however it would not improve performance
> > like it
> > > > does for consumer groups.
> > > >
> > > > > 6.
> > > > > For the Admin Client Changes section, how are DescribeConfigs and
> > > > > IncrementalAlterConfig requests going to get routed by the client to
> > > > > the different brokers in the cluster?
> > > >
> > > > They will all be routed to the least loaded node for the CLIENT
> > resource
> > > > type. I update the KIP with this.
> > > >
> > > > > 7.
> > > > > You mentioned that the producer and the consumer will validate the
> > > > > keys and values received from the broker through DescribeConfigs.
> > Will
> > > > > the ConfigCommand validate any of the keys or values specified in
> > > > > --add-config and --delete-config? Will the broker validate any of the
> > > > > keys or values received in the IncrementalAlterConfigs?
> > > >
> > > > Yes, ConfigCommand will only allow keys that we are providing support
> > for
> > > > to be sent to the broker. The broker will validate the keys as well as
> > the
> > > > values and return an InvalidRequest error code if any key-value pairs
> > are
> > > > invalid.
> > > >
> > > > 8. I removed this line.
> > > >
> > > > On 2020/07/31 18:27:38, Jose Garcia Sancio <js...@confluent.io>
> > wrote:
> > > > > Thanks for the KIP Ryan. Here are some of my observations.
> > > > >
> > > > > 1.
> > > > > > We will also be adding a new client configuration
> > > > enable.dynamic.config to both the producer and consumer that will be
> > true
> > > > by default so that the user has the option to disable this feature.
> > > > >
> > > > > How about?
> > > > > "The Java producer and consumer clients will have a new configuration
> > > > > property `enable.dynamic.config` with a default value of `true`. When
> > > > > this configuration property is true the proposed producer and
> > consumer
> > > > > changes in this KIP are enabled."
> > > > >
> > > > > You can go into more details in the producer and consumer changes
> > > > section.
> > > > >
> > > > > 2.
> > > > > > ... the following order of precedence:
> > > > >
> > > > > I would be a bit more specific here. For example, when the broker
> > > > > handles `DescribeConfigRequest`, it will first use the client config
> > > > > key-values store in `/config/clients/<default>`. If there is a
> > > > > matching `config/clients/<client-id>` then any client config
> > key-value
> > > > > in the matching znode will override the key-values found in
> > > > > `config/clients/<default>`.
> > > > >
> > > > > 3.
> > > > > > These dynamic configs will be stored in zookeeper as the children
> > of
> > > > the ZNode /config/clients
> > > > >
> > > > > How are the client dynamic config key-values stored here? I assume
> > > > > that they are updated based on the content of
> > > > > `IncrementalAlterConfigsRequest`. When is `.../<default>` updated?
> > > > > When is `.../<client-id>` updated?
> > > > >
> > > > > 4.
> > > > > > The interval on which dynamic configs are fetched will be the same
> > > > amount of time as the interval for MetadataRequest which is currently
> > five
> > > > minutes.
> > > > >
> > > > > Will this be hard-coded to 5 minutes? Or is this KIP going to use the
> > > > > same frequency as the producer config `metadata.max.age.ms`? Same
> > > > > question for the "Consumer Changes" section.
> > > > >
> > > > > 5.
> > > > > The Consumer Changes section mentions that the consumer would ask for
> > > > > the dynamic configuration from the broker before joining the group
> > > > > coordinator. This makes sense to me. How about the producer? Should
> > > > > the producer also describe the dynamic configuration before sending
> > > > > acks for the "produce" messages?
> > > > >
> > > > > 6.
> > > > > For the Admin Client Changes section, how are DescribeConfigs and
> > > > > IncrementalAlterConfig requests going to get routed by the client to
> > > > > the different brokers in the cluster?
> > > > >
> > > > > 7.
> > > > > You mentioned that the producer and the consumer will validate the
> > > > > keys and values received from the broker through DescribeConfigs.
> > Will
> > > > > the ConfigCommand validate any of the keys or values specified in
> > > > > --add-config and --delete-config? Will the broker validate any of the
> > > > > keys or values received in the IncrementalAlterConfigs?
> > > > >
> > > > > 8.
> > > > > In rejected ideas the KIP says:
> > > > > > This might make sense for certain configurations such as acks, but
> > > > does not for others such as timeouts.
> > > > >
> > > > > I don't think it makes sense even for acks since the clients of the
> > > > > Java Producer assume that all of the produce messages are sent with
> > > > > the same ack value.
> > > > >
> > > > > --
> > > > > -Jose
> > > > >
> > > >
> > >
> > >
> > > --
> > > David Arthur
> > >
> >
> 

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

[Jason Gustafson <ja...@confluent.io>]

Hi Ryan,

Thanks for the proposal. Just a few quick questions:

1. I wonder if we need to bother with `enable.dynamic.config`, especially
if the default is going to be true anyway. I think users who don't want to
use this capability can just not set dynamic configs. The only case I can
see an explicit opt-out being useful is when users are trying to avoid
getting affected by dynamic defaults. And on that note, is there a strong
case for supporting default overrides? Many client configs are tied closely
to application behavior, so it feels a bit dangerous to give users the
ability to override the configuration for all applications.

2. Tying dynamic configurations to clientId has some downsides. It is
common for users to use a different clientId for every application in a
consumer group so that it is easier to tie group members back to where
the client is running. This makes setting configurations at an application
level cumbersome. The alternative is to use the default, but that means
hitting /all/ applications which I think is probably not a good idea. A
convenient alternative for consumers would be to use group.id, but we don't
have anything similar for the producer. I am wondering if we need to give
the clients a separate config label of some kind so that there is a
convenient way to group configurations. For example `config.group`. Note
that this would be another way to opt into dynamic config support.

3. I'm trying to understand the contract between brokers and clients to
support dynamic configurations. I imagine that once this is available,
users will have a hard time telling which applications support the
capability and which do not. Also, we would likely add new dynamic config
support over time which would make this even harder since we cannot
retroactively change clients to add support for new dynamic configs. I'm
wondering if there is anything we can do to make it easier for users to
tell which dynamic configs are available for each application.

4. In the case of `session.timeout.ms`, even if the config is updated, the
group will need to be rebalanced for it to take effect. This is because the
session timeout is sent to the group coordinator in the JoinGroup request.
I'm wondering if we need to change the JoinGroup behavior so that it can be
used to update the session timeout without triggering a rebalance.

Thanks,
Jason




On Mon, Aug 3, 2020 at 3:10 PM Ryan Dielhenn <rd...@confluent.io> wrote:

> Hi David,
>
> Here are some additional thoughts...
>
> > 1. Once dynamic configs have been loaded and resolved, how can a client
> > know what values are selected?
>
> A copy of the original user-provided configs is kept by the client.
> Currently these are used to revert to the user-provided config if a dynamic
> config is deleted. However, they can also be used to distinguish between
> dynamic and user-provided configs.
>
> > 3. Are there other configs we'd like to allow the broker to push up to
> the
> > clients? Did we consider making this mechanism generic so the broker
> could
> > push any consumer/producer config up to the clients via dynamic configs?
>
> Rephrasing my answer to this question:
>
> The mechanism for sending and altering configs is rather generic. However,
> the client-side handling of these configs is not. The reason for this is
> that configs affect the behavior of the clients in specific ways, so the
> client must reconfigure itself in a specific way for each different config.
>
> An example of this is that when session.timeout.ms is dynamically
> configured, the consumer must rejoin the group by sending a
> JoinGroupRequest. This is because the session timeout is sent in the
> initial JoinGroupRequest to the coordinator and stored with the rest of the
> group member's metadata. To reconfigure the client, the value in the
> coordinator must also be changed. This does not need to be done for
> heartbeat.interval.ms.
>
>
> On 2020/08/03 17:47:19, David Arthur <mu...@gmail.com> wrote:
> > Hey Ryan, thanks for the KIP. This will be a really useful feature. Few
> > questions
> >
> > 1. Once dynamic configs have been loaded and resolved, how can a client
> > know what values are selected? Will we log the actual resolved values
> once
> > they are loaded?
> >
> > 2. Do we want to support the case that we load dynamic configs but also
> > allow client overrides?
> >
> > 3. Are there other configs we'd like to allow the broker to push up to
> the
> > clients? Did we consider making this mechanism generic so the broker
> could
> > push any consumer/producer config up to the clients via dynamic configs?
> >
> > Also, minor note, in ConfigResource#Type, I think the byte values are
> used
> > for bit masking somewhere, so 9 won't work -- you'd need to bump up to 16
> >
> > -David
> >
> > On Fri, Jul 31, 2020 at 6:06 PM Ryan Dielhenn <rd...@confluent.io>
> > wrote:
> >
> > > Thanks Jose,
> > > Here are the changes to the KIP:
> > >
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=10&selectedPageVersions=9
> > >
> > > 1. Done.
> > > 2. Done.
> > > 3. Done.
> > > 4. Yes metadata.max.age.ms will be use, I updated the KIP with this.
> > >
> > > > 5.
> > > > The Consumer Changes section mentions that the consumer would ask for
> > > > the dynamic configuration from the broker before joining the group
> > > > coordinator. This makes sense to me. How about the producer? Should
> > > > the producer also describe the dynamic configuration before sending
> > > > acks for the "produce" messages?
> > >
> > >
> > > I do think that the producer should do one synchronous
> > > DescribeConfigsRequest before producing messages, just for different
> > > reasons than the consumer. The decision to do this in the consumer was
> to
> > > avoid an expensive rebalance operation associated with sending an extra
> > > join group request.
> > >
> > > If this were done for the producer it would be to maintain durability
> > > semantics by ensuring that the correct dynamic acks value is being used
> > > before producing messages, however it would not improve performance
> like it
> > > does for consumer groups.
> > >
> > > > 6.
> > > > For the Admin Client Changes section, how are DescribeConfigs and
> > > > IncrementalAlterConfig requests going to get routed by the client to
> > > > the different brokers in the cluster?
> > >
> > > They will all be routed to the least loaded node for the CLIENT
> resource
> > > type. I update the KIP with this.
> > >
> > > > 7.
> > > > You mentioned that the producer and the consumer will validate the
> > > > keys and values received from the broker through DescribeConfigs.
> Will
> > > > the ConfigCommand validate any of the keys or values specified in
> > > > --add-config and --delete-config? Will the broker validate any of the
> > > > keys or values received in the IncrementalAlterConfigs?
> > >
> > > Yes, ConfigCommand will only allow keys that we are providing support
> for
> > > to be sent to the broker. The broker will validate the keys as well as
> the
> > > values and return an InvalidRequest error code if any key-value pairs
> are
> > > invalid.
> > >
> > > 8. I removed this line.
> > >
> > > On 2020/07/31 18:27:38, Jose Garcia Sancio <js...@confluent.io>
> wrote:
> > > > Thanks for the KIP Ryan. Here are some of my observations.
> > > >
> > > > 1.
> > > > > We will also be adding a new client configuration
> > > enable.dynamic.config to both the producer and consumer that will be
> true
> > > by default so that the user has the option to disable this feature.
> > > >
> > > > How about?
> > > > "The Java producer and consumer clients will have a new configuration
> > > > property `enable.dynamic.config` with a default value of `true`. When
> > > > this configuration property is true the proposed producer and
> consumer
> > > > changes in this KIP are enabled."
> > > >
> > > > You can go into more details in the producer and consumer changes
> > > section.
> > > >
> > > > 2.
> > > > > ... the following order of precedence:
> > > >
> > > > I would be a bit more specific here. For example, when the broker
> > > > handles `DescribeConfigRequest`, it will first use the client config
> > > > key-values store in `/config/clients/<default>`. If there is a
> > > > matching `config/clients/<client-id>` then any client config
> key-value
> > > > in the matching znode will override the key-values found in
> > > > `config/clients/<default>`.
> > > >
> > > > 3.
> > > > > These dynamic configs will be stored in zookeeper as the children
> of
> > > the ZNode /config/clients
> > > >
> > > > How are the client dynamic config key-values stored here? I assume
> > > > that they are updated based on the content of
> > > > `IncrementalAlterConfigsRequest`. When is `.../<default>` updated?
> > > > When is `.../<client-id>` updated?
> > > >
> > > > 4.
> > > > > The interval on which dynamic configs are fetched will be the same
> > > amount of time as the interval for MetadataRequest which is currently
> five
> > > minutes.
> > > >
> > > > Will this be hard-coded to 5 minutes? Or is this KIP going to use the
> > > > same frequency as the producer config `metadata.max.age.ms`? Same
> > > > question for the "Consumer Changes" section.
> > > >
> > > > 5.
> > > > The Consumer Changes section mentions that the consumer would ask for
> > > > the dynamic configuration from the broker before joining the group
> > > > coordinator. This makes sense to me. How about the producer? Should
> > > > the producer also describe the dynamic configuration before sending
> > > > acks for the "produce" messages?
> > > >
> > > > 6.
> > > > For the Admin Client Changes section, how are DescribeConfigs and
> > > > IncrementalAlterConfig requests going to get routed by the client to
> > > > the different brokers in the cluster?
> > > >
> > > > 7.
> > > > You mentioned that the producer and the consumer will validate the
> > > > keys and values received from the broker through DescribeConfigs.
> Will
> > > > the ConfigCommand validate any of the keys or values specified in
> > > > --add-config and --delete-config? Will the broker validate any of the
> > > > keys or values received in the IncrementalAlterConfigs?
> > > >
> > > > 8.
> > > > In rejected ideas the KIP says:
> > > > > This might make sense for certain configurations such as acks, but
> > > does not for others such as timeouts.
> > > >
> > > > I don't think it makes sense even for acks since the clients of the
> > > > Java Producer assume that all of the produce messages are sent with
> > > > the same ack value.
> > > >
> > > > --
> > > > -Jose
> > > >
> > >
> >
> >
> > --
> > David Arthur
> >
>

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

[Ryan Dielhenn <rd...@confluent.io>]

Hi David,

Here are some additional thoughts...

> 1. Once dynamic configs have been loaded and resolved, how can a client
> know what values are selected?

A copy of the original user-provided configs is kept by the client. Currently these are used to revert to the user-provided config if a dynamic config is deleted. However, they can also be used to distinguish between dynamic and user-provided configs.

> 3. Are there other configs we'd like to allow the broker to push up to the
> clients? Did we consider making this mechanism generic so the broker could
> push any consumer/producer config up to the clients via dynamic configs?

Rephrasing my answer to this question:

The mechanism for sending and altering configs is rather generic. However, the client-side handling of these configs is not. The reason for this is that configs affect the behavior of the clients in specific ways, so the client must reconfigure itself in a specific way for each different config. 

An example of this is that when session.timeout.ms is dynamically configured, the consumer must rejoin the group by sending a JoinGroupRequest. This is because the session timeout is sent in the initial JoinGroupRequest to the coordinator and stored with the rest of the group member's metadata. To reconfigure the client, the value in the coordinator must also be changed. This does not need to be done for heartbeat.interval.ms.


On 2020/08/03 17:47:19, David Arthur <mu...@gmail.com> wrote: 
> Hey Ryan, thanks for the KIP. This will be a really useful feature. Few
> questions
> 
> 1. Once dynamic configs have been loaded and resolved, how can a client
> know what values are selected? Will we log the actual resolved values once
> they are loaded?
> 
> 2. Do we want to support the case that we load dynamic configs but also
> allow client overrides?
> 
> 3. Are there other configs we'd like to allow the broker to push up to the
> clients? Did we consider making this mechanism generic so the broker could
> push any consumer/producer config up to the clients via dynamic configs?
> 
> Also, minor note, in ConfigResource#Type, I think the byte values are used
> for bit masking somewhere, so 9 won't work -- you'd need to bump up to 16
> 
> -David
> 
> On Fri, Jul 31, 2020 at 6:06 PM Ryan Dielhenn <rd...@confluent.io>
> wrote:
> 
> > Thanks Jose,
> > Here are the changes to the KIP:
> > https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=10&selectedPageVersions=9
> >
> > 1. Done.
> > 2. Done.
> > 3. Done.
> > 4. Yes metadata.max.age.ms will be use, I updated the KIP with this.
> >
> > > 5.
> > > The Consumer Changes section mentions that the consumer would ask for
> > > the dynamic configuration from the broker before joining the group
> > > coordinator. This makes sense to me. How about the producer? Should
> > > the producer also describe the dynamic configuration before sending
> > > acks for the "produce" messages?
> >
> >
> > I do think that the producer should do one synchronous
> > DescribeConfigsRequest before producing messages, just for different
> > reasons than the consumer. The decision to do this in the consumer was to
> > avoid an expensive rebalance operation associated with sending an extra
> > join group request.
> >
> > If this were done for the producer it would be to maintain durability
> > semantics by ensuring that the correct dynamic acks value is being used
> > before producing messages, however it would not improve performance like it
> > does for consumer groups.
> >
> > > 6.
> > > For the Admin Client Changes section, how are DescribeConfigs and
> > > IncrementalAlterConfig requests going to get routed by the client to
> > > the different brokers in the cluster?
> >
> > They will all be routed to the least loaded node for the CLIENT resource
> > type. I update the KIP with this.
> >
> > > 7.
> > > You mentioned that the producer and the consumer will validate the
> > > keys and values received from the broker through DescribeConfigs. Will
> > > the ConfigCommand validate any of the keys or values specified in
> > > --add-config and --delete-config? Will the broker validate any of the
> > > keys or values received in the IncrementalAlterConfigs?
> >
> > Yes, ConfigCommand will only allow keys that we are providing support for
> > to be sent to the broker. The broker will validate the keys as well as the
> > values and return an InvalidRequest error code if any key-value pairs are
> > invalid.
> >
> > 8. I removed this line.
> >
> > On 2020/07/31 18:27:38, Jose Garcia Sancio <js...@confluent.io> wrote:
> > > Thanks for the KIP Ryan. Here are some of my observations.
> > >
> > > 1.
> > > > We will also be adding a new client configuration
> > enable.dynamic.config to both the producer and consumer that will be true
> > by default so that the user has the option to disable this feature.
> > >
> > > How about?
> > > "The Java producer and consumer clients will have a new configuration
> > > property `enable.dynamic.config` with a default value of `true`. When
> > > this configuration property is true the proposed producer and consumer
> > > changes in this KIP are enabled."
> > >
> > > You can go into more details in the producer and consumer changes
> > section.
> > >
> > > 2.
> > > > ... the following order of precedence:
> > >
> > > I would be a bit more specific here. For example, when the broker
> > > handles `DescribeConfigRequest`, it will first use the client config
> > > key-values store in `/config/clients/<default>`. If there is a
> > > matching `config/clients/<client-id>` then any client config key-value
> > > in the matching znode will override the key-values found in
> > > `config/clients/<default>`.
> > >
> > > 3.
> > > > These dynamic configs will be stored in zookeeper as the children of
> > the ZNode /config/clients
> > >
> > > How are the client dynamic config key-values stored here? I assume
> > > that they are updated based on the content of
> > > `IncrementalAlterConfigsRequest`. When is `.../<default>` updated?
> > > When is `.../<client-id>` updated?
> > >
> > > 4.
> > > > The interval on which dynamic configs are fetched will be the same
> > amount of time as the interval for MetadataRequest which is currently five
> > minutes.
> > >
> > > Will this be hard-coded to 5 minutes? Or is this KIP going to use the
> > > same frequency as the producer config `metadata.max.age.ms`? Same
> > > question for the "Consumer Changes" section.
> > >
> > > 5.
> > > The Consumer Changes section mentions that the consumer would ask for
> > > the dynamic configuration from the broker before joining the group
> > > coordinator. This makes sense to me. How about the producer? Should
> > > the producer also describe the dynamic configuration before sending
> > > acks for the "produce" messages?
> > >
> > > 6.
> > > For the Admin Client Changes section, how are DescribeConfigs and
> > > IncrementalAlterConfig requests going to get routed by the client to
> > > the different brokers in the cluster?
> > >
> > > 7.
> > > You mentioned that the producer and the consumer will validate the
> > > keys and values received from the broker through DescribeConfigs. Will
> > > the ConfigCommand validate any of the keys or values specified in
> > > --add-config and --delete-config? Will the broker validate any of the
> > > keys or values received in the IncrementalAlterConfigs?
> > >
> > > 8.
> > > In rejected ideas the KIP says:
> > > > This might make sense for certain configurations such as acks, but
> > does not for others such as timeouts.
> > >
> > > I don't think it makes sense even for acks since the clients of the
> > > Java Producer assume that all of the produce messages are sent with
> > > the same ack value.
> > >
> > > --
> > > -Jose
> > >
> >
> 
> 
> -- 
> David Arthur
> 

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

[David Arthur <mu...@gmail.com>]

Hey Ryan, thanks for the KIP. This will be a really useful feature. Few
questions

1. Once dynamic configs have been loaded and resolved, how can a client
know what values are selected? Will we log the actual resolved values once
they are loaded?

2. Do we want to support the case that we load dynamic configs but also
allow client overrides?

3. Are there other configs we'd like to allow the broker to push up to the
clients? Did we consider making this mechanism generic so the broker could
push any consumer/producer config up to the clients via dynamic configs?

Also, minor note, in ConfigResource#Type, I think the byte values are used
for bit masking somewhere, so 9 won't work -- you'd need to bump up to 16

-David

On Fri, Jul 31, 2020 at 6:06 PM Ryan Dielhenn <rd...@confluent.io>
wrote:

> Thanks Jose,
> Here are the changes to the KIP:
> https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=10&selectedPageVersions=9
>
> 1. Done.
> 2. Done.
> 3. Done.
> 4. Yes metadata.max.age.ms will be use, I updated the KIP with this.
>
> > 5.
> > The Consumer Changes section mentions that the consumer would ask for
> > the dynamic configuration from the broker before joining the group
> > coordinator. This makes sense to me. How about the producer? Should
> > the producer also describe the dynamic configuration before sending
> > acks for the "produce" messages?
>
>
> I do think that the producer should do one synchronous
> DescribeConfigsRequest before producing messages, just for different
> reasons than the consumer. The decision to do this in the consumer was to
> avoid an expensive rebalance operation associated with sending an extra
> join group request.
>
> If this were done for the producer it would be to maintain durability
> semantics by ensuring that the correct dynamic acks value is being used
> before producing messages, however it would not improve performance like it
> does for consumer groups.
>
> > 6.
> > For the Admin Client Changes section, how are DescribeConfigs and
> > IncrementalAlterConfig requests going to get routed by the client to
> > the different brokers in the cluster?
>
> They will all be routed to the least loaded node for the CLIENT resource
> type. I update the KIP with this.
>
> > 7.
> > You mentioned that the producer and the consumer will validate the
> > keys and values received from the broker through DescribeConfigs. Will
> > the ConfigCommand validate any of the keys or values specified in
> > --add-config and --delete-config? Will the broker validate any of the
> > keys or values received in the IncrementalAlterConfigs?
>
> Yes, ConfigCommand will only allow keys that we are providing support for
> to be sent to the broker. The broker will validate the keys as well as the
> values and return an InvalidRequest error code if any key-value pairs are
> invalid.
>
> 8. I removed this line.
>
> On 2020/07/31 18:27:38, Jose Garcia Sancio <js...@confluent.io> wrote:
> > Thanks for the KIP Ryan. Here are some of my observations.
> >
> > 1.
> > > We will also be adding a new client configuration
> enable.dynamic.config to both the producer and consumer that will be true
> by default so that the user has the option to disable this feature.
> >
> > How about?
> > "The Java producer and consumer clients will have a new configuration
> > property `enable.dynamic.config` with a default value of `true`. When
> > this configuration property is true the proposed producer and consumer
> > changes in this KIP are enabled."
> >
> > You can go into more details in the producer and consumer changes
> section.
> >
> > 2.
> > > ... the following order of precedence:
> >
> > I would be a bit more specific here. For example, when the broker
> > handles `DescribeConfigRequest`, it will first use the client config
> > key-values store in `/config/clients/<default>`. If there is a
> > matching `config/clients/<client-id>` then any client config key-value
> > in the matching znode will override the key-values found in
> > `config/clients/<default>`.
> >
> > 3.
> > > These dynamic configs will be stored in zookeeper as the children of
> the ZNode /config/clients
> >
> > How are the client dynamic config key-values stored here? I assume
> > that they are updated based on the content of
> > `IncrementalAlterConfigsRequest`. When is `.../<default>` updated?
> > When is `.../<client-id>` updated?
> >
> > 4.
> > > The interval on which dynamic configs are fetched will be the same
> amount of time as the interval for MetadataRequest which is currently five
> minutes.
> >
> > Will this be hard-coded to 5 minutes? Or is this KIP going to use the
> > same frequency as the producer config `metadata.max.age.ms`? Same
> > question for the "Consumer Changes" section.
> >
> > 5.
> > The Consumer Changes section mentions that the consumer would ask for
> > the dynamic configuration from the broker before joining the group
> > coordinator. This makes sense to me. How about the producer? Should
> > the producer also describe the dynamic configuration before sending
> > acks for the "produce" messages?
> >
> > 6.
> > For the Admin Client Changes section, how are DescribeConfigs and
> > IncrementalAlterConfig requests going to get routed by the client to
> > the different brokers in the cluster?
> >
> > 7.
> > You mentioned that the producer and the consumer will validate the
> > keys and values received from the broker through DescribeConfigs. Will
> > the ConfigCommand validate any of the keys or values specified in
> > --add-config and --delete-config? Will the broker validate any of the
> > keys or values received in the IncrementalAlterConfigs?
> >
> > 8.
> > In rejected ideas the KIP says:
> > > This might make sense for certain configurations such as acks, but
> does not for others such as timeouts.
> >
> > I don't think it makes sense even for acks since the clients of the
> > Java Producer assume that all of the produce messages are sent with
> > the same ack value.
> >
> > --
> > -Jose
> >
>


-- 
David Arthur

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

[Ryan Dielhenn <rd...@confluent.io>]

Thanks Jose, 
Here are the changes to the KIP: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=158869615&selectedPageVersions=10&selectedPageVersions=9

1. Done.
2. Done.
3. Done.
4. Yes metadata.max.age.ms will be use, I updated the KIP with this.

> 5.
> The Consumer Changes section mentions that the consumer would ask for
> the dynamic configuration from the broker before joining the group
> coordinator. This makes sense to me. How about the producer? Should
> the producer also describe the dynamic configuration before sending
> acks for the "produce" messages?


I do think that the producer should do one synchronous DescribeConfigsRequest before producing messages, just for different reasons than the consumer. The decision to do this in the consumer was to avoid an expensive rebalance operation associated with sending an extra join group request. 

If this were done for the producer it would be to maintain durability semantics by ensuring that the correct dynamic acks value is being used before producing messages, however it would not improve performance like it does for consumer groups.

> 6.
> For the Admin Client Changes section, how are DescribeConfigs and
> IncrementalAlterConfig requests going to get routed by the client to
> the different brokers in the cluster?

They will all be routed to the least loaded node for the CLIENT resource type. I update the KIP with this.

> 7.
> You mentioned that the producer and the consumer will validate the
> keys and values received from the broker through DescribeConfigs. Will
> the ConfigCommand validate any of the keys or values specified in
> --add-config and --delete-config? Will the broker validate any of the
> keys or values received in the IncrementalAlterConfigs?

Yes, ConfigCommand will only allow keys that we are providing support for to be sent to the broker. The broker will validate the keys as well as the values and return an InvalidRequest error code if any key-value pairs are invalid.

8. I removed this line.

On 2020/07/31 18:27:38, Jose Garcia Sancio <js...@confluent.io> wrote: 
> Thanks for the KIP Ryan. Here are some of my observations.
> 
> 1.
> > We will also be adding a new client configuration enable.dynamic.config to both the producer and consumer that will be true by default so that the user has the option to disable this feature.
> 
> How about?
> "The Java producer and consumer clients will have a new configuration
> property `enable.dynamic.config` with a default value of `true`. When
> this configuration property is true the proposed producer and consumer
> changes in this KIP are enabled."
> 
> You can go into more details in the producer and consumer changes section.
> 
> 2.
> > ... the following order of precedence:
> 
> I would be a bit more specific here. For example, when the broker
> handles `DescribeConfigRequest`, it will first use the client config
> key-values store in `/config/clients/<default>`. If there is a
> matching `config/clients/<client-id>` then any client config key-value
> in the matching znode will override the key-values found in
> `config/clients/<default>`.
> 
> 3.
> > These dynamic configs will be stored in zookeeper as the children of the ZNode /config/clients
> 
> How are the client dynamic config key-values stored here? I assume
> that they are updated based on the content of
> `IncrementalAlterConfigsRequest`. When is `.../<default>` updated?
> When is `.../<client-id>` updated?
> 
> 4.
> > The interval on which dynamic configs are fetched will be the same amount of time as the interval for MetadataRequest which is currently five minutes.
> 
> Will this be hard-coded to 5 minutes? Or is this KIP going to use the
> same frequency as the producer config `metadata.max.age.ms`? Same
> question for the "Consumer Changes" section.
> 
> 5.
> The Consumer Changes section mentions that the consumer would ask for
> the dynamic configuration from the broker before joining the group
> coordinator. This makes sense to me. How about the producer? Should
> the producer also describe the dynamic configuration before sending
> acks for the "produce" messages?
> 
> 6.
> For the Admin Client Changes section, how are DescribeConfigs and
> IncrementalAlterConfig requests going to get routed by the client to
> the different brokers in the cluster?
> 
> 7.
> You mentioned that the producer and the consumer will validate the
> keys and values received from the broker through DescribeConfigs. Will
> the ConfigCommand validate any of the keys or values specified in
> --add-config and --delete-config? Will the broker validate any of the
> keys or values received in the IncrementalAlterConfigs?
> 
> 8.
> In rejected ideas the KIP says:
> > This might make sense for certain configurations such as acks, but does not for others such as timeouts.
> 
> I don't think it makes sense even for acks since the clients of the
> Java Producer assume that all of the produce messages are sent with
> the same ack value.
> 
> -- 
> -Jose
> 

Re: [DISCUSS] KIP-649: Dynamic Client Configuration

[Jose Garcia Sancio <js...@confluent.io>]

Thanks for the KIP Ryan. Here are some of my observations.

1.
> We will also be adding a new client configuration enable.dynamic.config to both the producer and consumer that will be true by default so that the user has the option to disable this feature.

How about?
"The Java producer and consumer clients will have a new configuration
property `enable.dynamic.config` with a default value of `true`. When
this configuration property is true the proposed producer and consumer
changes in this KIP are enabled."

You can go into more details in the producer and consumer changes section.

2.
> ... the following order of precedence:

I would be a bit more specific here. For example, when the broker
handles `DescribeConfigRequest`, it will first use the client config
key-values store in `/config/clients/<default>`. If there is a
matching `config/clients/<client-id>` then any client config key-value
in the matching znode will override the key-values found in
`config/clients/<default>`.

3.
> These dynamic configs will be stored in zookeeper as the children of the ZNode /config/clients

How are the client dynamic config key-values stored here? I assume
that they are updated based on the content of
`IncrementalAlterConfigsRequest`. When is `.../<default>` updated?
When is `.../<client-id>` updated?

4.
> The interval on which dynamic configs are fetched will be the same amount of time as the interval for MetadataRequest which is currently five minutes.

Will this be hard-coded to 5 minutes? Or is this KIP going to use the
same frequency as the producer config `metadata.max.age.ms`? Same
question for the "Consumer Changes" section.

5.
The Consumer Changes section mentions that the consumer would ask for
the dynamic configuration from the broker before joining the group
coordinator. This makes sense to me. How about the producer? Should
the producer also describe the dynamic configuration before sending
acks for the "produce" messages?

6.
For the Admin Client Changes section, how are DescribeConfigs and
IncrementalAlterConfig requests going to get routed by the client to
the different brokers in the cluster?

7.
You mentioned that the producer and the consumer will validate the
keys and values received from the broker through DescribeConfigs. Will
the ConfigCommand validate any of the keys or values specified in
--add-config and --delete-config? Will the broker validate any of the
keys or values received in the IncrementalAlterConfigs?

8.
In rejected ideas the KIP says:
> This might make sense for certain configurations such as acks, but does not for others such as timeouts.

I don't think it makes sense even for acks since the clients of the
Java Producer assume that all of the produce messages are sent with
the same ack value.

-- 
-Jose