[jira] [Commented] (KAFKA-13028) AbstractConfig should allow config provider configuration to use variables referencing other config providers earlier in the list

["Randall Hauch (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-13028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17373715#comment-17373715 ] 

Randall Hauch commented on KAFKA-13028:
---------------------------------------

h2. Is a KIP required?
[KIP-421|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=100829515] defines the current behavior, and it's not clear to me whether this new behavior requires a KIP. KIP-421 says:
{quote}
The KIP proposes on using the  existing AbstractConfig to automatically resolve indirect variables. The {{originals}} configurations will contain both the config provider configs as well as configuration properties. The constructor will first instantiate the ConfigProviders using the config provider configs, then it will find all the variables in the values of the {{originals}} configurations, attempt to resolve the variables using the named ConfigProviders, and then do the normal parsing and validation of the configurations.
{quote}

The KIP doesn't say _how_ it will instantiate and configure the ConfigProvider objects, just that it will do so. It does not say that variables used in the configuration properties for the ConfigProvider may or may not use variables, or even the order in which the ConfigProviders will be instantiated and configured. 

On one hand, the AbstractConfig constructor could instantiate each ConfigProvider and resolve any variables using any ConfigProvider objects it had previously instantiated. One could argue that this new behavior is simply using the same variable resolution logic just more consistently throughout the constructor.

On the other hand, even though this would only alter the behavior, it may be worthwhile to clarify this different behavior in a KIP.

> AbstractConfig should allow config provider configuration to use variables referencing other config providers earlier in the list
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-13028
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13028
>             Project: Kafka
>          Issue Type: Improvement
>          Components: clients, KafkaConnect
>            Reporter: Randall Hauch
>            Priority: Major
>
> When AbstractConfig recognizes config provider properties, it instantiates all of the config providers first and then uses those config providers to resolve any variables in remaining configurations. This means that if you define two config providers with:
> {code}
> config.providers=providerA,providerB
> ...
> {code}
> then the configuration properties for the second provider (e.g., `providerB`) cannot use variables that reference the first provider (e.g., `providerA`). In other words, this is not possible:
> {code}
> config.providers=providerA,providerB
> config.providers.providerA.class=FileConfigProvider
> config.providers.providerB.class=ComplexConfigProvider
> config.providers.providerA.param.client.key=${file:/usr/secrets:complex.client.key}
> config.providers.providerA.param.client.secret=${file:/usr/secrets:complex.client.secret}
> {code}
> This should be possible if the config providers are instantiated and configured in the same order as they appear in the `config.providers` property. The benefit is that it allows another level of indirection so that any secrets required by config provider can be resolved using an earlier simple config provider.
> For example, config providers are often defined in Connect worker configurations to resolve secrets within connector configurations, or to resolve secrets within the worker configuration itself (e.g., producer or consumer secrets). But it would be useful to also be able to resolve the secrets needed by one configuration provider using another configuration provider that is defined earlier in the list.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)