[jira] [Created] (KAFKA-6142) Connect worker configurations and connector configurations should accept environment variables

["Randall Hauch (JIRA)" <ji...@apache.org>]

Randall Hauch created KAFKA-6142:
------------------------------------

             Summary: Connect worker configurations and connector configurations should accept environment variables
                 Key: KAFKA-6142
                 URL: https://issues.apache.org/jira/browse/KAFKA-6142
             Project: Kafka
          Issue Type: Improvement
          Components: KafkaConnect
    Affects Versions: 1.0.0
            Reporter: Randall Hauch


Currently, when a worker or connector configuration is parsed, the values are used as-is without any kind of pre-processing before the value is used. It should be possible to define configuration properties such that string literal values or default values can use *_configuration variables_* that reference environment variables and/or system properties, and that these configuration variables are resolved/replaced before the configuration value is used.

I propose doing this enhancement in Kafka client's {{ConfigDef}} by adding a {{ConfigDef.Transformer}} interface:

{code:java}
    /**
     * Transform the configuration value.
     */
    public interface Transformer {
        /**
         * Transform the configuration value.
         * @param name The name of the configuration
         * @param value The value of the configuration
         * @return the preprocessed value
         * @throws ConfigException if the value is invalid.
         */
        Object apply(String name, Object value);
    }
{code}

and then allowing {{Transformer}} implementations to be passed to {{ConfigDef.define(...)}} such all existing signatures are maintained for backward compatibility. By default, the definition would use an identity transform that simply returns the value. The transformers would be called in {{ConfigDef.parseValue(...)}} before the {{parseType(...)}} method is called, and would also be called on the default value if one is provided.

Then, a {{ConfigDef.ReplaceSystemVariables}} implementation would be provided to look in {{String}} values for zero or more variables defined with this EBNF grammar:

{noformat}
'$' '{' varName { ',' varName } [ ':' defaultValue] '}'
{noformat}

where:
* {{varName}} is the name of a Java system property or {{env.}} followed by the name of an environment variable, and 
* {{defaultValue}} specifies the replacement value used when no environment variable or system property is found, and defaults to an empty string. 

The value of the first system property or environment variable resolved is then used to replace the variable expression. This implementation would have trace or debug level logging to describe what it is doing.

For example, the variable {{$\{env.KAFKA_HOME\}}} would replace the variable expression with the value of the {{KAFKA_HOME}} environment variable or an blank string if that variable doesn't exist. 

Likewise, the variable {{$\{foo.prop1,foo.prop2,env.MY_ENV_VAR:value\}}} would be replaced with the value of the {{foo.prop1}} system property if it exists, or with the value of the {{foo.prop2}} system property if it exists, or with the value of the {{MY_ENV_VAR}} environment variable if it exists, or {{value}} if none of the system properties exist.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)