[jira] [Created] (KAFKA-14871) Kafka Connect - TTL not respected for Config Provider-provided connector configurations

["Dave Sloan (Jira)" <ji...@apache.org>]

Dave Sloan created KAFKA-14871:
----------------------------------

             Summary: Kafka Connect - TTL not respected for Config Provider-provided connector configurations
                 Key: KAFKA-14871
                 URL: https://issues.apache.org/jira/browse/KAFKA-14871
             Project: Kafka
          Issue Type: Bug
          Components: KafkaConnect
    Affects Versions: 3.3.2
            Reporter: Dave Sloan


When defining a configuration provider using environment variables (eg via Docker), then a reload is scheduled according to the TTL of the returned configuration.

 

Here is an example:

 
|Environment Variable|Value|
|CONNECT_CONFIG_PROVIDERS_VAULT_PARAM_VAULT_ENGINE_VERSION|2|
|CONNECT_CONFIG_PROVIDERS_VAULT_PARAM_VAULT_TOKEN|9c08104f-98b7-4bce-ab86-4a6c63897ec4|
|CONNECT_CONFIG_PROVIDERS|vault|
|CONNECT_CONFIG_PROVIDERS_VAULT_PARAM_DEFAULT_TTL|30000|
|CONNECT_CONFIG_PROVIDERS_VAULT_PARAM_VAULT_ADDR|http://vault:8200|
|CONNECT_CONFIG_PROVIDERS_VAULT_PARAM_VAULT_AUTH_METHOD|token|
|CONNECT_CONFIG_PROVIDERS_VAULT_PARAM_FILE_WRITE|false|
|CONNECT_CONFIG_PROVIDERS_VAULT_CLASS|io.lenses.connect.secrets.providers.VaultSecretProvider|

 

 
{code:java}
{
    "name": "testSink",
    "config": {
        "topics": "vaultTest",
        "name": "testSink",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "test.sink.secret.value": "${vault:rotate-test/myVaultSecretPath:myVaultSecretKey}",
        "value.converter": "org.apache.kafka.connect.storage.StringConverter",
        "tasks.max": 1,
    }
}{code}
And this is what we see in the logs:
{code:java}
STDOUT: [2023-03-30 07:46:43,204] INFO Scheduling a restart of connector testSink in 4908 ms (org.apache.kafka.connect.runtime.WorkerConfigTransformer){code}

However when we try and achieve the same via connector properties, no restart is scheduled

 

 

 
{code:java}
{
    "name": "testSink",
    "config": {
        "topics": "vaultTest",
        "name": "testSink",
        "key.converter": "org.apache.kafka.connect.storage.StringConverter",
        "value.converter": "org.apache.kafka.connect.storage.StringConverter",
        "tasks.max": 1,
        "connector.class": "io.lenses.connect.secrets.test.TestSinkConnector",
        "test.sink.secret.value": "${vault:rotate-test/myVaultSecretPath:myVaultSecretKey}",
        "config.providers": "vault",
        "config.providers.vault.class": "io.lenses.connect.secrets.providers.VaultSecretProvider",
        "config.providers.vault.param.vault.engineversion": 2,
        "config.providers.vault.param.vault.token": "9c08104f-98b7-4bce-ab86-4a6c63897ec4",
        "config.providers.vault.param.default.ttl": 30000,
        "config.providers.vault.param.vault.addr": "http://vault:8200",
        "config.providers.vault.param.vault.auth.method": "token",
        "config.providers.vault.param.file.write": false
    }
}{code}
 

 

Upon looking deeper into the code I can see that on line 239 of AbstractConfig

[https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/common/config/AbstractConfig.java#L539]

 
{code:java}
ConfigTransformerResult result = configTransformer.transform(indirectVariables);{code}
The result contains the TTLs.  However these are not used.

 

Expectation:

TTLs should be used to schedule a restart of the connector so that the behaviour is the same as if using environment properties.

 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)