[jira] [Created] (KAFKA-16838) Kafka Connect loads old tasks from removed connectors

["Sergey Ivanov (Jira)" <ji...@apache.org>]

Sergey Ivanov created KAFKA-16838:
-------------------------------------

             Summary: Kafka Connect loads old tasks from removed connectors
                 Key: KAFKA-16838
                 URL: https://issues.apache.org/jira/browse/KAFKA-16838
             Project: Kafka
          Issue Type: Bug
          Components: connect
    Affects Versions: 3.6.1, 3.5.1, 3.8.0
            Reporter: Sergey Ivanov


Hello,

When creating connector we faced an error from one of our ConfigProviders about not existing resource, but we didn't try to set that resource as config value:
{code:java}
[2024-05-24T12:08:24.362][ERROR][request_id= ][tenant_id= ][thread=DistributedHerder-connect-1-1][class=org.apache.kafka.connect.runtime.distributed.DistributedHerder][method=lambda$reconfigureConnectorTasksWithExponentialBackoffRetries$44] [Worker clientId=connect-1, groupId=streaming-service_streaming_service] Failed to reconfigure connector's tasks (local-file-sink), retrying after backoff.
org.apache.kafka.common.config.ConfigException: Could not read properties from file /opt/kafka/provider.properties
 at org.apache.kafka.common.config.provider.FileConfigProvider.get(FileConfigProvider.java:98)
 at org.apache.kafka.common.config.ConfigTransformer.transform(ConfigTransformer.java:103)
 at org.apache.kafka.connect.runtime.WorkerConfigTransformer.transform(WorkerConfigTransformer.java:58)
 at org.apache.kafka.connect.storage.ClusterConfigState.taskConfig(ClusterConfigState.java:181)
 at org.apache.kafka.connect.runtime.AbstractHerder.taskConfigsChanged(AbstractHerder.java:804)
 at org.apache.kafka.connect.runtime.distributed.DistributedHerder.publishConnectorTaskConfigs(DistributedHerder.java:2089)
 at org.apache.kafka.connect.runtime.distributed.DistributedHerder.reconfigureConnector(DistributedHerder.java:2082)
 at org.apache.kafka.connect.runtime.distributed.DistributedHerder.reconfigureConnectorTasksWithExponentialBackoffRetries(DistributedHerder.java:2025)
 at org.apache.kafka.connect.runtime.distributed.DistributedHerder.lambda$null$42(DistributedHerder.java:2038)
 at org.apache.kafka.connect.runtime.distributed.DistributedHerder.runRequest(DistributedHerder.java:2232)
 at org.apache.kafka.connect.runtime.distributed.DistributedHerder.tick(DistributedHerder.java:470)
 at org.apache.kafka.connect.runtime.distributed.DistributedHerder.run(DistributedHerder.java:371)
 at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
 at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
 at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
 at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
 at java.base/java.lang.Thread.run(Thread.java:840)
 {code}
After investigation we found out, that few months ago on that cloud there was the connector with the same name and another value for config provider. Then it was removed, but by some reason when we tried to create connector with the same name months ago AbstractHerder tried to update tasks from our previous connector

As an example I use FileConfigProvider, but actually any ConfigProvider is accceptable which could raise exception if something wrong with config (like result doesn't exist).

We continued our investigation and found the issue https://issues.apache.org/jira/browse/KAFKA-7745 that says Connect doesn't tombstone commit and task messages in the config topic of Kafka. As we remember config topic is `compact` *that means commit and tasks are stored every time (months, years after connector removing)* and impact further connector creations with the same name.

We didn't investigate reasons in ConfigClusterStore and how to avoid that issue, because would {+}like to ask{+}, probably it's better to fix KAFKA-7745 and send tombstones for commit and task messages as connect does for connector and target messages?

I have synthetic TC to reproduce that error if needed.

 

This is linked with https://issues.apache.org/jira/browse/KAFKA-16837 but it's not the same issue.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)