[jira] [Created] (KAFKA-14725) Improve cancellation semantics for source tasks

["Chris Egerton (Jira)" <ji...@apache.org>]

Chris Egerton created KAFKA-14725:
-------------------------------------

             Summary: Improve cancellation semantics for source tasks
                 Key: KAFKA-14725
                 URL: https://issues.apache.org/jira/browse/KAFKA-14725
             Project: Kafka
          Issue Type: Improvement
          Components: KafkaConnect
            Reporter: Chris Egerton


This came about during discussion on [https://github.com/apache/kafka/pull/13208/,] which addressed KAFKA-5756.

 

Right now, we make some effort to disable and shut down tasks that have been scheduled for shutdown but taken longer than the [graceful shutdown timeout period|https://kafka.apache.org/documentation.html#connectconfigs_task.shutdown.graceful.timeout.ms].

The logic for performing this disablement is contained in the {{cancel}} method for the [WorkerTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerTask.java#L129-L136] and its subclasses (at the time of writing, that would be the [AbstractWorkerSourceTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java], [WorkerSourceTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java], [ExactlyOnceWorkerSourceTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/ExactlyOnceWorkerSourceTask.java], and [WorkerSinkTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java] classes). Right now we don't do much to interrupt in-progress operations, which may lead to zombie tasks lying around on a worker that have not yet relinquished resources like Kafka clients, file descriptors, or database connections despite being scheduled for shutdown.

We can and should make the cancellation logic for tasks more stringent, including but not limited to:
 * Interrupting the work thread for the task
 * Interrupting any in-progress offset commits
 * Preemptively shutting down any Kafka clients created for use by the task



--
This message was sent by Atlassian Jira
(v8.20.10#820010)