[jira] [Updated] (KAFKA-14725) Improve cancellation semantics for connector tasks

["Chris Egerton (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-14725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Chris Egerton updated KAFKA-14725:
----------------------------------
    Summary: Improve cancellation semantics for connector tasks  (was: Improve cancellation semantics for source tasks)

> Improve cancellation semantics for connector tasks
> --------------------------------------------------
>
>                 Key: KAFKA-14725
>                 URL: https://issues.apache.org/jira/browse/KAFKA-14725
>             Project: Kafka
>          Issue Type: Improvement
>          Components: KafkaConnect
>            Reporter: Chris Egerton
>            Priority: Major
>
> This came about during discussion on [https://github.com/apache/kafka/pull/13208/,] which addressed KAFKA-5756.
>  
> Right now, we make some effort to disable and shut down tasks that have been scheduled for shutdown but taken longer than the [graceful shutdown timeout period|https://kafka.apache.org/documentation.html#connectconfigs_task.shutdown.graceful.timeout.ms].
> The logic for performing this disablement is contained in the {{cancel}} method for the [WorkerTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerTask.java#L129-L136] and its subclasses (at the time of writing, that would be the [AbstractWorkerSourceTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java], [WorkerSourceTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSourceTask.java], [ExactlyOnceWorkerSourceTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/ExactlyOnceWorkerSourceTask.java], and [WorkerSinkTask|https://github.com/apache/kafka/blob/b9754747d6eaa029c4bb69b073d749ff8df15908/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java] classes). Right now we don't do much to interrupt in-progress operations, which may lead to zombie tasks lying around on a worker that have not yet relinquished resources like Kafka clients, file descriptors, or database connections despite being scheduled for shutdown.
> We can and should make the cancellation logic for tasks more stringent, including but not limited to:
>  * Interrupting the work thread for the task
>  * Interrupting any in-progress offset commits
>  * Preemptively shutting down any Kafka clients created for use by the task



--
This message was sent by Atlassian Jira
(v8.20.10#820010)