[jira] [Created] (KAFKA-16069) Source Tasks re-transform records after Retriable exceptions

["Greg Harris (Jira)" <ji...@apache.org>]

Greg Harris created KAFKA-16069:
-----------------------------------

             Summary: Source Tasks re-transform records after Retriable exceptions
                 Key: KAFKA-16069
                 URL: https://issues.apache.org/jira/browse/KAFKA-16069
             Project: Kafka
          Issue Type: Bug
          Components: KafkaConnect
            Reporter: Greg Harris


In the SinkTask, records which fail to be delivered to the task#put with a Retriable exception are re-delivered on the next iteration. The SourceTask follows a similar pattern, where records which fail to be delivered to Producer#send with a Retriable exception are retried.

However, the SinkTask accumulates the post-transform records, and does not recompute the transformations over again after a retriable exception. The SourceTask does not accumulate ProducerRecords, and instead recomputes the transformations and converters starting from the pre-transformation AbstractWorkerSourceTask#toSend list.

This means that stateful transformations and converters may see the records rewind, without any indication that the records are the same. For stateless transformations and converters, this means that redundant computation is performed that may be better allocated to other tasks.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)