[jira] [Commented] (KAFKA-15018) Potential tombstone offsets corruption for exactly-once source connectors

["Chris Egerton (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17725502#comment-17725502 ] 

Chris Egerton commented on KAFKA-15018:
---------------------------------------

One possible fix for this could be to preemptively write tombstone offsets to the global offsets topic before writing any offsets to the per-connector offsets topic, and preserve the existing write logic for all non-tombstone offsets.

Tombstone offsets should be fairly rare and so in the common case, this will have no impact on connector performance or availability. However, when this case is hit, the proposed fix would require two synchronous writes to topics that are potentially hosted on different clusters. This is not ideal, but it's unclear whether a better alternative exists.

> Potential tombstone offsets corruption for exactly-once source connectors
> -------------------------------------------------------------------------
>
>                 Key: KAFKA-15018
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15018
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions: 3.3.0, 3.4.0, 3.3.1, 3.3.2, 3.5.0, 3.4.1
>            Reporter: Chris Egerton
>            Priority: Major
>
> When exactly-once support is enabled for source connectors, source offsets can potentially be written to two different offsets topics: a topic specific to the connector, and the global offsets topic (which was used for all connectors prior to KIP-618 / version 3.3.0).
> Precedence is given to offsets in the per-connector offsets topic, but if none are found for a given partition, then the global offsets topic is used as a fallback.
> When committing offsets, a transaction is used to ensure that source records and source offsets are written to the Kafka cluster targeted by the source connector. This transaction only includes the connector-specific offsets topic. Writes to the global offsets topic take place after writes to the connector-specific offsets topic have completed successfully, and if they fail, a warning message is logged, but no other action is taken.
> Normally, this ensures that, for offsets committed by exactly-once-supported source connectors, the per-connector offsets topic is at least as up-to-date as the global offsets topic, and sometimes even ahead.
> However, for tombstone offsets, we lose that guarantee. If a tombstone offset is successfully written to the per-connector offsets topic, but cannot be written to the global offsets topic, then the global offsets topic will still contain that source offset, but the per-connector topic will not. Due to the fallback-on-global logic used by the worker, if a task requests offsets for one of the tombstoned partitions, the worker will provide it with the offsets present in the global offsets topic, instead of indicating to the task that no offsets can be found.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)