[jira] [Commented] (KAFKA-5247) Consumer GroupCoordinator should continue to materialize committed offsets in offset order even for transactional offset commits

["Apurva Mehta (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-5247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16010959#comment-16010959 ] 

Apurva Mehta commented on KAFKA-5247:
-------------------------------------

Note that this is only an issue when we mix offset commits from the consumer and from the transactional producer, which may conceivably be the case during an upgrade. If we don't fix this it is possible for the offset commits to change after compaction.

For instance, if a transactional offset commit has position 100 in the log, and consumer offset commit has position 101. After the transaction is committed, the offset commit at position 100 will be materialized. If the coordinator changes, the logic stays the same and the offset at position 100 will continue to be materialized. However,  if compaction kicks in, the offset at position 101 will now be materialized. This bug is about fixing this inconsistency so that the same offset commit is always materialized in a stable state.

> Consumer GroupCoordinator should continue to materialize committed offsets in offset order even for transactional offset commits
> --------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-5247
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5247
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Apurva Mehta
>              Labels: exactly-once
>             Fix For: 0.11.0.0
>
>
> In the TxnOffsetCommit patch, we thought it was ok for the group coordinator to use "transaction order" semantics when updating the cache, but we weren't thinking about the log cleaner.
> The log cleaner uses offset order when cleaning which means that the key with the largest offset always wins. So if we use transaction order when dynamically updating the cache, we will get different results from when we're loading the cache (even if the loading logic also uses transaction order).
> The fix should be straightforward: we need to remember the offset in the offsets topic of the offset that we cache. Then we only update it if the new entry has a higher offset.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)