[jira] [Commented] (KAFKA-7897) Invalid use of epoch cache with old message format versions

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-7897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16765215#comment-16765215 ] 

ASF GitHub Bot commented on KAFKA-7897:
---------------------------------------

hachikuji commented on pull request #6253: KAFKA-7897; Do not write epoch start offset for older message format versions
URL: https://github.com/apache/kafka/pull/6253
 
 
   When an older message format is in use, we should disable the leader epoch cache so that we resort to truncation by high watermark. Previously we updated the cache for all versions when a broker became leader for a partition. This can cause large and unnecessary truncations after leader changes because we relied on the presence of _any_ cached epoch in order to tell whether to use the improved truncation logic possible with the OffsetsForLeaderEpoch API.
   
   Note this is a simplified fix than what was merged to trunk in #6232 since the branches have diverged significantly. Rather than removing the epoch cache file, we guard usage of the cache with the record version.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Invalid use of epoch cache with old message format versions
> -----------------------------------------------------------
>
>                 Key: KAFKA-7897
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7897
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Jason Gustafson
>            Assignee: Jason Gustafson
>            Priority: Major
>
> Message format downgrades are not supported, but they generally work as long as broker/clients at least can continue to parse both message formats. After a downgrade, the truncation logic should revert to using the high watermark, but currently we use the existence of any cached epoch as the sole prerequisite in order to leverage OffsetsForLeaderEpoch. This has the effect of causing a massive truncation after startup which causes re-replication.
> I think our options to fix this are to either 1) clear the cache when we notice a downgrade, or 2) forbid downgrades and raise an error.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)