[jira] [Commented] (KAFKA-17212) Segments containing a single message can be incorrectly marked as local only

["Guang Zhao (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-17212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17909012#comment-17909012 ] 

Guang Zhao commented on KAFKA-17212:
------------------------------------

Hi [~guillaumemallet], can I look into this issue if still open?  Thanks.

> Segments containing a single message can be incorrectly marked as local only
> ----------------------------------------------------------------------------
>
>                 Key: KAFKA-17212
>                 URL: https://issues.apache.org/jira/browse/KAFKA-17212
>             Project: Kafka
>          Issue Type: Bug
>          Components: Tiered-Storage
>    Affects Versions: 3.8.0, 3.7.1, 3.9.0
>            Reporter: Guillaume Mallet
>            Priority: Trivial
>
> There is an edge case triggered when a segment containing a single message causes the segment to be considered as local only which skews the deletion process towards deleting more data.
>  
> *This is very unlikely to happen in a real scenario but can happen in tests when segment are rolled manually.* 
> *It could possibly happen when segment are rolled based on time but even then the skew would be minimal.*
> h2. What happens
> In order to delete the right amount of data against the byte retention policy, we first count all the bytes in [buildRetentionSizeData|https://github.com/apache/kafka/blob/09be14bb09dc336f941a7859232094bfb3cb3b96/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1335] function that are breaching {{{}retention.bytes{}}}. In order to do this, the size of each segment is added to the size of the segments present only on the disk {{{}onlyLocalLogSegmentsSize{}}}.
> Listing the segment only present on disk is made through the function [onlyLocalLogSegmentSize|https://github.com/apache/kafka/blob/a0f6e6f816c6ac3fbbc4e0dc503dc43bfacfe6c7/core/src/main/scala/kafka/log/UnifiedLog.scala#L1618-L1619] by adding the size of each segments that have a _baseOffset_ greater or equal compared to {{{}highestOffsetInRemoteStorage{}}}{_}.{_}
> {{highestOffsetInRemoteStorage}} is the highest offset that has been successfully sent to the remote store{_}.{_}
> The _baseOffset_ of a segment is “a [lower bound ({*}inclusive{*}) of the offset in the segment”|https://github.com/apache/kafka/blob/a0f6e6f816c6ac3fbbc4e0dc503dc43bfacfe6c7/storage/src/main/java/org/apache/kafka/storage/internals/log/LogSegment.java#L115]. 
> In the case of a segment with a single message, the baseOffset can be equal to _highestOffsetInRemoteStorage,_ which means that despite the offset being offloaded to the RemoteStorage, we would count that segment as local only.
> This has consequence when counting the bytes to delete as we will count the size of this segment twice in the [buildRetentionSizeData|https://github.com/apache/kafka/blob/09be14bb09dc336f941a7859232094bfb3cb3b96/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1155], once as a segment offloaded in the RemoteStorage and once as a local segment when [onlyLocalSegmentSize|https://github.com/apache/kafka/blob/a0f6e6f816c6ac3fbbc4e0dc503dc43bfacfe6c7/core/src/main/java/kafka/log/remote/RemoteLogManager.java#L1361-L1363] is added. 
> The result is that {{remainingBreachedSize}} will be higher than expected which can lead to more byte deleted than what we would initially expect, up to the size of the segment which is double counted.
> The issue is due to the fact we are using a greater or equal rather than equal. A segment present only locally will have a {{baseOffset}} strictly greater than {{highestOffsetInRemoteStorage.}}
> h2. Reproducing the issue
> The problem is highlighted in the 2 tests added in this [commit |https://github.com/apache/kafka/commit/97af351db517d69a2b37c92861e463a6d0c5cb8f]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)