[jira] [Commented] (KAFKA-15609) Corrupted index uploaded to remote tier

["Divij Vaidya (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-15609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17782148#comment-17782148 ] 

Divij Vaidya commented on KAFKA-15609:
--------------------------------------

Hey folks

I did some more digging on this and I am not convinced that this is not a bug but I don't have proof right now.

My understanding is as follow:
1. MMap will read/write from page cache in the OS. 

2. MMap flushed data to page cache "eventually" after writes. Note that MappedByteBuffer.force() method forces a flush of MMap to page cache.
3. In Kafka, we flush indexes leading to MappedByteBuffer.force() on segment roll, but this is performed asynchronously in a separate thread.

Now consider the situation:

1. Data is written to segment and index.
2. Segment rolls and schedules a flush.
3. Since segment is closed, RSM will consider it for copying to remote.
4. RSM will read from page cache which is guaranteed to give correct (even non-flushed / dirty) data for segment but it may give out of date data for indexes. 

Result: Indexes uploaded to remote tier will not contain all data that is expected in it.

I will keep this ticket closed until I have some reproducer but I would solicit your thoughts on this.

Proposal: To fix this, we should call AbstractIndex.flush() in RLM before passing the index to RSM for upload.

> Corrupted index uploaded to remote tier
> ---------------------------------------
>
>                 Key: KAFKA-15609
>                 URL: https://issues.apache.org/jira/browse/KAFKA-15609
>             Project: Kafka
>          Issue Type: Bug
>          Components: Tiered-Storage
>    Affects Versions: 3.6.0
>            Reporter: Divij Vaidya
>            Priority: Minor
>
> While testing Tiered Storage, we have observed corrupt indexes being present in remote tier. One such situation is covered here at https://issues.apache.org/jira/browse/KAFKA-15401. This Jira presents another such possible case of corruption.
> Potential cause of index corruption:
> We want to ensure that the file we are passing to RSM plugin contains all the data which is present in MemoryByteBuffer i.e. we should have flushed the MemoryByteBuffer to the file using force(). In Kafka, when we close a segment, indexes are flushed asynchronously [1]. Hence, it might be possible that when we are passing the file to RSM, the file doesn't contain flushed data. Hence, we may end up uploading indexes which haven't been flushed yet. Ideally, the contract should enforce that we force flush the content of MemoryByteBuffer before we give the file for RSM. This will ensure that indexes are not corrupted/incomplete.
> [1] [https://github.com/apache/kafka/blob/4150595b0a2e0f45f2827cebc60bcb6f6558745d/core/src/main/scala/kafka/log/UnifiedLog.scala#L1613] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)