[jira] [Commented] (KAFKA-14038) Optimize calculation of size for log in remote tier

["Mickael Maison (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-14038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17709286#comment-17709286 ] 

Mickael Maison commented on KAFKA-14038:
----------------------------------------

Removing from 3.5.0 as the KIP as the vote has not passed yet

> Optimize calculation of size for log in remote tier
> ---------------------------------------------------
>
>                 Key: KAFKA-14038
>                 URL: https://issues.apache.org/jira/browse/KAFKA-14038
>             Project: Kafka
>          Issue Type: Improvement
>          Components: core
>            Reporter: Divij Vaidya
>            Assignee: Divij Vaidya
>            Priority: Major
>              Labels: needs-kip
>             Fix For: 3.5.0
>
>
> {color:#24292f}As per the Tiered Storage feature introduced in [KIP-405|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage], users can configure the retention of remote tier based on time, by size, or both. The work of computing the log segments to be deleted based on the retention config is [owned by RemoteLogManager|https://cwiki.apache.org/confluence/display/KAFKA/KIP-405%3A+Kafka+Tiered+Storage#KIP405:KafkaTieredStorage-1.RemoteLogManager(RLM)ThreadPool] (RLM).{color}
> {color:#24292f}To compute remote segments eligible for deletion based on retention by size config, {color}RLM needs to compute the {{total_remote_log_size}} i.e. the total size of logs available in the remote tier for that topic-partition. RLM could use the {{RemoteLogMetadataManager.listRemoteLogSegments()}} to fetch metadata for all the remote segments and then aggregate the segment sizes by using {{{}RemoteLogSegmentMetadata.segmentSizeInBytes(){}}}to find the total log size stored in the remote tier.
> The above method involves iterating through all metadata of all the segments i.e. O({color:#24292f}num_remote_segments{color}) on each execution of RLM thread. {color:#24292f}Since the main feature of tiered storage is storing a large amount of data, we expect num_remote_segments to be large and a frequent linear scan could be expensive (depending on the underlying storage used by RemoteLogMetadataManager).
> Segment offloads and segment deletions are run together in the same task and a fixed size thread pool is shared among all topic-partitions. A slow logic for calculation of total_log_size could result in the loss of availability as demonstrated in the following scenario:{color}
>  # {color:#24292f}Calculation of total_size is slow and the threads in the thread pool are busy with segment deletions{color}
>  # Segment offloads are delayed (since they run together with deletions)
>  # Local disk fills up, since local deletion requires the segment to be offloaded
>  # If local disk is completely full, Kafka fails
> Details are in KIP - [https://cwiki.apache.org/confluence/display/KAFKA/KIP-852%3A+Optimize+calculation+of+size+for+log+in+remote+tier] 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)