[jira] [Commented] (KAFKA-9373) Improve shutdown performance via lazy accessing the offset and time indices.

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-9373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17066726#comment-17066726 ] 

ASF GitHub Bot commented on KAFKA-9373:
---------------------------------------

ijuma commented on pull request #8346: KAFKA-9373: Reduce shutdown time by avoiding unnecessary loading of indexes
URL: https://github.com/apache/kafka/pull/8346
 
 
   KAFKA-7283 enabled lazy mmap on index files by initializing indices
   on-demand rather than performing costly disk/memory operations when
   creating all indices on broker startup. This helped reducing the startup
   time of brokers. However, segment indices are still created on closing
   segments, regardless of whether they need to be closed or not.
   
   This is a cleaned up version of #7900, which was submitted by @efeg. It
   eliminates unnecessary disk accesses and memory map operations while
   deleting, renaming or closing offset and time indexes.
   
   In a cluster with 31 brokers, where each broker has 13K to 20K segments,
   @efeg and team observed up to 2 orders of magnitude faster LogManager
   shutdown times - i.e. dropping the LogManager shutdown time of each
   broker from 10s of seconds to 100s of milliseconds.
   
   To avoid confusion between `renameTo` and `setFile`, I replaced the
   latter with the more restricted updateParentDir` (it turns out that's
   all we need).
   
   Co-authored-by: Adem Efe Gencer <ag...@linkedin.com>
   Co-authored-by: Ismael Juma <is...@juma.me.uk>
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   
 
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Improve shutdown performance via lazy accessing the offset and time indices.
> ----------------------------------------------------------------------------
>
>                 Key: KAFKA-9373
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9373
>             Project: Kafka
>          Issue Type: Bug
>          Components: log
>    Affects Versions: 2.3.0, 2.4.0, 2.3.1
>            Reporter: Adem Efe Gencer
>            Assignee: Adem Efe Gencer
>            Priority: Major
>
> KAFKA-7283 enabled lazy mmap on index files by initializing indices on-demand rather than performing costly disk/memory operations when creating all indices on broker startup. This helped reducing the startup time of brokers. However, segment indices are still created on closing segments, regardless of whether they need to be closed or not.
>  
> Ideally we should:
>  * Improve shutdown performance via lazy accessing the offset and time indices.
>  * Eliminate redundant disk accesses and memory mapped operations while deleting or renaming files that back segment indices.
>  * Prevent illegal accesses to underlying indices of a closed segment, which would lead to memory leaks due to recreation of the underlying memory mapped objects.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)