You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@kafka.apache.org by "Jun Rao (JIRA)" <ji...@apache.org> on 2019/02/21 05:53:00 UTC

[jira] [Resolved] (KAFKA-7283) mmap indexes lazily and skip sanity check for segments below recovery point

     [ https://issues.apache.org/jira/browse/KAFKA-7283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jun Rao resolved KAFKA-7283.
----------------------------
       Resolution: Fixed
    Fix Version/s: 2.3.0

Merged the PR to trunk.

> mmap indexes lazily and skip sanity check for segments below recovery point
> ---------------------------------------------------------------------------
>
>                 Key: KAFKA-7283
>                 URL: https://issues.apache.org/jira/browse/KAFKA-7283
>             Project: Kafka
>          Issue Type: New Feature
>            Reporter: Zhanxiang (Patrick) Huang
>            Assignee: Zhanxiang (Patrick) Huang
>            Priority: Major
>             Fix For: 2.3.0
>
>
> This is a follow-up ticket for KIP-263.
> Currently broker will mmap the index files, read the length as well as the last entry of the file, and sanity check index files of all log segments in the log directory after the broker is started. These operations can be slow because broker needs to open index file and read data into page cache. In this case, the time to restart a broker will increase proportional to the number of segments in the log directory.
> Per the KIP discussion, we think we can skip sanity check for segments below the recovery point since Kafka does not provide guarantee for segments already flushed to disk and sanity checking only index file benefits little when the segment is also corrupted because of disk failure. Therefore, we can make the following changes to improve broker startup time:
>  # Mmap the index file and populate fields of the index file on-demand rather than performing costly disk operations when creating the index object on broker startup.
>  # Skip sanity checks on indexes of segments below the recovery point.
> With these changes, the broker startup time will increase only proportional to the number of partitions in the log directly after cleaned shutdown because only active segments are mmaped and sanity checked.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)