[jira] [Created] (KAFKA-6172) Cache lastEntry in TimeIndex to avoid unnecessary disk access

["Dong Lin (JIRA)" <ji...@apache.org>]

Dong Lin created KAFKA-6172:
-------------------------------

             Summary: Cache lastEntry in TimeIndex to avoid unnecessary disk access
                 Key: KAFKA-6172
                 URL: https://issues.apache.org/jira/browse/KAFKA-6172
             Project: Kafka
          Issue Type: Improvement
            Reporter: Dong Lin
            Assignee: Dong Lin


LogSegment.close() calls timeIndex.maybeAppend(...), which in turns make a number of calls to timeIndex.lastEntry(). Currently timeIndex.lastEntry() involves disk seek operation because it tries to read the content of the last few bytes of the index files on the disk. This slows down the broker shutdown process.

For a given broker of 6k partitions and 19k segments, we find that LogManager.shutdown() takes 15 minutes. The broker is configured to use 10 threads to close log in parallel. According to the thread dump taken while the broker is in the process of LogManager.shutdown(), roughly 5 out of the 10 threads are in RUNNABLE state at TimeIndex.lastEntry(). This suggests that TimeIndex.lastEntry() is very likely costing a lot of shutdown time.

This patch intends to reduce the broker shutdown time by caching the lastEntry in memory so that broker does not have to always read disk to get the lastEntry.





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)