You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Ismael Juma (JIRA)" <ji...@apache.org> on 2017/01/25 16:36:26 UTC
[jira] [Commented] (KAFKA-3300) Calculate the initial/max size of
offset index files and reduce the memory footprint for memory mapped index
files.
[ https://issues.apache.org/jira/browse/KAFKA-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15838069#comment-15838069 ]
Ismael Juma commented on KAFKA-3300:
------------------------------------
This was deemed not necessary if I recall correctly. If that's right, shall we close this JIRA?
> Calculate the initial/max size of offset index files and reduce the memory footprint for memory mapped index files.
> -------------------------------------------------------------------------------------------------------------------
>
> Key: KAFKA-3300
> URL: https://issues.apache.org/jira/browse/KAFKA-3300
> Project: Kafka
> Issue Type: Improvement
> Affects Versions: 0.9.0.1
> Reporter: Jiangjie Qin
> Assignee: Jiangjie Qin
> Fix For: 0.10.2.0
>
>
> Currently the initial/max size of offset index file is configured by {{log.index.max.bytes}}. This will be the offset index file size for active log segment until it rolls out.
> Theoretically, we can calculate the upper bound of offset index size using the following formula:
> {noformat}
> log.segment.bytes / index.interval.bytes * 8
> {noformat}
> With default setting the bytes needed for an offset index size is 1GB / 4K * 8 = 2MB. And the default log.index.max.bytes is 10MB.
> This means we are over-allocating at least 8MB on disk and mapping it to memory.
> We can probably do the following:
> 1. When creating a new offset index, calculate the size using the above formula,
> 2. If the result in (1) is greater than log.index.max.bytes, we allocate log.index.max.bytes instead.
> This should be able to significantly save memory if a broker has a lot of partitions on it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)