You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Haifeng Chen (Jira)" <ji...@apache.org> on 2026/05/06 23:58:00 UTC
[jira] [Updated] (KAFKA-20552) Support log segments larger than 2 GB
[ https://issues.apache.org/jira/browse/KAFKA-20552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Haifeng Chen updated KAFKA-20552:
---------------------------------
Description:
The {{log.segment.bytes}} broker config (and its topic-level synonym {{{}segment.bytes{}}}) is currently defined as {{{}ConfigDef.Type.INT{}}}, capping the maximum segment size at {{Integer.MAX_VALUE}} (2,147,483,647 bytes, ~2 GB). Additionally, the {{.index}} file format stores physical file positions as 4-byte signed integers, which also cannot address beyond ~2 GB.
With modern storage hardware (multi-TB NVMe drives) and high-throughput workloads, the 2 GB cap is increasingly a problem:
* {*}Excessive file handle usage{*}: Each segment needs 4 files ({{{}.log{}}}, {{{}.index{}}}, {{{}.timeindex{}}}, {{{}.txnindex{}}}). A 10 TB partition with 2 GB segments means ~20,000 open files.
* {*}Frequent segment rolls{*}: A topic ingesting 500 MB/s rolls a new segment every ~4 seconds, amplifying index build, flush, and cleaner overhead.
* {*}More log cleaning / compaction work{*}: More segments means more compaction cycles with more small groups.
* {*}Remote storage overhead{*}: Each segment is an individual unit for tiered storage copy/delete operations.
Allowing segments of 4 GB, 8 GB, or larger would significantly reduce these overheads for high-throughput, large-retention workloads.
KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-1333%3A+Support+log+segments+larger+than+2+GB
was:
The {{log.segment.bytes}} broker config (and its topic-level synonym {{{}segment.bytes{}}}) is currently defined as {{{}ConfigDef.Type.INT{}}}, capping the maximum segment size at {{Integer.MAX_VALUE}} (2,147,483,647 bytes, ~2 GB). Additionally, the {{.index}} file format stores physical file positions as 4-byte signed integers, which also cannot address beyond ~2 GB.
With modern storage hardware (multi-TB NVMe drives) and high-throughput workloads, the 2 GB cap is increasingly a problem:
* {*}Excessive file handle usage{*}: Each segment needs 4 files ({{{}.log{}}}, {{{}.index{}}}, {{{}.timeindex{}}}, {{{}.txnindex{}}}). A 10 TB partition with 2 GB segments means ~20,000 open files.
* {*}Frequent segment rolls{*}: A topic ingesting 500 MB/s rolls a new segment every ~4 seconds, amplifying index build, flush, and cleaner overhead.
* {*}More log cleaning / compaction work{*}: More segments means more compaction cycles with more small groups.
* {*}Remote storage overhead{*}: Each segment is an individual unit for tiered storage copy/delete operations.
Allowing segments of 4 GB, 8 GB, or larger would significantly reduce these overheads for high-throughput, large-retention workloads.
> Support log segments larger than 2 GB
> -------------------------------------
>
> Key: KAFKA-20552
> URL: https://issues.apache.org/jira/browse/KAFKA-20552
> Project: Kafka
> Issue Type: Improvement
> Components: core
> Reporter: Haifeng Chen
> Priority: Major
>
> The {{log.segment.bytes}} broker config (and its topic-level synonym {{{}segment.bytes{}}}) is currently defined as {{{}ConfigDef.Type.INT{}}}, capping the maximum segment size at {{Integer.MAX_VALUE}} (2,147,483,647 bytes, ~2 GB). Additionally, the {{.index}} file format stores physical file positions as 4-byte signed integers, which also cannot address beyond ~2 GB.
> With modern storage hardware (multi-TB NVMe drives) and high-throughput workloads, the 2 GB cap is increasingly a problem:
> * {*}Excessive file handle usage{*}: Each segment needs 4 files ({{{}.log{}}}, {{{}.index{}}}, {{{}.timeindex{}}}, {{{}.txnindex{}}}). A 10 TB partition with 2 GB segments means ~20,000 open files.
> * {*}Frequent segment rolls{*}: A topic ingesting 500 MB/s rolls a new segment every ~4 seconds, amplifying index build, flush, and cleaner overhead.
> * {*}More log cleaning / compaction work{*}: More segments means more compaction cycles with more small groups.
> * {*}Remote storage overhead{*}: Each segment is an individual unit for tiered storage copy/delete operations.
> Allowing segments of 4 GB, 8 GB, or larger would significantly reduce these overheads for high-throughput, large-retention workloads.
> KIP https://cwiki.apache.org/confluence/display/KAFKA/KIP-1333%3A+Support+log+segments+larger+than+2+GB
--
This message was sent by Atlassian Jira
(v8.20.10#820010)