You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Chia-Ping Tsai (Jira)" <ji...@apache.org> on 2025/10/06 14:29:00 UTC
[jira] [Commented] (KAFKA-19390) AbstractIndex#resize() does not release old mmap on Linux
[ https://issues.apache.org/jira/browse/KAFKA-19390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18027568#comment-18027568 ]
Chia-Ping Tsai commented on KAFKA-19390:
----------------------------------------
4.0: https://github.com/apache/kafka/commit/7e27a78f491c9df0bf41538e20e1254bd0c3dfe8
> AbstractIndex#resize() does not release old mmap on Linux
> ---------------------------------------------------------
>
> Key: KAFKA-19390
> URL: https://issues.apache.org/jira/browse/KAFKA-19390
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 3.8.1
> Reporter: Masahiro Mori
> Assignee: Masahiro Mori
> Priority: Major
> Labels: Linux
> Fix For: 3.9.2, 4.2.0, 4.0.2, 4.1.1
>
> Attachments: orig_cpu.png, orig_disk_io.png, orig_network_processor.png, orig_request_handler.png, patched_cpu.png, patched_disk_io.png, patched_network_processor.png, patched_request_handler.png
>
>
> Our kafka broker crashed with the following error:
> {code:java}
> [2025-03-29 09:37:03,218] ERROR Error while appending records to <topic>-<partition> in dir /kafka-logs/data ...
> java.io.IOException: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.mapInternal(FileChannelImpl.java:1127)
> at java.base/sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:1032)
> at org.apache.kafka.storage.internals.log.AbstractIndex.createMappedBuffer(AbstractIndex.java:467)
> at org.apache.kafka.storage.internals.log.AbstractIndex.createAndAssignMmap(AbstractIndex.java:105)
> at org.apache.kafka.storage.internals.log.AbstractIndex.<init>(AbstractIndex.java:83)
> at org.apache.kafka.storage.internals.log.TimeIndex.<init>(TimeIndex.java:65)
> at org.apache.kafka.storage.internals.log.LazyIndex.loadIndex(LazyIndex.java:242)
> at org.apache.kafka.storage.internals.log.LazyIndex.get(LazyIndex.java:179)
> at org.apache.kafka.storage.internals.log.LogSegment.timeIndex(LogSegment.java:146)
> at org.apache.kafka.storage.internals.log.LogSegment.readMaxTimestampAndOffsetSoFar(LogSegment.java:201)
> at org.apache.kafka.storage.internals.log.LogSegment.maxTimestampSoFar(LogSegment.java:211)
> at org.apache.kafka.storage.internals.log.LogSegment.append(LogSegment.java:262)
> at kafka.log.LocalLog.append(LocalLog.scala:417)
> ...
> Caused by: java.lang.OutOfMemoryError: Map failed
> at java.base/sun.nio.ch.FileChannelImpl.map0(Native Method)
> at java.base/sun.nio.ch.FileChannelImpl.mapInternal(FileChannelImpl.java:1124)
> ... 33 more{code}
> We found that kafka process hit the vm.max_map_count limit (which was set to 262144) and most of the mapped entries correspond to deleted index files.
> {code:java}
> > sudo cat /proc/${KAFKA_PID}/maps | grep deleted
> 7d8c5cc00000-7d8c5d600000 rw-s 00000000 08:11 202854769 /kafka-logs/data/topic1-22/00000000332910579773.timeindex.deleted (deleted)
> 7d8c5d800000-7d8c5e200000 rw-s 00000000 08:11 202854768 /kafka-logs/data/topic1-22/00000000332910579773.index.deleted (deleted)
> 7d8c67400000-7d8c67e00000 rw-s 00000000 08:11 202562514 /kafka-logs/data/topic2-116/00000000165968090794.timeindex.deleted (deleted)
> 7d8c68000000-7d8c68a00000 rw-s 00000000 08:11 202562513 /kafka-logs/data/topic2-116/00000000165968090794.index.deleted (deleted)
> 7d8c6d400000-7d8c6de00000 rw-s 00000000 08:11 202596518 /kafka-logs/data/topic2-356/00000000168702579081.timeindex.deleted (deleted)
> 7d8c6e000000-7d8c6ea00000 rw-s 00000000 08:11 202596517 /kafka-logs/data/topic2-356/00000000168702579081.index.deleted (deleted)
> 7d8c71c00000-7d8c72600000 rw-s 00000000 08:11 202798981 /kafka-logs/data/topic3-433/00000000116740630582.timeindex.deleted (deleted)
> 7d8c72800000-7d8c73200000 rw-s 00000000 08:11 202798980 /kafka-logs/data/topic3-433/00000000116740630582.index.deleted (deleted)
> 7d8c77c00000-7d8c78600000 rw-s 00000000 08:11 202754947 /kafka-logs/data/topic3-74/00000000118067749684.timeindex.deleted (deleted)
> 7d8c78800000-7d8c79200000 rw-s 00000000 08:11 202754946 /kafka-logs/data/topic3-74/00000000118067749684.index.deleted (deleted)
> 7d8c79400000-7d8c79e00000 rw-s 00000000 08:11 202813710 /kafka-logs/data/topic2-82/00000000162756700035.timeindex.deleted (deleted)
> 7d8c7a000000-7d8c7aa00000 rw-s 00000000 08:11 202813709 /kafka-logs/data/topic2-82/00000000162756700035.index.deleted (deleted)
> 7d8c7ac00000-7d8c7b600000 rw-s 00000000 08:11 202596526 /kafka-logs/data/topic2-355/00000000169939763750.timeindex.deleted (deleted)
> 7d8c7b800000-7d8c7c200000 rw-s 00000000 08:11 202596525 /kafka-logs/data/topic2-355/00000000169939763750.index.deleted (deleted)
> 7d8c7c400000-7d8c7ce00000 rw-s 00000000 08:11 202562498 /kafka-logs/data/topic2-295/00000000168913981903.timeindex.deleted (deleted)
> 7d8c7d000000-7d8c7da00000 rw-s 00000000 08:11 202562497 /kafka-logs/data/topic2-295/00000000168913981903.index.deleted (deleted)
> 7d8c80c00000-7d8c81600000 rw-s 00000000 08:11 202754939 /kafka-logs/data/topic3-13/00000000115588098896.timeindex.deleted (deleted)
> 7d8c81800000-7d8c82200000 rw-s 00000000 08:11 202754938 /kafka-logs/data/topic3-13/00000000115588098896.index.deleted (deleted)
> 7d8c83c00000-7d8c84600000 rw-s 00000000 08:11 202798989 /kafka-logs/data/topic3-314/00000000118254254601.timeindex.deleted (deleted)
> 7d8c84800000-7d8c85200000 rw-s 00000000 08:11 202798988 /kafka-logs/data/topic3-314/00000000118254254601.index.deleted (deleted)
> ...{code}
> In AbstractIndex.resize(), the old memory mapping is explicitly unmapped on windows or z/OS using safeForceUnmap(), but on Linux the unmapping step is skipped.
> The same issue was originally reported in KAFKA-7442, but the corresponding pull request was never merged.
> We propose that resize() should call safeForceUnmap() on all platforms to prevent stale mappings from lingering.
> ref: [https://speakerdeck.com/lycorptech_jp/20250609b]
> h2. Test
> We tested our patch on the following system environment and found no measurable performance regression.
> *Environment*
> * Kafka: 3.8.1
> * Java: OpenJDK 17
> * OS: Rocky Linux 9
> * Broker: 10 brokers for original, 10 brokers for patched
> ** CPU: Intel Xeon Silver 4310 (2 sockets)
> ** Memory: 250 GB
> ** Disk: HDD
> *Workloads*
> * Peak throughput per broker:
> ** 6K req/s
> ** 85.2 MB/s
> h3. Idle percentage of request handler and network processor threads
> original:
> !orig_request_handler.png|width=1000,height=222!
> !orig_network_processor.png|width=999,height=163!
> patched:
> !patched_request_handler.png|width=997,height=228!
> !patched_network_processor.png|width=1004,height=160!
> h3. Resource load
> original:
> !orig_cpu.png|width=1000,height=227!
> !orig_disk_io.png|width=997,height=168!
> patched:
> !patched_cpu.png|width=999,height=227!
> !patched_disk_io.png|width=1002,height=160!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)