You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "nandini12396 (via GitHub)" <gi...@apache.org> on 2025/12/07 20:43:50 UTC

[PR] KAFKA-19967: Reduce GC pressure in tiered storage read path using direct memory buffers [kafka]

nandini12396 opened a new pull request, #21102:
URL: https://github.com/apache/kafka/pull/21102

   This change addresses high GC pressure by allocating tiered storage fetch buffers in direct (off-heap) memory instead of the JVM heap. When direct memory is exhausted, the system gracefully falls back to heap allocation with a warning.
   
   Problem:
   During tiered storage reads, heap-allocated buffers bypass young generation and go directly to old generation (humongous allocations). Under high read load, these accumulate rapidly and trigger frequent, expensive G1 Old Generation collections causing significant GC pause times.
   
   Solution:
   - Introduced DirectBufferPool that pools direct buffers using WeakReferences, allowing GC to reclaim buffers under memory pressure
   - Modified RemoteLogInputStream to use pooled direct buffers instead of per-request heap allocation
   - Graceful fallback to heap allocation when direct memory is exhausted
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] KAFKA-19967: Reduce GC pressure in tiered storage read path using direct memory buffers [kafka]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21102:
URL: https://github.com/apache/kafka/pull/21102#issuecomment-4257401648

   This PR has been closed since it has not had any activity in 120 days. If you feel like this
   was a mistake, or you would like to continue working on it, please feel free to re-open the 
   PR and ask for a review.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] KAFKA-19967: Reduce GC pressure in tiered storage read path using direct memory buffers [kafka]

Posted by "apoorvmittal10 (via GitHub)" <gi...@apache.org>.
apoorvmittal10 commented on PR #21102:
URL: https://github.com/apache/kafka/pull/21102#issuecomment-3627304724

   Thanks for the PR, I can see new metrics have been introduced in the PR which falls under `Monitoring` and should require a KIP (https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] KAFKA-19967: Reduce GC pressure in tiered storage read path using direct memory buffers [kafka]

Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #21102:
URL: https://github.com/apache/kafka/pull/21102#discussion_r2623330454


##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -379,7 +385,13 @@ public static ConfigDef configDef() {
                         DEFAULT_REMOTE_LIST_OFFSETS_REQUEST_TIMEOUT_MS,
                         atLeast(1),
                         MEDIUM,
-                        REMOTE_LIST_OFFSETS_REQUEST_TIMEOUT_MS_DOC);
+                        REMOTE_LIST_OFFSETS_REQUEST_TIMEOUT_MS_DOC)
+                .define(REMOTE_LOG_DIRECT_BUFFER_POOL_ENABLED_PROP,

Review Comment:
   could you mark the config as internal until the KIP gets approved? 
   
   `.define` -> `.defineInternal`



##########
clients/src/main/java/org/apache/kafka/common/record/RemoteLogInputStream.java:
##########
@@ -17,24 +17,36 @@
 package org.apache.kafka.common.record;
 
 import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.utils.DirectBufferPool;
 import org.apache.kafka.common.utils.Utils;
 
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Closeable;
 import java.io.IOException;
 import java.io.InputStream;
 import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
 
 import static org.apache.kafka.common.record.Records.HEADER_SIZE_UP_TO_MAGIC;
 import static org.apache.kafka.common.record.Records.LOG_OVERHEAD;
 import static org.apache.kafka.common.record.Records.MAGIC_OFFSET;
 import static org.apache.kafka.common.record.Records.SIZE_OFFSET;
 
-public class RemoteLogInputStream implements LogInputStream<RecordBatch> {
+public class RemoteLogInputStream implements LogInputStream<RecordBatch>, Closeable {
+    private static final Logger log = LoggerFactory.getLogger(RemoteLogInputStream.class);

Review Comment:
   logger is unused



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] KAFKA-19967: Reduce GC pressure in tiered storage read path using direct memory buffers [kafka]

Posted by "nandini12396 (via GitHub)" <gi...@apache.org>.
nandini12396 commented on PR #21102:
URL: https://github.com/apache/kafka/pull/21102#issuecomment-3641389891

   Hi @showuon thanks for your review! 
   Pooling heap buffers would reduce allocation frequency but doesn't solve the core issue. Heap buffers still trigger humongous allocation and get scanned by GC every cycle. Direct buffers move the data off-heap entirely into native memory where GC doesn't see it. We also get zero-copy I/O since the data is already in native memory for socket writes.
   
   To share some test results, with heap buffers, GC ran every 100ms with 1.3GB retained after each collection; with direct buffers, GC dropped to every 30-40 seconds with only 325MB retained.
   
   I will remove the metrics changes from this PR so we can focus on the buffer pool implementation. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] KAFKA-19967: Reduce GC pressure in tiered storage read path using direct memory buffers [kafka]

Posted by "nandini12396 (via GitHub)" <gi...@apache.org>.
nandini12396 commented on PR #21102:
URL: https://github.com/apache/kafka/pull/21102#issuecomment-3655619780

   I've updated the PR without the metrics changes to focus on fixing the issue. Please could you take a look and review. Thank you!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] KAFKA-19967: Reduce GC pressure in tiered storage read path using direct memory buffers [kafka]

Posted by "nandini12396 (via GitHub)" <gi...@apache.org>.
nandini12396 commented on code in PR #21102:
URL: https://github.com/apache/kafka/pull/21102#discussion_r2624904985


##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManagerConfig.java:
##########
@@ -379,7 +385,13 @@ public static ConfigDef configDef() {
                         DEFAULT_REMOTE_LIST_OFFSETS_REQUEST_TIMEOUT_MS,
                         atLeast(1),
                         MEDIUM,
-                        REMOTE_LIST_OFFSETS_REQUEST_TIMEOUT_MS_DOC);
+                        REMOTE_LIST_OFFSETS_REQUEST_TIMEOUT_MS_DOC)
+                .define(REMOTE_LOG_DIRECT_BUFFER_POOL_ENABLED_PROP,

Review Comment:
   updated



##########
clients/src/main/java/org/apache/kafka/common/record/RemoteLogInputStream.java:
##########
@@ -17,24 +17,36 @@
 package org.apache.kafka.common.record;
 
 import org.apache.kafka.common.errors.CorruptRecordException;
+import org.apache.kafka.common.utils.DirectBufferPool;
 import org.apache.kafka.common.utils.Utils;
 
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.Closeable;
 import java.io.IOException;
 import java.io.InputStream;
 import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.List;
 
 import static org.apache.kafka.common.record.Records.HEADER_SIZE_UP_TO_MAGIC;
 import static org.apache.kafka.common.record.Records.LOG_OVERHEAD;
 import static org.apache.kafka.common.record.Records.MAGIC_OFFSET;
 import static org.apache.kafka.common.record.Records.SIZE_OFFSET;
 
-public class RemoteLogInputStream implements LogInputStream<RecordBatch> {
+public class RemoteLogInputStream implements LogInputStream<RecordBatch>, Closeable {
+    private static final Logger log = LoggerFactory.getLogger(RemoteLogInputStream.class);

Review Comment:
   removed



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] KAFKA-19967: Reduce GC pressure in tiered storage read path using direct memory buffers [kafka]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #21102:
URL: https://github.com/apache/kafka/pull/21102#issuecomment-4072225670

   This PR is being marked as stale since it has not had any activity in 90 days. If you
   would like to keep this PR alive, please leave a comment asking for a review. If the PR has 
   merge conflicts, update it with the latest from the base branch.
   <p>
   If you are having difficulty finding a reviewer, please reach out on the 
   [mailing list](https://kafka.apache.org/contact).
   <p>
   If this PR is no longer valid or desired, please feel free to close it. If no activity
   occurs in the next 30 days, it will be automatically closed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [PR] KAFKA-19967: Reduce GC pressure in tiered storage read path using direct memory buffers [kafka]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #21102: KAFKA-19967: Reduce GC pressure in tiered storage read path using direct memory buffers
URL: https://github.com/apache/kafka/pull/21102


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org