[jira] [Resolved] (KAFKA-12736) KafkaProducer.flush holds onto completed ProducerBatch(s) until flush completes

["Jun Rao (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-12736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jun Rao resolved KAFKA-12736.
-----------------------------
    Fix Version/s: 3.0.0
         Assignee: Lucas Bradstreet
       Resolution: Fixed

> KafkaProducer.flush holds onto completed ProducerBatch(s) until flush completes
> -------------------------------------------------------------------------------
>
>                 Key: KAFKA-12736
>                 URL: https://issues.apache.org/jira/browse/KAFKA-12736
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Lucas Bradstreet
>            Assignee: Lucas Bradstreet
>            Priority: Minor
>             Fix For: 3.0.0
>
>
> When flush is called a copy of the incomplete batches is made. This means that the full ProducerBatch(s) are held in memory until the flush has completed. For batches where the existing memory pool is used this is not as wasteful as the memory will already be returned to the pool, but for non pool memory it can only be GC'd after the flush has completed. Rather than use copyAll we can make a new array with only the produceFuture(s) and await on those.
>  
> {code:java}
> /**
>  * Mark all partitions as ready to send and block until the send is complete
>  */
> public void awaitFlushCompletion() throws InterruptedException {
>  try {
>  for (ProducerBatch batch : this.incomplete.copyAll())
>  batch.produceFuture.await();
>  } finally {
>  this.flushesInProgress.decrementAndGet();
>  }
> }
> {code}
> This may help in cases where the application is already memory constrained and memory usage is slowing progress on completion of the incomplete batches.
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)