[jira] [Commented] (KAFKA-6352) Delay message down-conversion until response is ready to be sent

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16342755#comment-16342755 ] 

ASF GitHub Bot commented on KAFKA-6352:
---------------------------------------

hachikuji opened a new pull request #4484: KAFKA-6352: Down-convert fetched records lazily for more efficent memory usage
URL: https://github.com/apache/kafka/pull/4484
 
 
   This patch adds support for lazy down-conversion of records for older clients. Effectively, it moves down-conversion out of the handler threads and into the network threads. This should improve memory usage since responses that have only been queued for sending will not occupy the memory needed to hold the converted records.
   
   This is largely a refactor which is covered through existing unit and integration tests. I have also added unit tests for the new `LazyDownConvertingRecords` implementation.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Delay message down-conversion until response is ready to be sent
> ----------------------------------------------------------------
>
>                 Key: KAFKA-6352
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6352
>             Project: Kafka
>          Issue Type: Improvement
>            Reporter: Jason Gustafson
>            Assignee: Jason Gustafson
>            Priority: Major
>
> We have observed some users beginning to use the new message format before their clients have been upgraded. As we know, this can cause a lot of memory pressure due to the fact that we down-convert the full response in memory. Currently we do this down-conversion prior to enqueuing the response to be sent by the Processor, which may cause more heap utilization than needed with a steady queue size. 
> A possible improvement is to delay the down-conversion until the response is actually ready to be sent. Even better would be to do the down-conversion in a streaming fashion, converting only as much as is needed at any time. One potential drawback is that this moves the down-conversion into the network threads.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)