[GitHub] [kafka] chia7712 commented on pull request #9499: KAFKA-10470: Zstd upgrade and buffering

[GitBox <gi...@apache.org>]

chia7712 commented on pull request #9499:
URL: https://github.com/apache/kafka/pull/9499#issuecomment-716935467


   @yuzawa-san Thanks for great profile and explanation. I'm eager (too) to see this patch to be merged since it not only improves the memory usage but also brings better performance (especially for small message). The following story is just my two cents. The reason to use kafka pool instead of (zstd) recycling buffers is not only about performance but also other improvements. For example, synchronization issue (that was discussed in https://github.com/luben/zstd-jni/issues/131). Also, on our kafka (private branch), we introduces some sensors to (kafka) pool to get oom/duration metrics to trace compression status. In short, it would be great to introduce kafka pool to compression. (We don't use snappy since it is not best compression on our use cases)
   
   At any rate, I'm +1 to merge this PR as it does bring significant improvement. We can address kafka pool in next PR.
   
   @yuzawa-san thank you (again) that your effort on both zstd and kafka!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org