[GitHub] [druid] bananaaggle edited a comment on issue #11816: Zstd version is incompatible with the version of Zstd used by the Kafka Indexing service

[GitBox <gi...@apache.org>]

bananaaggle edited a comment on issue #11816:
URL: https://github.com/apache/druid/issues/11816#issuecomment-1000195848


   Hi! I know what happened! I used to test ZSTD on Kafka. It was caused by commit [10123](https://github.com/apache/kafka/pull/10123/files). You can see ZstdFactory.java, there is `return new BufferedOutputStream(new ZstdOutputStreamNoFinalizer(buffer, RecyclingBufferPool.INSTANCE), 16 * 1024);` and
   `return new BufferedInputStream(new ZstdInputStreamNoFinalizer(new ByteBufferInputStream(buffer),
                   RecyclingBufferPool.INSTANCE), 16 * 1024);` It used a reused buffer to reduce GC times. This buffer is a new feature in zstd 1.4.5-8. This commit is released in Kafka 2.8.0. If Druid uses any Kafka clients after 2.8.0, it should update zstd to 1.4.5-8.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org