You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by dongjinleekr <gi...@git.apache.org> on 2016/12/16 09:52:18 UTC
[GitHub] kafka pull request #2267: KAFKA-4514: Add Codec for ZStandard Compression
GitHub user dongjinleekr opened a pull request:
https://github.com/apache/kafka/pull/2267
KAFKA-4514: Add Codec for ZStandard Compression
Hello. This PR resolves [KAFKA-4514: Add Codec for ZStandard Compression](https://issues.apache.org/jira/browse/KAFKA-4514). Please have a look when you are free. Since I am a total newbie of Apache Kafka, feel free to point out the deficiencies.
Add to the feature itself, I have a question: **Should we support an option for ZStandard compression level?**
According to [ZStandard official documentation](https://github.com/facebook/zstd/blob/dev/lib/zstd.h), it supports compression level of 1 ~ 22. Because of that, [Hadoop added a new configuration option named "io.compression.codec.zstd.level", whose default value is 3](https://issues.apache.org/jira/browse/HADOOP-13578). In this PR, I configured the compression level to 1 as a temporary one but wondering following problems:
- Should we provide a configurable option?
- Would it better to change the default value, from 1 to another one?
I am looking forward to your advice. Thanks.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/dongjinleekr/kafka feature/KAFKA-4514
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2267.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2267
----
commit aa751148b75cac9062b13b1115738a595556dd39
Author: Lee Dongjin <do...@apache.org>
Date: 2016-12-16T08:50:20Z
Add zstandard dependency
commit 22fe8d64ffbc2d2725bdfb2166f1ecdc75a96c8d
Author: Lee Dongjin <do...@apache.org>
Date: 2016-12-16T08:51:20Z
Add ZStdCompressionCodec
commit ab535e9f0a18132e35dcbef032dcde320204e12b
Author: Lee Dongjin <do...@apache.org>
Date: 2016-12-16T08:59:31Z
Add CompressionType.ZSTD and update ProducerConfig, MemoryRecordsBuilder
commit 87c3e803ab30344a9834c19075a3ebe617cf5787
Author: Lee Dongjin <do...@apache.org>
Date: 2016-12-16T09:00:55Z
Update CompressionCodec & CompressionFactory
commit 18af6439f1fad6811783da43d0a0f475164853c8
Author: Lee Dongjin <do...@apache.org>
Date: 2016-12-16T09:03:58Z
Add tests for ZStandard
commit 41ae0ac1ac809822712eb609726db00d00e0f36d
Author: Lee Dongjin <do...@apache.org>
Date: 2016-12-16T09:14:28Z
Add documentation comments
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---