You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Stephan Lachowsky (JIRA)" <ji...@apache.org> on 2014/06/03 03:06:02 UTC
[jira] [Comment Edited] (KAFKA-1456) Add LZ4 and LZ4C as a
compression codec
[ https://issues.apache.org/jira/browse/KAFKA-1456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14016113#comment-14016113 ]
Stephan Lachowsky edited comment on KAFKA-1456 at 6/3/14 1:05 AM:
------------------------------------------------------------------
Hello all,
First off, I would really like to see this functionality get added, but I'd like to make sure the wire protocol is done properly, before it is picked by a release and there is no going back.
Here are, in my estimation, the issues with the current implementation:
- LZ4 and LZ4HC generate the same output format, so they shouldn't have different compression codec enums... this is a producer configuration issue only. The same caveat applies to the compressor "level" parameter, it is basically a producer CPU/compression tradeoff.
- The LZ4Block format used by net.jpountz.lz4.LZ4BlockInputStream and net.jpountz.lz4.LZ4BlockOutputStream is a block based format that isn't well documented outside of the java code. I would recommend that something documented be used, like the format defined by the LZ4 author: http://fastcompression.blogspot.com/2013/04/lz4-streaming-format-final.html
was (Author: stephanl):
Hello all,
First off, I would really like to see this functionality get added, but I'd like to make sure the wire protocol is done properly, before it is picked by a release and there is no going back.
Here are, in my estimation, the issues with the current implementation:
- LZ4 and LZ4HC generate the same output format, so they shouldn't have different compression codec enums... this is a producer configuration issue only. The same caveat applies to the compressor "level" parameter, it is basically a producer CPU/compression tradeoff.
- The LZ4Block format used by net.jpountz.lz4.LZ4BlockInputStream and net.jpountz.lz4.LZ4BlockOutputStream is a block based format that isn't well documented outside of the java code. I would recommend that something documented by used, like the format defined by the LZ4 author: http://fastcompression.blogspot.com/2013/04/lz4-streaming-format-final.html
> Add LZ4 and LZ4C as a compression codec
> ---------------------------------------
>
> Key: KAFKA-1456
> URL: https://issues.apache.org/jira/browse/KAFKA-1456
> Project: Kafka
> Issue Type: Improvement
> Reporter: Joe Stein
> Labels: newbie
> Fix For: 0.8.2
>
> Attachments: KAFKA-1456.patch, KAFKA-1456_2014-05-19_15:01:10.patch, KAFKA-1456_2014-05-19_16:39:01.patch, KAFKA-1456_2014-05-19_18:19:32.patch, KAFKA-1456_2014-05-19_23:24:27.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)