You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Uddhav Arote <ar...@gmail.com> on 2018/02/14 05:28:56 UTC
Compression in Kafka
Hi Kafka users,
I am trying to understand the behavior of compression in Kafka. Consider a
scenario, where producer sets compression.codec "snappy" and broker's
compression.code "lz4"?
In this scenario, what is the behavior of the compression?
As far as I have understood is the following,
The messages compressed by the producer are wrapped in the wrapper message
and send to the broker. If the broker compression.codec is "producer", the
message is written as is to the log.
In the code,
https://github.com/apache/kafka/blob/962bc638f9c2ab249e5008a587ee78e3ba35fcb9/core/src/main/scala/kafka/log/LogValidator.scala#L218
what I understand is that if the producer and broker codecs are not same,
then the compression should happen again.
But I am not sure about this. Can somebody tell me how this works?
Thanks,
Uddhav
Re: Compression in Kafka
Posted by Uddhav Arote <ar...@gmail.com>.
Oh, that makes sense.
So, to summarize
1. producer and broker compression codecs different: the broker
decompresses and re-compresses the message batches
2. producer and broker compression codecs same: (lz4 & lz4) -- retain the
producer compression **
3. producer and broker compression codec (lz4 and 'producer') -- retain the
original compression codec
Case 2 and 3 same process? **
What is the need to recompress in case of different codecs?
Thanks
On Wed, Feb 14, 2018 at 8:53 PM, Manikumar <ma...@gmail.com>
wrote:
> It is not double compression. When I say re-compression, brokers
> decompress the messages and compress again with
> new codec.
>
> On Wed, Feb 14, 2018 at 5:18 PM, Uddhav Arote <ar...@gmail.com>
> wrote:
>
> > Thanks.
> >
> > I am using console-producer with following settings with lz4 broker
> > compression codec
> > 1. None producer compression codec
> > 2. Snappy producer compression codec
> > 3. lz4 producer compression codec
> >
> > I send a 354 Byte message with each of the above settings. However, I do
> > not see any kind of double compression happening, when the producer and
> > broker compression codecs are different
> >
> > *Output 1:*
> > offset: 0 position: 0 CreateTime: 1518607686194 isvalid: true
> payloadsize:
> > 61 magic: 1 compresscodec: LZ4CompressionCodec crc: 3693540371 payload:
> > xxxx compressed 354Byte message
> >
> > using --deep-iteration
> > offset: 0 position: 0 CreateTime: 1518607686194 isvalid: true
> payloadsize:
> > 354 magic: 1 compresscodec: NoCompressionCodec crc: 4190573446 payload:
> 354
> > byte message
> >
> >
> > *Output 2:*offset: 7 position: 517 CreateTime: 1518608039723 isvalid:
> true
> > payloadsize: 61 magic: 1 compresscodec: LZ4CompressionCodec crc:
> 4075439033
> > payload: compressed 354B message
> >
> > using --deep-iteration
> > offset: 7 position: 517 CreateTime: 1518608039723 isvalid: true
> > payloadsize: 354 magic: 1 compresscodec: NoCompressionCodec crc:
> 4061704088
> > payload: Same 354B message
> >
> > *Output 3:*
> > offset: 11 position: 883 CreateTime: 1518608269618 isvalid: true
> > payloadsize: 61 magic: 1 compresscodec: LZ4CompressionCodec crc:
> 981370250
> > payload: compressed 354B message
> >
> > using --deep-iteration
> > offset: 11 position: 883 CreateTime: 1518608269618 isvalid: true
> > payloadsize: 354 magic: 1 compresscodec: NoCompressionCodec crc:
> 468622988
> > payload: same 354B message
> >
> > Please note the compression codecs in the --deep-iteration case,
> > Case 1 is OK, but in case 2 shouldn't it be SnappyCompression and 3 may
> be
> > LZ4Compression
> >
> > Or is it visible when the message batch into large batches?
> >
> > Thanks
> > Uddhav
> >
> > On Wed, Feb 14, 2018 at 6:05 PM, Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > If the broker "compression.type" is "producer", then the broker
> retains
> > > the original compression codec set by the producer.
> > > If the producer and broker codecs are different, then broker
> recompress
> > > the data using broker "compression.type".
> > >
> > > On Wed, Feb 14, 2018 at 10:58 AM, Uddhav Arote <ar...@gmail.com>
> > > wrote:
> > >
> > > > Hi Kafka users,
> > > >
> > > > I am trying to understand the behavior of compression in Kafka.
> > Consider
> > > a
> > > > scenario, where producer sets compression.codec "snappy" and broker's
> > > > compression.code "lz4"?
> > > > In this scenario, what is the behavior of the compression?
> > > >
> > > > As far as I have understood is the following,
> > > > The messages compressed by the producer are wrapped in the wrapper
> > > message
> > > > and send to the broker. If the broker compression.codec is
> "producer",
> > > the
> > > > message is written as is to the log.
> > > > In the code,
> > > >
> > > > https://github.com/apache/kafka/blob/962bc638f9c2ab249e5008a587ee78
> > > > e3ba35fcb9/core/src/main/scala/kafka/log/LogValidator.scala#L218
> > > >
> > > >
> > > > what I understand is that if the producer and broker codecs are not
> > same,
> > > > then the compression should happen again.
> > > >
> > > > But I am not sure about this. Can somebody tell me how this works?
> > > >
> > > > Thanks,
> > > > Uddhav
> > > >
> > >
> >
>
Re: Compression in Kafka
Posted by Manikumar <ma...@gmail.com>.
It is not double compression. When I say re-compression, brokers
decompress the messages and compress again with
new codec.
On Wed, Feb 14, 2018 at 5:18 PM, Uddhav Arote <ar...@gmail.com> wrote:
> Thanks.
>
> I am using console-producer with following settings with lz4 broker
> compression codec
> 1. None producer compression codec
> 2. Snappy producer compression codec
> 3. lz4 producer compression codec
>
> I send a 354 Byte message with each of the above settings. However, I do
> not see any kind of double compression happening, when the producer and
> broker compression codecs are different
>
> *Output 1:*
> offset: 0 position: 0 CreateTime: 1518607686194 isvalid: true payloadsize:
> 61 magic: 1 compresscodec: LZ4CompressionCodec crc: 3693540371 payload:
> xxxx compressed 354Byte message
>
> using --deep-iteration
> offset: 0 position: 0 CreateTime: 1518607686194 isvalid: true payloadsize:
> 354 magic: 1 compresscodec: NoCompressionCodec crc: 4190573446 payload: 354
> byte message
>
>
> *Output 2:*offset: 7 position: 517 CreateTime: 1518608039723 isvalid: true
> payloadsize: 61 magic: 1 compresscodec: LZ4CompressionCodec crc: 4075439033
> payload: compressed 354B message
>
> using --deep-iteration
> offset: 7 position: 517 CreateTime: 1518608039723 isvalid: true
> payloadsize: 354 magic: 1 compresscodec: NoCompressionCodec crc: 4061704088
> payload: Same 354B message
>
> *Output 3:*
> offset: 11 position: 883 CreateTime: 1518608269618 isvalid: true
> payloadsize: 61 magic: 1 compresscodec: LZ4CompressionCodec crc: 981370250
> payload: compressed 354B message
>
> using --deep-iteration
> offset: 11 position: 883 CreateTime: 1518608269618 isvalid: true
> payloadsize: 354 magic: 1 compresscodec: NoCompressionCodec crc: 468622988
> payload: same 354B message
>
> Please note the compression codecs in the --deep-iteration case,
> Case 1 is OK, but in case 2 shouldn't it be SnappyCompression and 3 may be
> LZ4Compression
>
> Or is it visible when the message batch into large batches?
>
> Thanks
> Uddhav
>
> On Wed, Feb 14, 2018 at 6:05 PM, Manikumar <ma...@gmail.com>
> wrote:
>
> > If the broker "compression.type" is "producer", then the broker retains
> > the original compression codec set by the producer.
> > If the producer and broker codecs are different, then broker recompress
> > the data using broker "compression.type".
> >
> > On Wed, Feb 14, 2018 at 10:58 AM, Uddhav Arote <ar...@gmail.com>
> > wrote:
> >
> > > Hi Kafka users,
> > >
> > > I am trying to understand the behavior of compression in Kafka.
> Consider
> > a
> > > scenario, where producer sets compression.codec "snappy" and broker's
> > > compression.code "lz4"?
> > > In this scenario, what is the behavior of the compression?
> > >
> > > As far as I have understood is the following,
> > > The messages compressed by the producer are wrapped in the wrapper
> > message
> > > and send to the broker. If the broker compression.codec is "producer",
> > the
> > > message is written as is to the log.
> > > In the code,
> > >
> > > https://github.com/apache/kafka/blob/962bc638f9c2ab249e5008a587ee78
> > > e3ba35fcb9/core/src/main/scala/kafka/log/LogValidator.scala#L218
> > >
> > >
> > > what I understand is that if the producer and broker codecs are not
> same,
> > > then the compression should happen again.
> > >
> > > But I am not sure about this. Can somebody tell me how this works?
> > >
> > > Thanks,
> > > Uddhav
> > >
> >
>
Re: Compression in Kafka
Posted by Uddhav Arote <ar...@gmail.com>.
Thanks.
I am using console-producer with following settings with lz4 broker
compression codec
1. None producer compression codec
2. Snappy producer compression codec
3. lz4 producer compression codec
I send a 354 Byte message with each of the above settings. However, I do
not see any kind of double compression happening, when the producer and
broker compression codecs are different
*Output 1:*
offset: 0 position: 0 CreateTime: 1518607686194 isvalid: true payloadsize:
61 magic: 1 compresscodec: LZ4CompressionCodec crc: 3693540371 payload:
xxxx compressed 354Byte message
using --deep-iteration
offset: 0 position: 0 CreateTime: 1518607686194 isvalid: true payloadsize:
354 magic: 1 compresscodec: NoCompressionCodec crc: 4190573446 payload: 354
byte message
*Output 2:*offset: 7 position: 517 CreateTime: 1518608039723 isvalid: true
payloadsize: 61 magic: 1 compresscodec: LZ4CompressionCodec crc: 4075439033
payload: compressed 354B message
using --deep-iteration
offset: 7 position: 517 CreateTime: 1518608039723 isvalid: true
payloadsize: 354 magic: 1 compresscodec: NoCompressionCodec crc: 4061704088
payload: Same 354B message
*Output 3:*
offset: 11 position: 883 CreateTime: 1518608269618 isvalid: true
payloadsize: 61 magic: 1 compresscodec: LZ4CompressionCodec crc: 981370250
payload: compressed 354B message
using --deep-iteration
offset: 11 position: 883 CreateTime: 1518608269618 isvalid: true
payloadsize: 354 magic: 1 compresscodec: NoCompressionCodec crc: 468622988
payload: same 354B message
Please note the compression codecs in the --deep-iteration case,
Case 1 is OK, but in case 2 shouldn't it be SnappyCompression and 3 may be
LZ4Compression
Or is it visible when the message batch into large batches?
Thanks
Uddhav
On Wed, Feb 14, 2018 at 6:05 PM, Manikumar <ma...@gmail.com>
wrote:
> If the broker "compression.type" is "producer", then the broker retains
> the original compression codec set by the producer.
> If the producer and broker codecs are different, then broker recompress
> the data using broker "compression.type".
>
> On Wed, Feb 14, 2018 at 10:58 AM, Uddhav Arote <ar...@gmail.com>
> wrote:
>
> > Hi Kafka users,
> >
> > I am trying to understand the behavior of compression in Kafka. Consider
> a
> > scenario, where producer sets compression.codec "snappy" and broker's
> > compression.code "lz4"?
> > In this scenario, what is the behavior of the compression?
> >
> > As far as I have understood is the following,
> > The messages compressed by the producer are wrapped in the wrapper
> message
> > and send to the broker. If the broker compression.codec is "producer",
> the
> > message is written as is to the log.
> > In the code,
> >
> > https://github.com/apache/kafka/blob/962bc638f9c2ab249e5008a587ee78
> > e3ba35fcb9/core/src/main/scala/kafka/log/LogValidator.scala#L218
> >
> >
> > what I understand is that if the producer and broker codecs are not same,
> > then the compression should happen again.
> >
> > But I am not sure about this. Can somebody tell me how this works?
> >
> > Thanks,
> > Uddhav
> >
>
Re: Compression in Kafka
Posted by Manikumar <ma...@gmail.com>.
If the broker "compression.type" is "producer", then the broker retains
the original compression codec set by the producer.
If the producer and broker codecs are different, then broker recompress
the data using broker "compression.type".
On Wed, Feb 14, 2018 at 10:58 AM, Uddhav Arote <ar...@gmail.com>
wrote:
> Hi Kafka users,
>
> I am trying to understand the behavior of compression in Kafka. Consider a
> scenario, where producer sets compression.codec "snappy" and broker's
> compression.code "lz4"?
> In this scenario, what is the behavior of the compression?
>
> As far as I have understood is the following,
> The messages compressed by the producer are wrapped in the wrapper message
> and send to the broker. If the broker compression.codec is "producer", the
> message is written as is to the log.
> In the code,
>
> https://github.com/apache/kafka/blob/962bc638f9c2ab249e5008a587ee78
> e3ba35fcb9/core/src/main/scala/kafka/log/LogValidator.scala#L218
>
>
> what I understand is that if the producer and broker codecs are not same,
> then the compression should happen again.
>
> But I am not sure about this. Can somebody tell me how this works?
>
> Thanks,
> Uddhav
>