Broker to consumer compression

[Pablo Barrera González <pa...@gmail.com>]

Hi

Is it possible to enable compression between the broker and the consumer?

We are thinking in develop this feature in kafka 0.7 but first I would
like to check if there is something out there.

Our escenario is like this:

- the producer is a CPU bounded machine, so we want to keep the CPU
consumption as low as possible, so we can't enable compression here
- the consumers can fetch data from the same data center (no
compression needed) or from a remote data center
- intersite bandwidth is limited so compression would be interesting

Our approach is to add compress the connection at kafka level between
broker and consumer, inside kafka, so the final user can read plain
data.

Regards

Pablo

Re: Broker to consumer compression

[Neha Narkhede <ne...@gmail.com>]

That is not available for performance reasons. Broker uses zero-copy
to transfer data from disk to the network on the consumer side. If we
post process data already written to disk before sending it to
consumer, we will lose the performance advantage that we have due to
zero copy.

Thanks,
Neha

On Fri, Apr 12, 2013 at 12:59 PM, Pablo Barrera González
<pa...@gmail.com> wrote:
> Thanks for the replay Neha, but that's is end-to-end and I am looking
> for a broker-consumer compression.
>
> So:
>
> Producer -> uncompressed -> broker -> compressed -> consumer
>
> Regards
>
> Pablo
>
>
> 2013/4/12 Neha Narkhede <ne...@gmail.com>:
>> Kafka already supports end-to-end compression which means data
>> transfer between brokers and consumers is compressed. There are two
>> supported compression codecs - GZIP and Snappy. The latter is lighter
>> on CPU consumption. See this blog post for comparison -
>> http://geekmantra.wordpress.com/2013/03/28/compression-in-kafka-gzip-or-snappy/
>>
>> Thanks,
>> Neha
>>
>> On Fri, Apr 12, 2013 at 10:56 AM, Pablo Barrera González
>> <pa...@gmail.com> wrote:
>>> Hi
>>>
>>> Is it possible to enable compression between the broker and the consumer?
>>>
>>> We are thinking in develop this feature in kafka 0.7 but first I would
>>> like to check if there is something out there.
>>>
>>> Our escenario is like this:
>>>
>>> - the producer is a CPU bounded machine, so we want to keep the CPU
>>> consumption as low as possible, so we can't enable compression here
>>> - the consumers can fetch data from the same data center (no
>>> compression needed) or from a remote data center
>>> - intersite bandwidth is limited so compression would be interesting
>>>
>>> Our approach is to add compress the connection at kafka level between
>>> broker and consumer, inside kafka, so the final user can read plain
>>> data.
>>>
>>> Regards
>>>
>>> Pablo

Re: Broker to consumer compression

[Pablo Barrera González <pa...@gmail.com>]

Thanks for the replay Neha, but that's is end-to-end and I am looking
for a broker-consumer compression.

So:

Producer -> uncompressed -> broker -> compressed -> consumer

Regards

Pablo


2013/4/12 Neha Narkhede <ne...@gmail.com>:
> Kafka already supports end-to-end compression which means data
> transfer between brokers and consumers is compressed. There are two
> supported compression codecs - GZIP and Snappy. The latter is lighter
> on CPU consumption. See this blog post for comparison -
> http://geekmantra.wordpress.com/2013/03/28/compression-in-kafka-gzip-or-snappy/
>
> Thanks,
> Neha
>
> On Fri, Apr 12, 2013 at 10:56 AM, Pablo Barrera González
> <pa...@gmail.com> wrote:
>> Hi
>>
>> Is it possible to enable compression between the broker and the consumer?
>>
>> We are thinking in develop this feature in kafka 0.7 but first I would
>> like to check if there is something out there.
>>
>> Our escenario is like this:
>>
>> - the producer is a CPU bounded machine, so we want to keep the CPU
>> consumption as low as possible, so we can't enable compression here
>> - the consumers can fetch data from the same data center (no
>> compression needed) or from a remote data center
>> - intersite bandwidth is limited so compression would be interesting
>>
>> Our approach is to add compress the connection at kafka level between
>> broker and consumer, inside kafka, so the final user can read plain
>> data.
>>
>> Regards
>>
>> Pablo

Re: Broker to consumer compression

[Neha Narkhede <ne...@gmail.com>]

Kafka already supports end-to-end compression which means data
transfer between brokers and consumers is compressed. There are two
supported compression codecs - GZIP and Snappy. The latter is lighter
on CPU consumption. See this blog post for comparison -
http://geekmantra.wordpress.com/2013/03/28/compression-in-kafka-gzip-or-snappy/

Thanks,
Neha

On Fri, Apr 12, 2013 at 10:56 AM, Pablo Barrera González
<pa...@gmail.com> wrote:
> Hi
>
> Is it possible to enable compression between the broker and the consumer?
>
> We are thinking in develop this feature in kafka 0.7 but first I would
> like to check if there is something out there.
>
> Our escenario is like this:
>
> - the producer is a CPU bounded machine, so we want to keep the CPU
> consumption as low as possible, so we can't enable compression here
> - the consumers can fetch data from the same data center (no
> compression needed) or from a remote data center
> - intersite bandwidth is limited so compression would be interesting
>
> Our approach is to add compress the connection at kafka level between
> broker and consumer, inside kafka, so the final user can read plain
> data.
>
> Regards
>
> Pablo