Compress before or after serialization?

[David Yu <gu...@gmail.com>]

We are using Avro as our format when writing to a Kafka topic. After we
enabled snappy compression on the producer, we don't see a change in the
compression ratio (still 1). I was wondering if we should compress the
message before serialization.

Thanks,
David

Re: Compress before or after serialization?

[Tom Crayford <tc...@heroku.com>]

Kafka compresses batches in the producer before sending them to the broker.
You'll get notably better compression from this than you will from per
message compression. I'd recommend checking your producer config and maybe
looking at the log segments on the broker with DumpLogSegments.

If you have sample producer configs and code, maybe this would help in
diagnosing the issue.

Thanks

Tom Crayford
Heroku Kafka

On Friday, 5 August 2016, David Yu <gu...@gmail.com> wrote:

> We are using Avro as our format when writing to a Kafka topic. After we
> enabled snappy compression on the producer, we don't see a change in the
> compression ratio (still 1). I was wondering if we should compress the
> message before serialization.
>
> Thanks,
> David
>