You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Vincent Dautremont <vi...@olamobile.com> on 2016/11/11 06:36:44 UTC
Kafka and compression
Hi,
Can anyone explain me in more detail how Kakfa works with compression ?
I've read the doc but it's not all clear to me.
- There are compression settings on the broker, the topic of a broker, a
producer.
Are they all the same setting and one takes precedence on another ?
- Is there a difference between compression for transfer and compression
for storage ?
- Is the compression done per message or per topic ?
For example, my message which will transit from / to kafka are in a JSON
format (so with similar structure). will compression compress a chunk of
several message (which I guess would be more efficient in my case
considering my bottle neck is the transport link) or will it compress every
message separately ?
The documentation would be great if illustrated with few examples like :
Broker has compression type GZIP
Topic-A has compression Snappy
Topic-B has no compression setting
Producer-X has compression GZIP
Producer-Y has no compression settings.
explain what happens (on the wire, and on disk) when
X Produces to A
X Produces to B
Y Produces to A
Y Produces to B
+ explain if some settings n-uple are nonsense / imcompatible / could
result in errors.
Thank you.
--
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
Re: Kafka and compression
Posted by Francesco laTorre <fr...@openbet.com>.
+1
On 11 November 2016 at 06:36, Vincent Dautremont <
vincent.dautremont@olamobile.com> wrote:
> Hi,
> Can anyone explain me in more detail how Kakfa works with compression ?
> I've read the doc but it's not all clear to me.
>
>
> - There are compression settings on the broker, the topic of a broker, a
> producer.
> Are they all the same setting and one takes precedence on another ?
>
> - Is there a difference between compression for transfer and compression
> for storage ?
>
> - Is the compression done per message or per topic ?
> For example, my message which will transit from / to kafka are in a JSON
> format (so with similar structure). will compression compress a chunk of
> several message (which I guess would be more efficient in my case
> considering my bottle neck is the transport link) or will it compress every
> message separately ?
>
>
> The documentation would be great if illustrated with few examples like :
> Broker has compression type GZIP
> Topic-A has compression Snappy
> Topic-B has no compression setting
> Producer-X has compression GZIP
> Producer-Y has no compression settings.
>
> explain what happens (on the wire, and on disk) when
> X Produces to A
> X Produces to B
> Y Produces to A
> Y Produces to B
> + explain if some settings n-uple are nonsense / imcompatible / could
> result in errors.
>
> Thank you.
>
> --
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>
--
<http://www.openbet.com/> Francesco laTorre
Senior Developer
T: +44 208 742 1600
+44 203 249 8394
E: francesco.latorre@openbet.com
W: www.openbet.com
OpenBet Ltd
Chiswick Park Building 9
566 Chiswick High Rd
London
W4 5XT
<https://www.openbet.com/email_promo>
This message is confidential and intended only for the addressee. If you
have received this message in error, please immediately notify the
postmaster@openbet.com and delete it from your system as well as any
copies. The content of e-mails as well as traffic data may be monitored by
OpenBet for employment and security purposes. To protect the environment
please do not print this e-mail unless necessary. OpenBet Ltd. Registered
Office: Chiswick Park Building 9, 566 Chiswick High Road, London, W4 5XT,
United Kingdom. A company registered in England and Wales. Registered no.
3134634. VAT no. GB927523612