You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sven Ludwig <s_...@gmx.de> on 2017/06/26 10:59:06 UTC
Broker-side Compression and Migration to lz4
Hi,
1. I would like to test lz4 compression on a new broker node that I want add to my cluster, while the other nodes remain in snappy, in order to compare disk usage etc. I am not sure if this scenario is even possible with only one cluster, since in the docs it is mentioned that compression.type is a topic-level setting, and thus maybe not really per broker. I want to use Broker-side compression only. The situation is:
broker 1 - Status: Up - compression.type snappy
broker 2 - Status: Up - compression.type snappy
broker 3 - Status: Down - compression.type lz4
The topic for the test already exists on brokers 1 and 2 which are up and has some 20GB content.
When I start broker 3, will it use snappy or lz4 for the topic?
2. How can one migrate a cluster (or a topic) from one broker-side compression type to another?
Kind Regards,
Sven
Aw: Re: Broker-side Compression and Migration to lz4
Posted by Sven Ludwig <s_...@gmx.de>.
Hi,
thanks. On another aspect in this: On a cluster that has topics with data, is it safe to gracefully shutdown a broker, change the configured compression from snappy to lz4 and start the broker? What will haben if one does that? All afterwards created new topics will come with lz4 compression. But what happens to the existing topics in snappy?
Regards
Sven
Gesendet: Montag, 26. Juni 2017 um 13:10 Uhr
Von: "Ismael Juma" <is...@juma.me.uk>
An: "Kafka Users" <us...@kafka.apache.org>
Betreff: Re: Broker-side Compression and Migration to lz4
Hi Sven,
If you change the topic config, any new data received by that broker will
be in the new compression type. However, followers don't uncompress data,
so they will store the data as it was in the leader. An easier way to test
what you are trying to test is to use MirrorMaker to mirror the data to
another cluster.
Ismael
On Mon, Jun 26, 2017 at 11:59 AM, Sven Ludwig <s_...@gmx.de> wrote:
> Hi,
>
>
> 1. I would like to test lz4 compression on a new broker node that I want
> add to my cluster, while the other nodes remain in snappy, in order to
> compare disk usage etc. I am not sure if this scenario is even possible
> with only one cluster, since in the docs it is mentioned that
> compression.type is a topic-level setting, and thus maybe not really per
> broker. I want to use Broker-side compression only. The situation is:
>
> broker 1 - Status: Up - compression.type snappy
> broker 2 - Status: Up - compression.type snappy
> broker 3 - Status: Down - compression.type lz4
>
> The topic for the test already exists on brokers 1 and 2 which are up and
> has some 20GB content.
>
> When I start broker 3, will it use snappy or lz4 for the topic?
>
>
> 2. How can one migrate a cluster (or a topic) from one broker-side
> compression type to another?
>
>
> Kind Regards,
> Sven
>
Re: Broker-side Compression and Migration to lz4
Posted by Ismael Juma <is...@juma.me.uk>.
Hi Sven,
If you change the topic config, any new data received by that broker will
be in the new compression type. However, followers don't uncompress data,
so they will store the data as it was in the leader. An easier way to test
what you are trying to test is to use MirrorMaker to mirror the data to
another cluster.
Ismael
On Mon, Jun 26, 2017 at 11:59 AM, Sven Ludwig <s_...@gmx.de> wrote:
> Hi,
>
>
> 1. I would like to test lz4 compression on a new broker node that I want
> add to my cluster, while the other nodes remain in snappy, in order to
> compare disk usage etc. I am not sure if this scenario is even possible
> with only one cluster, since in the docs it is mentioned that
> compression.type is a topic-level setting, and thus maybe not really per
> broker. I want to use Broker-side compression only. The situation is:
>
> broker 1 - Status: Up - compression.type snappy
> broker 2 - Status: Up - compression.type snappy
> broker 3 - Status: Down - compression.type lz4
>
> The topic for the test already exists on brokers 1 and 2 which are up and
> has some 20GB content.
>
> When I start broker 3, will it use snappy or lz4 for the topic?
>
>
> 2. How can one migrate a cluster (or a topic) from one broker-side
> compression type to another?
>
>
> Kind Regards,
> Sven
>