Need help regarding Compression

[Sven Ludwig <s_...@gmx.de>]

Hi,

I need help. We have a Kafka cluster in production with topics compressed with snappy. We want to configure compression.type lz4 on the brokers and restart them (either rolling update, or all down all up if this has benefits).


What happens to the existing topics compressed with snappy after compression.type has changed to lz4 on a leader / on a follower?


Regarding the documentation, nothing should happen to existing topics, because the topic configuration should not change just because that default on a broker changed. But I want to be more sure. My concerns behind this are: Is there any danger of getting topics that are partially in snappy and partially in lz4, and if so, would that be a problem? Is there moreover a danger of records/segments being replicated to other brokers in lz4 while older records/segments on these brokers are in snappy, and if so, would that be a problem?

Kind Regards,
Sven

Aw: Need help regarding Compression

[Sven Ludwig <s_...@gmx.de>]

Hi,

received a first answer today in https://issues.apache.org/jira/browse/KAFKA-1499 from Manikumar.

So it looks like a topic with mixed compression type can be resulting in the described scenario.

I wrote a follow-up question in the ticket whether it could be prevented by post-configuring a topic to give it a topic-level override for compression.type

Kind Regards
Sven
 

Gesendet: Dienstag, 11. Juli 2017 um 11:41 Uhr
Von: "Sven Ludwig" <s_...@gmx.de>
An: users@kafka.apache.org
Betreff: Need help regarding Compression
Hi,

I need help. We have a Kafka cluster in production with topics compressed with snappy. We want to configure compression.type lz4 on the brokers and restart them (either rolling update, or all down all up if this has benefits).


What happens to the existing topics compressed with snappy after compression.type has changed to lz4 on a leader / on a follower?


Regarding the documentation, nothing should happen to existing topics, because the topic configuration should not change just because that default on a broker changed. But I want to be more sure. My concerns behind this are: Is there any danger of getting topics that are partially in snappy and partially in lz4, and if so, would that be a problem? Is there moreover a danger of records/segments being replicated to other brokers in lz4 while older records/segments on these brokers are in snappy, and if so, would that be a problem?

Kind Regards,
Sven