You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sameer Kumar <sa...@gmail.com> on 2017/11/22 11:26:42 UTC
Switching compression codec at runtime
Hi,
Is it possible to switch from gzip to lz4 at runtime on kafka brokers. My
servers are currently running on gzip, and I want to switch them to lz4.
-Sameer.
Re: Switching compression codec at runtime
Posted by Sameer Kumar <sa...@gmail.com>.
ok..got it..thanks Manikumar...😀
On Thu, Nov 23, 2017 at 2:58 PM, Manikumar <ma...@gmail.com>
wrote:
> topic-level config "compression.type" takes precedence over producer
> compression code.
> By default topic-level config "compression.type" is "producer", which means
> retain
> the original compression codec set by the producer.
>
> If you set topic-level config "compression.type" to any other compression
> code,
> then on broker-side, it will recompress the data using topic-level config
> "compression.type".
> No need to restart broker or clients. Disadvantage with approach is
> recompression.
>
> maybe you can try these settings on a test server.
>
> On Thu, Nov 23, 2017 at 2:39 PM, Sameer Kumar <sa...@gmail.com>
> wrote:
>
> > Ok. So you mean stop all producers, change the compress type for topic at
> > runtime and switch the compression type for producers and have them start
> > again.
> >
> > On Thu, Nov 23, 2017 at 12:46 PM, Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > You can dynamically change topic level configs on brokers.
> > > http://kafka.apache.org/documentation.html#topicconfigs
> > >
> > > On Thu, Nov 23, 2017 at 12:38 PM, Sameer Kumar <sam.kum.work@gmail.com
> >
> > > wrote:
> > >
> > > > I am not too sure if I can have different compression types for both
> > > > producer and broker. It has to be same.
> > > >
> > > > This is possible by stopping all brokers, producers and changing
> > values.
> > > > But for that broker cluster has to be done. I was looking if there is
> > any
> > > > way we can do that in a running cluster.
> > > >
> > > > -Sameer.
> > > >
> > > > On Wed, Nov 22, 2017 at 7:24 PM, Manikumar <
> manikumar.reddy@gmail.com>
> > > > wrote:
> > > >
> > > > > You can set topic specific compression type by setting topic level
> > > config
> > > > > "compression.type"
> > > > > another option is change compression type config on producer side.
> > > > >
> > > > > On Wed, Nov 22, 2017 at 4:56 PM, Sameer Kumar <
> > sam.kum.work@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > Is it possible to switch from gzip to lz4 at runtime on kafka
> > > brokers.
> > > > My
> > > > > > servers are currently running on gzip, and I want to switch them
> to
> > > > lz4.
> > > > > >
> > > > > >
> > > > > >
> > > > > > -Sameer.
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Switching compression codec at runtime
Posted by Manikumar <ma...@gmail.com>.
topic-level config "compression.type" takes precedence over producer
compression code.
By default topic-level config "compression.type" is "producer", which means
retain
the original compression codec set by the producer.
If you set topic-level config "compression.type" to any other compression
code,
then on broker-side, it will recompress the data using topic-level config
"compression.type".
No need to restart broker or clients. Disadvantage with approach is
recompression.
maybe you can try these settings on a test server.
On Thu, Nov 23, 2017 at 2:39 PM, Sameer Kumar <sa...@gmail.com>
wrote:
> Ok. So you mean stop all producers, change the compress type for topic at
> runtime and switch the compression type for producers and have them start
> again.
>
> On Thu, Nov 23, 2017 at 12:46 PM, Manikumar <ma...@gmail.com>
> wrote:
>
> > You can dynamically change topic level configs on brokers.
> > http://kafka.apache.org/documentation.html#topicconfigs
> >
> > On Thu, Nov 23, 2017 at 12:38 PM, Sameer Kumar <sa...@gmail.com>
> > wrote:
> >
> > > I am not too sure if I can have different compression types for both
> > > producer and broker. It has to be same.
> > >
> > > This is possible by stopping all brokers, producers and changing
> values.
> > > But for that broker cluster has to be done. I was looking if there is
> any
> > > way we can do that in a running cluster.
> > >
> > > -Sameer.
> > >
> > > On Wed, Nov 22, 2017 at 7:24 PM, Manikumar <ma...@gmail.com>
> > > wrote:
> > >
> > > > You can set topic specific compression type by setting topic level
> > config
> > > > "compression.type"
> > > > another option is change compression type config on producer side.
> > > >
> > > > On Wed, Nov 22, 2017 at 4:56 PM, Sameer Kumar <
> sam.kum.work@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Is it possible to switch from gzip to lz4 at runtime on kafka
> > brokers.
> > > My
> > > > > servers are currently running on gzip, and I want to switch them to
> > > lz4.
> > > > >
> > > > >
> > > > >
> > > > > -Sameer.
> > > > >
> > > >
> > >
> >
>
Re: Switching compression codec at runtime
Posted by Sameer Kumar <sa...@gmail.com>.
Ok. So you mean stop all producers, change the compress type for topic at
runtime and switch the compression type for producers and have them start
again.
On Thu, Nov 23, 2017 at 12:46 PM, Manikumar <ma...@gmail.com>
wrote:
> You can dynamically change topic level configs on brokers.
> http://kafka.apache.org/documentation.html#topicconfigs
>
> On Thu, Nov 23, 2017 at 12:38 PM, Sameer Kumar <sa...@gmail.com>
> wrote:
>
> > I am not too sure if I can have different compression types for both
> > producer and broker. It has to be same.
> >
> > This is possible by stopping all brokers, producers and changing values.
> > But for that broker cluster has to be done. I was looking if there is any
> > way we can do that in a running cluster.
> >
> > -Sameer.
> >
> > On Wed, Nov 22, 2017 at 7:24 PM, Manikumar <ma...@gmail.com>
> > wrote:
> >
> > > You can set topic specific compression type by setting topic level
> config
> > > "compression.type"
> > > another option is change compression type config on producer side.
> > >
> > > On Wed, Nov 22, 2017 at 4:56 PM, Sameer Kumar <sa...@gmail.com>
> > > wrote:
> > >
> > > > Hi,
> > > >
> > > > Is it possible to switch from gzip to lz4 at runtime on kafka
> brokers.
> > My
> > > > servers are currently running on gzip, and I want to switch them to
> > lz4.
> > > >
> > > >
> > > >
> > > > -Sameer.
> > > >
> > >
> >
>
Re: Switching compression codec at runtime
Posted by Manikumar <ma...@gmail.com>.
You can dynamically change topic level configs on brokers.
http://kafka.apache.org/documentation.html#topicconfigs
On Thu, Nov 23, 2017 at 12:38 PM, Sameer Kumar <sa...@gmail.com>
wrote:
> I am not too sure if I can have different compression types for both
> producer and broker. It has to be same.
>
> This is possible by stopping all brokers, producers and changing values.
> But for that broker cluster has to be done. I was looking if there is any
> way we can do that in a running cluster.
>
> -Sameer.
>
> On Wed, Nov 22, 2017 at 7:24 PM, Manikumar <ma...@gmail.com>
> wrote:
>
> > You can set topic specific compression type by setting topic level config
> > "compression.type"
> > another option is change compression type config on producer side.
> >
> > On Wed, Nov 22, 2017 at 4:56 PM, Sameer Kumar <sa...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Is it possible to switch from gzip to lz4 at runtime on kafka brokers.
> My
> > > servers are currently running on gzip, and I want to switch them to
> lz4.
> > >
> > >
> > >
> > > -Sameer.
> > >
> >
>
Re: Switching compression codec at runtime
Posted by Sameer Kumar <sa...@gmail.com>.
I am not too sure if I can have different compression types for both
producer and broker. It has to be same.
This is possible by stopping all brokers, producers and changing values.
But for that broker cluster has to be done. I was looking if there is any
way we can do that in a running cluster.
-Sameer.
On Wed, Nov 22, 2017 at 7:24 PM, Manikumar <ma...@gmail.com>
wrote:
> You can set topic specific compression type by setting topic level config
> "compression.type"
> another option is change compression type config on producer side.
>
> On Wed, Nov 22, 2017 at 4:56 PM, Sameer Kumar <sa...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Is it possible to switch from gzip to lz4 at runtime on kafka brokers. My
> > servers are currently running on gzip, and I want to switch them to lz4.
> >
> >
> >
> > -Sameer.
> >
>
Re: Switching compression codec at runtime
Posted by Manikumar <ma...@gmail.com>.
You can set topic specific compression type by setting topic level config
"compression.type"
another option is change compression type config on producer side.
On Wed, Nov 22, 2017 at 4:56 PM, Sameer Kumar <sa...@gmail.com>
wrote:
> Hi,
>
> Is it possible to switch from gzip to lz4 at runtime on kafka brokers. My
> servers are currently running on gzip, and I want to switch them to lz4.
>
>
>
> -Sameer.
>