You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Iftach Ben-Yosef <ib...@outbrain.com> on 2020/07/09 05:06:29 UTC
compression on topic vs producer?
Hello,
Up until now we have configured compression on producer level. Since moving
to mm2, we are having some issues with producer level compression and mm2,
and are planning on trying out topic level compression for this case.
Are there any inherent differences to each method other than how/where they
are configured?
Thanks,
Iftach
--
The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.
This email and any
attachments hereto may be confidential or privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person. Thanks.
Re: compression on topic vs producer?
Posted by Iftach Ben-Yosef <ib...@outbrain.com>.
Hi Nitin,
According to your suggestion I tried to set this up in our test env. We got
'test' as source and 'dev' as dest;
'*test->dev.producer.override.**compression.type=gzip*'
I output the logs from mm2 to a file. I still see the compression is set to
none.
grep compre /tmp/log
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
compression.type = none
The closest I got to it working was with '*test.producer.compression.type =
gzip'* then the output from logs looked like this;
2020-07-08 02:10:53,040] WARN The configuration 'producer.compression.type'
was supplied but isn't a known config.
(org.apache.kafka.clients.admin.AdminClientConfig:355)
compression.type = none
[2020-07-08 02:10:53,059] WARN The configuration
'producer.compression.type' was supplied but isn't a known config.
(org.apache.kafka.clients.producer.ProducerConfig:355)
[2020-07-08 02:10:53,073] WARN The configuration
'producer.compression.type' was supplied but isn't a known config.
(org.apache.kafka.clients.consumer.ConsumerConfig:355)
compression.type = none
compression.type = none
compression.type = gzip
compression.type = none
compression.type = gzip
compression.type = none
compression.type = none
compression.type = gzip
compression.type = gzip
compression.type = none
So it seemed like some workers were getting the gzip compression config and
some did not get the config. The topics on destination didn't seem to be
affected size wise though. Also the warning message started to appear.
Thanks,
Iftach
On Thu, Jul 9, 2020 at 10:13 AM nitin agarwal <ni...@gmail.com>
wrote:
> You can add the following configuration for enabling the compression:
> If the cluster names are c1 and c2 then
> *c1->c2.producer.override.**compression.type=lz4*
>
> Thanks,
> Nitin
>
>
> On Thu, Jul 9, 2020 at 11:30 AM Iftach Ben-Yosef <ib...@outbrain.com>
> wrote:
>
> > Hello Nitin, I have been unable to successfully setup producer level
> > compression for my dedicated mm2 cluster, I have a separate mail
> > correspondence on that issue titled 'destination topics in mm2 larger
> than
> > source topic'. I will forward you this correspondence now.
> >
> > I achieved only partial success with trying to implement the producer
> level
> > compression, so I am now thinking about trying to implement topic level
> > compression as that should make the compression unrelated to mm2 itself.
> >
> > Thanks,
> > Iftach
> >
> >
> > On Thu, Jul 9, 2020 at 8:12 AM nitin agarwal <ni...@gmail.com>
> > wrote:
> >
> > > What is the issue with compression at producer level in mm2 ? Can you
> > > please explain?
> > >
> > > Thanks,
> > > Nitin
> > >
> > > On Thu, Jul 9, 2020 at 10:37 AM Iftach Ben-Yosef <
> > iben-yosef@outbrain.com>
> > > wrote:
> > >
> > > > Hello,
> > > >
> > > > Up until now we have configured compression on producer level. Since
> > > moving
> > > > to mm2, we are having some issues with producer level compression and
> > > mm2,
> > > > and are planning on trying out topic level compression for this case.
> > > >
> > > > Are there any inherent differences to each method other than
> how/where
> > > they
> > > > are configured?
> > > >
> > > > Thanks,
> > > > Iftach
> > > >
> > > > --
> > > > The above terms reflect a potential business arrangement, are
> provided
> > > > solely as a basis for further discussion, and are not intended to be
> > and
> > > > do
> > > > not constitute a legally binding obligation. No legally binding
> > > > obligations
> > > > will be created, implied, or inferred until an agreement in final
> form
> > is
> > > > executed in writing by all parties involved.
> > > >
> > > >
> > > > This email and any
> > > > attachments hereto may be confidential or privileged. If you
> received
> > > > this
> > > > communication by mistake, please don't forward it to anyone else,
> > please
> > > > erase all copies and attachments, and please let me know that it has
> > gone
> > > > to the wrong person. Thanks.
> > > >
> > >
> >
> > --
> > The above terms reflect a potential business arrangement, are provided
> > solely as a basis for further discussion, and are not intended to be and
> > do
> > not constitute a legally binding obligation. No legally binding
> > obligations
> > will be created, implied, or inferred until an agreement in final form is
> > executed in writing by all parties involved.
> >
> >
> > This email and any
> > attachments hereto may be confidential or privileged. If you received
> > this
> > communication by mistake, please don't forward it to anyone else, please
> > erase all copies and attachments, and please let me know that it has gone
> > to the wrong person. Thanks.
> >
>
--
The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.
This email and any
attachments hereto may be confidential or privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person. Thanks.
Re: compression on topic vs producer?
Posted by nitin agarwal <ni...@gmail.com>.
You can add the following configuration for enabling the compression:
If the cluster names are c1 and c2 then
*c1->c2.producer.override.**compression.type=lz4*
Thanks,
Nitin
On Thu, Jul 9, 2020 at 11:30 AM Iftach Ben-Yosef <ib...@outbrain.com>
wrote:
> Hello Nitin, I have been unable to successfully setup producer level
> compression for my dedicated mm2 cluster, I have a separate mail
> correspondence on that issue titled 'destination topics in mm2 larger than
> source topic'. I will forward you this correspondence now.
>
> I achieved only partial success with trying to implement the producer level
> compression, so I am now thinking about trying to implement topic level
> compression as that should make the compression unrelated to mm2 itself.
>
> Thanks,
> Iftach
>
>
> On Thu, Jul 9, 2020 at 8:12 AM nitin agarwal <ni...@gmail.com>
> wrote:
>
> > What is the issue with compression at producer level in mm2 ? Can you
> > please explain?
> >
> > Thanks,
> > Nitin
> >
> > On Thu, Jul 9, 2020 at 10:37 AM Iftach Ben-Yosef <
> iben-yosef@outbrain.com>
> > wrote:
> >
> > > Hello,
> > >
> > > Up until now we have configured compression on producer level. Since
> > moving
> > > to mm2, we are having some issues with producer level compression and
> > mm2,
> > > and are planning on trying out topic level compression for this case.
> > >
> > > Are there any inherent differences to each method other than how/where
> > they
> > > are configured?
> > >
> > > Thanks,
> > > Iftach
> > >
> > > --
> > > The above terms reflect a potential business arrangement, are provided
> > > solely as a basis for further discussion, and are not intended to be
> and
> > > do
> > > not constitute a legally binding obligation. No legally binding
> > > obligations
> > > will be created, implied, or inferred until an agreement in final form
> is
> > > executed in writing by all parties involved.
> > >
> > >
> > > This email and any
> > > attachments hereto may be confidential or privileged. If you received
> > > this
> > > communication by mistake, please don't forward it to anyone else,
> please
> > > erase all copies and attachments, and please let me know that it has
> gone
> > > to the wrong person. Thanks.
> > >
> >
>
> --
> The above terms reflect a potential business arrangement, are provided
> solely as a basis for further discussion, and are not intended to be and
> do
> not constitute a legally binding obligation. No legally binding
> obligations
> will be created, implied, or inferred until an agreement in final form is
> executed in writing by all parties involved.
>
>
> This email and any
> attachments hereto may be confidential or privileged. If you received
> this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone
> to the wrong person. Thanks.
>
Re: compression on topic vs producer?
Posted by Iftach Ben-Yosef <ib...@outbrain.com>.
Hello Nitin, I have been unable to successfully setup producer level
compression for my dedicated mm2 cluster, I have a separate mail
correspondence on that issue titled 'destination topics in mm2 larger than
source topic'. I will forward you this correspondence now.
I achieved only partial success with trying to implement the producer level
compression, so I am now thinking about trying to implement topic level
compression as that should make the compression unrelated to mm2 itself.
Thanks,
Iftach
On Thu, Jul 9, 2020 at 8:12 AM nitin agarwal <ni...@gmail.com> wrote:
> What is the issue with compression at producer level in mm2 ? Can you
> please explain?
>
> Thanks,
> Nitin
>
> On Thu, Jul 9, 2020 at 10:37 AM Iftach Ben-Yosef <ib...@outbrain.com>
> wrote:
>
> > Hello,
> >
> > Up until now we have configured compression on producer level. Since
> moving
> > to mm2, we are having some issues with producer level compression and
> mm2,
> > and are planning on trying out topic level compression for this case.
> >
> > Are there any inherent differences to each method other than how/where
> they
> > are configured?
> >
> > Thanks,
> > Iftach
> >
> > --
> > The above terms reflect a potential business arrangement, are provided
> > solely as a basis for further discussion, and are not intended to be and
> > do
> > not constitute a legally binding obligation. No legally binding
> > obligations
> > will be created, implied, or inferred until an agreement in final form is
> > executed in writing by all parties involved.
> >
> >
> > This email and any
> > attachments hereto may be confidential or privileged. If you received
> > this
> > communication by mistake, please don't forward it to anyone else, please
> > erase all copies and attachments, and please let me know that it has gone
> > to the wrong person. Thanks.
> >
>
--
The above terms reflect a potential business arrangement, are provided
solely as a basis for further discussion, and are not intended to be and do
not constitute a legally binding obligation. No legally binding obligations
will be created, implied, or inferred until an agreement in final form is
executed in writing by all parties involved.
This email and any
attachments hereto may be confidential or privileged. If you received this
communication by mistake, please don't forward it to anyone else, please
erase all copies and attachments, and please let me know that it has gone
to the wrong person. Thanks.
Re: compression on topic vs producer?
Posted by nitin agarwal <ni...@gmail.com>.
What is the issue with compression at producer level in mm2 ? Can you
please explain?
Thanks,
Nitin
On Thu, Jul 9, 2020 at 10:37 AM Iftach Ben-Yosef <ib...@outbrain.com>
wrote:
> Hello,
>
> Up until now we have configured compression on producer level. Since moving
> to mm2, we are having some issues with producer level compression and mm2,
> and are planning on trying out topic level compression for this case.
>
> Are there any inherent differences to each method other than how/where they
> are configured?
>
> Thanks,
> Iftach
>
> --
> The above terms reflect a potential business arrangement, are provided
> solely as a basis for further discussion, and are not intended to be and
> do
> not constitute a legally binding obligation. No legally binding
> obligations
> will be created, implied, or inferred until an agreement in final form is
> executed in writing by all parties involved.
>
>
> This email and any
> attachments hereto may be confidential or privileged. If you received
> this
> communication by mistake, please don't forward it to anyone else, please
> erase all copies and attachments, and please let me know that it has gone
> to the wrong person. Thanks.
>