You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kafka Life <li...@gmail.com> on 2022/02/06 09:19:41 UTC
Mirror Maker Issue : Huge difference in topic size on disk
Dear Kafka Experts , Need your advice please
I am running a mirror maker in kafka 2.8 to replicate a topic from kafka
0.11 instance.
The size of each partition for a topic on 0.11 is always in 5 to 6 GB but
the replicated topic in 2.8 instances is in 40 GB for the same partition.
The topic configuration in terms of retention, topic partition max size is
same across both the instances. Is there any issue with MM on 2.8 kafka or
any other pointers to look at please.
Many thanks in advance .
Re: Mirror Maker Issue : Huge difference in topic size on disk
Posted by Samuel Cantero <sc...@gmail.com>.
You could check whether the producers on the source cluster are using some
compression algo. Use the same one on mm2 as currently it will decompress
and then write again to the target cluster uncompressed unless you
configure it.
On Sun, Feb 6, 2022 at 05:20 Kafka Life <li...@gmail.com> wrote:
> Dear Kafka Experts , Need your advice please
>
> I am running a mirror maker in kafka 2.8 to replicate a topic from kafka
> 0.11 instance.
> The size of each partition for a topic on 0.11 is always in 5 to 6 GB but
> the replicated topic in 2.8 instances is in 40 GB for the same partition.
>
> The topic configuration in terms of retention, topic partition max size is
> same across both the instances. Is there any issue with MM on 2.8 kafka or
> any other pointers to look at please.
>
> Many thanks in advance .
>
Re: KAFKA- UNDER_REPLICATED_PARTIONS - AUTOMATE
Posted by Kafka Life <li...@gmail.com>.
Thank you Malcolm. Will go through this.
On Sat, Feb 26, 2022 at 2:22 AM Malcolm McFarland <mm...@cavulus.com>
wrote:
> Maybe this could help?
> https://github.com/dimas/kafka-reassign-tool
>
> Cheers,
> Malcolm McFarland
> Cavulus
>
>
> On Fri, Feb 25, 2022 at 9:00 AM Kafka Life <li...@gmail.com> wrote:
>
> > Dear Experts
> >
> > do you have any solution for this please
> >
> > On Tue, Feb 22, 2022 at 8:31 PM Kafka Life <li...@gmail.com>
> wrote:
> >
> > > Dear Kafka Experts
> > >
> > > Does anyone have a dynamically generated Json file based on the Under
> > > replicated partition in the kafka cluster.
> > > Everytime when the URP is increased to over 500 , it is a tedious job
> to
> > > manually create a Json file .
> > >
> > > I request you to share any such dynamically generated script /json
> file.
> > >
> > > Thanks in advance.
> > >
> > >>
> >
>
Re: KAFKA- UNDER_REPLICATED_PARTIONS - AUTOMATE
Posted by Kafka Life <li...@gmail.com>.
Thank you Malcolm. Will go through this.
On Sat, Feb 26, 2022 at 2:22 AM Malcolm McFarland <mm...@cavulus.com>
wrote:
> Maybe this could help?
> https://github.com/dimas/kafka-reassign-tool
>
> Cheers,
> Malcolm McFarland
> Cavulus
>
>
> On Fri, Feb 25, 2022 at 9:00 AM Kafka Life <li...@gmail.com> wrote:
>
> > Dear Experts
> >
> > do you have any solution for this please
> >
> > On Tue, Feb 22, 2022 at 8:31 PM Kafka Life <li...@gmail.com>
> wrote:
> >
> > > Dear Kafka Experts
> > >
> > > Does anyone have a dynamically generated Json file based on the Under
> > > replicated partition in the kafka cluster.
> > > Everytime when the URP is increased to over 500 , it is a tedious job
> to
> > > manually create a Json file .
> > >
> > > I request you to share any such dynamically generated script /json
> file.
> > >
> > > Thanks in advance.
> > >
> > >>
> >
>
Re: KAFKA- UNDER_REPLICATED_PARTIONS - AUTOMATE
Posted by Malcolm McFarland <mm...@cavulus.com>.
Maybe this could help?
https://github.com/dimas/kafka-reassign-tool
Cheers,
Malcolm McFarland
Cavulus
On Fri, Feb 25, 2022 at 9:00 AM Kafka Life <li...@gmail.com> wrote:
> Dear Experts
>
> do you have any solution for this please
>
> On Tue, Feb 22, 2022 at 8:31 PM Kafka Life <li...@gmail.com> wrote:
>
> > Dear Kafka Experts
> >
> > Does anyone have a dynamically generated Json file based on the Under
> > replicated partition in the kafka cluster.
> > Everytime when the URP is increased to over 500 , it is a tedious job to
> > manually create a Json file .
> >
> > I request you to share any such dynamically generated script /json file.
> >
> > Thanks in advance.
> >
> >>
>
Re: KAFKA- UNDER_REPLICATED_PARTIONS - AUTOMATE
Posted by Malcolm McFarland <mm...@cavulus.com>.
Maybe this could help?
https://github.com/dimas/kafka-reassign-tool
Cheers,
Malcolm McFarland
Cavulus
On Fri, Feb 25, 2022 at 9:00 AM Kafka Life <li...@gmail.com> wrote:
> Dear Experts
>
> do you have any solution for this please
>
> On Tue, Feb 22, 2022 at 8:31 PM Kafka Life <li...@gmail.com> wrote:
>
> > Dear Kafka Experts
> >
> > Does anyone have a dynamically generated Json file based on the Under
> > replicated partition in the kafka cluster.
> > Everytime when the URP is increased to over 500 , it is a tedious job to
> > manually create a Json file .
> >
> > I request you to share any such dynamically generated script /json file.
> >
> > Thanks in advance.
> >
> >>
>
Re: KAFKA- UNDER_REPLICATED_PARTIONS - AUTOMATE
Posted by Kafka Life <li...@gmail.com>.
Dear Experts
do you have any solution for this please
On Tue, Feb 22, 2022 at 8:31 PM Kafka Life <li...@gmail.com> wrote:
> Dear Kafka Experts
>
> Does anyone have a dynamically generated Json file based on the Under
> replicated partition in the kafka cluster.
> Everytime when the URP is increased to over 500 , it is a tedious job to
> manually create a Json file .
>
> I request you to share any such dynamically generated script /json file.
>
> Thanks in advance.
>
>>
Re: KAFKA- UNDER_REPLICATED_PARTIONS - AUTOMATE
Posted by Kafka Life <li...@gmail.com>.
Dear Experts
do you have any solution for this please
On Tue, Feb 22, 2022 at 8:31 PM Kafka Life <li...@gmail.com> wrote:
> Dear Kafka Experts
>
> Does anyone have a dynamically generated Json file based on the Under
> replicated partition in the kafka cluster.
> Everytime when the URP is increased to over 500 , it is a tedious job to
> manually create a Json file .
>
> I request you to share any such dynamically generated script /json file.
>
> Thanks in advance.
>
>>
KAFKA- UNDER_REPLICATED_PARTIONS - AUTOMATE
Posted by Kafka Life <li...@gmail.com>.
Dear Kafka Experts
Does anyone have a dynamically generated Json file based on the Under
replicated partition in the kafka cluster.
Everytime when the URP is increased to over 500 , it is a tedious job to
manually create a Json file .
I request you to share any such dynamically generated script /json file.
Thanks in advance.
>
Re: Mirror Maker Issue : Huge difference in topic size on disk
Posted by Samuel Cantero <sc...@gmail.com>.
You could check whether the producers on the source cluster are using some
compression algo. Use the same one on mm2 as currently it will decompress
and then write again to the target cluster uncompressed unless you
configure it.
On Sun, Feb 6, 2022 at 05:20 Kafka Life <li...@gmail.com> wrote:
> Dear Kafka Experts , Need your advice please
>
> I am running a mirror maker in kafka 2.8 to replicate a topic from kafka
> 0.11 instance.
> The size of each partition for a topic on 0.11 is always in 5 to 6 GB but
> the replicated topic in 2.8 instances is in 40 GB for the same partition.
>
> The topic configuration in terms of retention, topic partition max size is
> same across both the instances. Is there any issue with MM on 2.8 kafka or
> any other pointers to look at please.
>
> Many thanks in advance .
>
KAFKA- UNDER_REPLICATED_PARTIONS - AUTOMATE
Posted by Kafka Life <li...@gmail.com>.
Dear Kafka Experts
Does anyone have a dynamically generated Json file based on the Under
replicated partition in the kafka cluster.
Everytime when the URP is increased to over 500 , it is a tedious job to
manually create a Json file .
I request you to share any such dynamically generated script /json file.
Thanks in advance.
>