You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Lud Antonie <lu...@coosto.com.INVALID> on 2023/11/30 14:39:18 UTC
Kafka 2.7.2 to 3.5.1 upgrade
Hello,
After upgrading from 2.7.2 to 3.5.1 some topics are missing a partition for
one or two brokers.
The kafka manager shows "Under replicated%" for the topic.
Looking at the topic for some brokers (of 3) partitions are missing (in my
case 1 partition).
A rollback will restore the "Under replicated%" to 0 again (this is the
wanted number).
Is this a bug of kafka or the kafka manager?
Best regards,
Lud Antonie
--
Met vriendelijke groet / Kind regards,
*Lud Antonie*
<https://www.coosto.com/>
Kennedyplein 101, 5611 ZS, Eindhoven
+31(0)402492700 <0031402492700>
www.coosto.com
<https://twitter.com/CoostoNL> <https://www.facebook.com/coosto.solution>
<https://www.instagram.com/coosto_com/>
<https://www.linkedin.com/company/coosto>
Re: Kafka 2.7.2 to 3.5.1 upgrade
Posted by Luke Chen <sh...@gmail.com>.
Hi Lud,
This is a known issue(KAFKA-15353
<https://issues.apache.org/jira/browse/KAFKA-15353>) and I've fixed it in
v3.5.2 (will get released soon) and v3.6.0.
Thanks.
Luke
On Mon, Dec 4, 2023 at 6:01 PM Lud Antonie <lu...@coosto.com.invalid>
wrote:
> Hi Megh,
>
> No, the number of partitions haven't increased.
> The upgrade is done in a test environment, the topic is just created just
> before the upgrade.
>
> As a test I performed a rollback to 2.7.2 and did an upgrade to 3.4.1.
> This upgrade went ok i.e. without under replicated partitions.
>
> Best regards,
> Lud
>
> Op za 2 dec 2023 om 04:08 schreef megh vidani <vi...@gmail.com>:
>
> > Hi Lud,
> >
> > The topics for which you're seeing under replicated partitions, Did you
> try
> > to increase the number of partitions anytime after creation of those
> topics
> > before the upgrade?
> >
> > We have earlier faced issues with 2.8.0, in which we had increased the
> > number of partitions for some topics, and for those topics we used to see
> > under replicated partitions after every restart.
> >
> > The reason this happened was, there was a bug in Kafka which assigned a
> new
> > topicId (different from the original topicId) to newly added partitions
> in
> > the partition.metadata file, and upon restart of kafka brokers, this
> > topicId didn't reconcile between brokers and ZK.
> >
> > Thanks,
> > Megh
> >
> > On Thu, Nov 30, 2023, 20:10 Lud Antonie <lu...@coosto.com.invalid>
> > wrote:
> >
> > > Hello,
> > >
> > > After upgrading from 2.7.2 to 3.5.1 some topics are missing a partition
> > for
> > > one or two brokers.
> > > The kafka manager shows "Under replicated%" for the topic.
> > > Looking at the topic for some brokers (of 3) partitions are missing (in
> > my
> > > case 1 partition).
> > > A rollback will restore the "Under replicated%" to 0 again (this is the
> > > wanted number).
> > >
> > > Is this a bug of kafka or the kafka manager?
> > >
> > > Best regards,
> > > Lud Antonie
> > >
> > >
> > > --
> > > Met vriendelijke groet / Kind regards,
> > >
> > > *Lud Antonie*
> > >
> > > <https://www.coosto.com/>
> > > Kennedyplein 101, 5611 ZS, Eindhoven
> > > +31(0)402492700 <0031402492700>
> > > www.coosto.com
> > > <https://twitter.com/CoostoNL> <
> https://www.facebook.com/coosto.solution
> > >
> > > <https://www.instagram.com/coosto_com/>
> > > <https://www.linkedin.com/company/coosto>
> > >
> >
>
>
> --
> Met vriendelijke groet / Kind regards,
>
> *Lud Antonie*
>
> <https://www.coosto.com/>
> Kennedyplein 101, 5611 ZS, Eindhoven
> +31(0)402492700 <0031402492700>
> www.coosto.com
> <https://twitter.com/CoostoNL> <https://www.facebook.com/coosto.solution>
> <https://www.instagram.com/coosto_com/>
> <https://www.linkedin.com/company/coosto>
>
Re: Kafka 2.7.2 to 3.5.1 upgrade
Posted by Lud Antonie <lu...@coosto.com.INVALID>.
Hi Megh,
No, the number of partitions haven't increased.
The upgrade is done in a test environment, the topic is just created just
before the upgrade.
As a test I performed a rollback to 2.7.2 and did an upgrade to 3.4.1.
This upgrade went ok i.e. without under replicated partitions.
Best regards,
Lud
Op za 2 dec 2023 om 04:08 schreef megh vidani <vi...@gmail.com>:
> Hi Lud,
>
> The topics for which you're seeing under replicated partitions, Did you try
> to increase the number of partitions anytime after creation of those topics
> before the upgrade?
>
> We have earlier faced issues with 2.8.0, in which we had increased the
> number of partitions for some topics, and for those topics we used to see
> under replicated partitions after every restart.
>
> The reason this happened was, there was a bug in Kafka which assigned a new
> topicId (different from the original topicId) to newly added partitions in
> the partition.metadata file, and upon restart of kafka brokers, this
> topicId didn't reconcile between brokers and ZK.
>
> Thanks,
> Megh
>
> On Thu, Nov 30, 2023, 20:10 Lud Antonie <lu...@coosto.com.invalid>
> wrote:
>
> > Hello,
> >
> > After upgrading from 2.7.2 to 3.5.1 some topics are missing a partition
> for
> > one or two brokers.
> > The kafka manager shows "Under replicated%" for the topic.
> > Looking at the topic for some brokers (of 3) partitions are missing (in
> my
> > case 1 partition).
> > A rollback will restore the "Under replicated%" to 0 again (this is the
> > wanted number).
> >
> > Is this a bug of kafka or the kafka manager?
> >
> > Best regards,
> > Lud Antonie
> >
> >
> > --
> > Met vriendelijke groet / Kind regards,
> >
> > *Lud Antonie*
> >
> > <https://www.coosto.com/>
> > Kennedyplein 101, 5611 ZS, Eindhoven
> > +31(0)402492700 <0031402492700>
> > www.coosto.com
> > <https://twitter.com/CoostoNL> <https://www.facebook.com/coosto.solution
> >
> > <https://www.instagram.com/coosto_com/>
> > <https://www.linkedin.com/company/coosto>
> >
>
--
Met vriendelijke groet / Kind regards,
*Lud Antonie*
<https://www.coosto.com/>
Kennedyplein 101, 5611 ZS, Eindhoven
+31(0)402492700 <0031402492700>
www.coosto.com
<https://twitter.com/CoostoNL> <https://www.facebook.com/coosto.solution>
<https://www.instagram.com/coosto_com/>
<https://www.linkedin.com/company/coosto>
Re: Kafka 2.7.2 to 3.5.1 upgrade
Posted by megh vidani <vi...@gmail.com>.
Hi Lud,
The topics for which you're seeing under replicated partitions, Did you try
to increase the number of partitions anytime after creation of those topics
before the upgrade?
We have earlier faced issues with 2.8.0, in which we had increased the
number of partitions for some topics, and for those topics we used to see
under replicated partitions after every restart.
The reason this happened was, there was a bug in Kafka which assigned a new
topicId (different from the original topicId) to newly added partitions in
the partition.metadata file, and upon restart of kafka brokers, this
topicId didn't reconcile between brokers and ZK.
Thanks,
Megh
On Thu, Nov 30, 2023, 20:10 Lud Antonie <lu...@coosto.com.invalid>
wrote:
> Hello,
>
> After upgrading from 2.7.2 to 3.5.1 some topics are missing a partition for
> one or two brokers.
> The kafka manager shows "Under replicated%" for the topic.
> Looking at the topic for some brokers (of 3) partitions are missing (in my
> case 1 partition).
> A rollback will restore the "Under replicated%" to 0 again (this is the
> wanted number).
>
> Is this a bug of kafka or the kafka manager?
>
> Best regards,
> Lud Antonie
>
>
> --
> Met vriendelijke groet / Kind regards,
>
> *Lud Antonie*
>
> <https://www.coosto.com/>
> Kennedyplein 101, 5611 ZS, Eindhoven
> +31(0)402492700 <0031402492700>
> www.coosto.com
> <https://twitter.com/CoostoNL> <https://www.facebook.com/coosto.solution>
> <https://www.instagram.com/coosto_com/>
> <https://www.linkedin.com/company/coosto>
>
Re: Kafka 2.7.2 to 3.5.1 upgrade
Posted by Lud Antonie <lu...@coosto.com.INVALID>.
Hi,
Just rerun the upgrade from 2.7.2 to 3.5.1 and got the same under
replicated issue:
kafka@playground2:~/kafka/bin$ ./kafka-topics.sh --describe
--under-replicated-partitions --bootstrap-server pg1:30092
Topic: smo-test Partition: 3 Leader: 1 Replicas: 1,3,2 Isr: 3,2
So it looks like something is going wrong with kafka. The kafka-manager
gives the same result.
Best regards,
Lud
Op za 2 dec 2023 om 02:18 schreef Haruki Okada <oc...@gmail.com>:
> Hi.
>
> I'm not sure if KafkaManager has such bug though, you should check if
> there's any under replicated partitions actually by `kafka-topics.sh`
> command with `--under-replicated-partitions` option first.
>
> 2023年11月30日(木) 23:41 Lud Antonie <lu...@coosto.com.invalid>:
>
> > Hello,
> >
> > After upgrading from 2.7.2 to 3.5.1 some topics are missing a partition
> for
> > one or two brokers.
> > The kafka manager shows "Under replicated%" for the topic.
> > Looking at the topic for some brokers (of 3) partitions are missing (in
> my
> > case 1 partition).
> > A rollback will restore the "Under replicated%" to 0 again (this is the
> > wanted number).
> >
> > Is this a bug of kafka or the kafka manager?
> >
> > Best regards,
> > Lud Antonie
> >
> >
> > --
> > Met vriendelijke groet / Kind regards,
> >
> > *Lud Antonie*
> >
> > <https://www.coosto.com/>
> > Kennedyplein 101, 5611 ZS, Eindhoven
> > +31(0)402492700 <0031402492700>
> > www.coosto.com
> > <https://twitter.com/CoostoNL> <https://www.facebook.com/coosto.solution
> >
> > <https://www.instagram.com/coosto_com/>
> > <https://www.linkedin.com/company/coosto>
> >
>
>
> --
> ========================
> Okada Haruki
> ocadaruma@gmail.com
> ========================
>
--
Met vriendelijke groet / Kind regards,
*Lud Antonie*
<https://www.coosto.com/>
Kennedyplein 101, 5611 ZS, Eindhoven
+31(0)402492700 <0031402492700>
www.coosto.com
<https://twitter.com/CoostoNL> <https://www.facebook.com/coosto.solution>
<https://www.instagram.com/coosto_com/>
<https://www.linkedin.com/company/coosto>
Re: Kafka 2.7.2 to 3.5.1 upgrade
Posted by Haruki Okada <oc...@gmail.com>.
Hi.
I'm not sure if KafkaManager has such bug though, you should check if
there's any under replicated partitions actually by `kafka-topics.sh`
command with `--under-replicated-partitions` option first.
2023年11月30日(木) 23:41 Lud Antonie <lu...@coosto.com.invalid>:
> Hello,
>
> After upgrading from 2.7.2 to 3.5.1 some topics are missing a partition for
> one or two brokers.
> The kafka manager shows "Under replicated%" for the topic.
> Looking at the topic for some brokers (of 3) partitions are missing (in my
> case 1 partition).
> A rollback will restore the "Under replicated%" to 0 again (this is the
> wanted number).
>
> Is this a bug of kafka or the kafka manager?
>
> Best regards,
> Lud Antonie
>
>
> --
> Met vriendelijke groet / Kind regards,
>
> *Lud Antonie*
>
> <https://www.coosto.com/>
> Kennedyplein 101, 5611 ZS, Eindhoven
> +31(0)402492700 <0031402492700>
> www.coosto.com
> <https://twitter.com/CoostoNL> <https://www.facebook.com/coosto.solution>
> <https://www.instagram.com/coosto_com/>
> <https://www.linkedin.com/company/coosto>
>
--
========================
Okada Haruki
ocadaruma@gmail.com
========================