You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Nicolas Carlot <ni...@chronopost.fr.INVALID> on 2022/05/09 15:30:59 UTC
Topic without Leader / ISR
Hello,
I have a situation where when creating a new topic it stays in the
following state, having not a single ISR, no leader for any partition,
nothing.
Of course I cannot neitherconsume not produce from/to this topic.
Topic: douane.request.amendment.traitementdecla-to-custmestrans
Partition: 0 Leader: none Replicas: 1,2,3 Isr:
Topic: douane.request.amendment.traitementdecla-to-custmestrans
Partition: 1 Leader: none Replicas: 2,3,1 Isr:
Topic: douane.request.amendment.traitementdecla-to-custmestrans
Partition: 2 Leader: none Replicas: 3,1,2 Isr:
Topic: douane.request.amendment.traitementdecla-to-custmestrans
Partition: 3 Leader: none Replicas: 1,3,2 Isr:
Topic: douane.request.amendment.traitementdecla-to-custmestrans
Partition: 4 Leader: none Replicas: 2,1,3 Isr:
Topic: douane.request.amendment.traitementdecla-to-custmestrans
Partition: 5 Leader: none Replicas: 3,2,1 Isr:
Topic: douane.request.amendment.traitementdecla-to-custmestrans
Partition: 6 Leader: none Replicas: 1,2,3 Isr:
Topic: douane.request.amendment.traitementdecla-to-custmestrans
Partition: 7 Leader: none Replicas: 2,3,1 Isr:
Topic: douane.request.amendment.traitementdecla-to-custmestrans
Partition: 8 Leader: none Replicas: 3,1,2 Isr:
Topic: douane.request.amendment.traitementdecla-to-custmestrans
Partition: 9 Leader: none Replicas: 1,3,2 Isr:
Any idea of what may be happening ?
--
[image: img] <https://www.chronopost.fr/fr?xtatc=INT-149>
*Nicolas Carlot*
*Lead dev*Direction des Systèmes d'Information
3 boulevard Romain Rolland
75014 Paris
[image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ> [image:
img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
[image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3B3PQ-K1pRPQ>
[image: img]
Re: Topic without Leader / ISR
Posted by Nicolas Carlot <ni...@chronopost.fr.INVALID>.
Those logs where found on one of the 3 brokers
I can't find anything related to this topic on any of the cluster's nodes.
Brokers' version is 2.8.1
I may have a lead, but this is weird.
I restarted one of the brokers, node1, it sent tons of errors which seem
inconsistent on the "available brokers" count.
[2022-05-10 17:12:24,430] INFO [Admin Manager on Broker 1]: Error
processing create topic request
CreatableTopic(name='douane.disjoint.traitementdecla-to-eai',
numPartitions=1, replicationFactor=3, assignments=[], configs=[])
(kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException:
Replication factor: 3 larger than available brokers: 1.
[2022-05-10 17:12:24,430] INFO [Admin Manager on Broker 1]: Error
processing create topic request
CreatableTopic(name='douane.disjoint.eai-to-traitementdecla',
numPartitions=1, replicationFactor=3, assignments=[], configs=[])
(kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException:
Replication factor: 3 larger than available brokers: 0.
[2022-05-10 17:12:24,430] INFO [Admin Manager on Broker 1]: Error
processing create topic request
CreatableTopic(name='douane.declaration.custmestrans-to-traitementdecla',
numPartitions=1, replicationFactor=3, assignments=[], configs=[])
(kafka.server.ZkAdminManager)
org.apache.kafka.common.errors.InvalidReplicationFactorException:
Replication factor: 3 larger than available brokers: 1.
I then stoped all brokers, restarted them one per one, still had the same
issue on node1.
After a few restart of node1, it resumed processing normaly, and my topic
finally got created...
Le mar. 10 mai 2022 à 15:37, Luke Chen <sh...@gmail.com> a écrit :
> Hi Nicolas,
>
> This is the log in "client" side.
> I think the broker side log will have more info.
> Also, which Kafka version are you using?
>
> Thank you.
> Luke
>
> On Tue, May 10, 2022 at 8:17 PM Nicolas Carlot
> <ni...@chronopost.fr.invalid> wrote:
>
> > Not a single line regarding this topic, but at creation time :/
> > [2022-05-04 14:42:28,091] INFO Creating topic
> > douane.request.amendment.traitementdecla-to-custmestranss with
> > configuration {delete.retention.ms=604800000, retention.ms=604800000,
> > cleanup.policy=delete, compression.type=lz4} and initial partition
> > assignment HashMap(0 -> ArrayBuffer(3, 1, 2), 1 -> ArrayBuffer(1, 2, 3),
> 2
> > -> ArrayBuffer(2, 3, 1), 3 -> ArrayBuffer(3, 2, 1), 4 -> ArrayBuffer(1,
> 3,
> > 2), 5 -> ArrayBuffer(2, 1, 3), 6 -> ArrayBuffer(3, 1, 2), 7 ->
> > ArrayBuffer(1, 2, 3), 8 -> ArrayBuffer(2, 3, 1), 9 -> ArrayBuffer(3, 2,
> 1))
> > (kafka.zk.AdminZkClient)
> >
> > Le mar. 10 mai 2022 à 05:15, Luke Chen <sh...@gmail.com> a écrit :
> >
> > > Hi Nicolas,
> > >
> > > Could you check the logs in broker side?
> > > There must be some errors while electing leaders or something.
> > >
> > > Thank you.
> > > Luke
> > >
> > > On Mon, May 9, 2022 at 11:31 PM Nicolas Carlot
> > > <ni...@chronopost.fr.invalid> wrote:
> > >
> > > > Hello,
> > > >
> > > > I have a situation where when creating a new topic it stays in the
> > > > following state, having not a single ISR, no leader for any
> partition,
> > > > nothing.
> > > > Of course I cannot neitherconsume not produce from/to this topic.
> > > >
> > > > Topic:
> douane.request.amendment.traitementdecla-to-custmestrans
> > > > Partition: 0 Leader: none Replicas: 1,2,3 Isr:
> > > > Topic:
> douane.request.amendment.traitementdecla-to-custmestrans
> > > > Partition: 1 Leader: none Replicas: 2,3,1 Isr:
> > > > Topic:
> douane.request.amendment.traitementdecla-to-custmestrans
> > > > Partition: 2 Leader: none Replicas: 3,1,2 Isr:
> > > > Topic:
> douane.request.amendment.traitementdecla-to-custmestrans
> > > > Partition: 3 Leader: none Replicas: 1,3,2 Isr:
> > > > Topic:
> douane.request.amendment.traitementdecla-to-custmestrans
> > > > Partition: 4 Leader: none Replicas: 2,1,3 Isr:
> > > > Topic:
> douane.request.amendment.traitementdecla-to-custmestrans
> > > > Partition: 5 Leader: none Replicas: 3,2,1 Isr:
> > > > Topic:
> douane.request.amendment.traitementdecla-to-custmestrans
> > > > Partition: 6 Leader: none Replicas: 1,2,3 Isr:
> > > > Topic:
> douane.request.amendment.traitementdecla-to-custmestrans
> > > > Partition: 7 Leader: none Replicas: 2,3,1 Isr:
> > > > Topic:
> douane.request.amendment.traitementdecla-to-custmestrans
> > > > Partition: 8 Leader: none Replicas: 3,1,2 Isr:
> > > > Topic:
> douane.request.amendment.traitementdecla-to-custmestrans
> > > > Partition: 9 Leader: none Replicas: 1,3,2 Isr:
> > > >
> > > > Any idea of what may be happening ?
> > > >
> > > > --
> > > > [image: img] <https://www.chronopost.fr/fr?xtatc=INT-149>
> > > >
> > > >
> > > > *Nicolas Carlot*
> > > > *Lead dev*Direction des Systèmes d'Information
> > > >
> > > >
> > > > 3 boulevard Romain Rolland
> > > > 75014 Paris
> > > > [image: img] <
> https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> > > > [image:
> > > > img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ>
> [image:
> > > > img]
> > > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image:
> img]
> > > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
> > > >
> > > > [image: img]
> > > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3B3PQ-K1pRPQ>
> > > >
> > > > [image: img]
> > > >
> > >
> >
> >
> > --
> > [image: img] <https://www.chronopost.fr/fr?xtatc=INT-149>
> >
> >
> > *Nicolas Carlot*
> > *Lead dev*Direction des Systèmes d'Information
> >
> >
> > 3 boulevard Romain Rolland
> > 75014 Paris
> > [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> > [image:
> > img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> > img]
> > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
> >
> > [image: img]
> > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3B3PQ-K1pRPQ>
> >
> > [image: img]
> >
>
--
[image: img] <https://www.chronopost.fr/fr?xtatc=INT-149>
*Nicolas Carlot*
*Lead dev*Direction des Systèmes d'Information
3 boulevard Romain Rolland
75014 Paris
[image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ> [image:
img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
[image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3B3PQ-K1pRPQ>
[image: img]
Re: Topic without Leader / ISR
Posted by Luke Chen <sh...@gmail.com>.
Hi Nicolas,
This is the log in "client" side.
I think the broker side log will have more info.
Also, which Kafka version are you using?
Thank you.
Luke
On Tue, May 10, 2022 at 8:17 PM Nicolas Carlot
<ni...@chronopost.fr.invalid> wrote:
> Not a single line regarding this topic, but at creation time :/
> [2022-05-04 14:42:28,091] INFO Creating topic
> douane.request.amendment.traitementdecla-to-custmestranss with
> configuration {delete.retention.ms=604800000, retention.ms=604800000,
> cleanup.policy=delete, compression.type=lz4} and initial partition
> assignment HashMap(0 -> ArrayBuffer(3, 1, 2), 1 -> ArrayBuffer(1, 2, 3), 2
> -> ArrayBuffer(2, 3, 1), 3 -> ArrayBuffer(3, 2, 1), 4 -> ArrayBuffer(1, 3,
> 2), 5 -> ArrayBuffer(2, 1, 3), 6 -> ArrayBuffer(3, 1, 2), 7 ->
> ArrayBuffer(1, 2, 3), 8 -> ArrayBuffer(2, 3, 1), 9 -> ArrayBuffer(3, 2, 1))
> (kafka.zk.AdminZkClient)
>
> Le mar. 10 mai 2022 à 05:15, Luke Chen <sh...@gmail.com> a écrit :
>
> > Hi Nicolas,
> >
> > Could you check the logs in broker side?
> > There must be some errors while electing leaders or something.
> >
> > Thank you.
> > Luke
> >
> > On Mon, May 9, 2022 at 11:31 PM Nicolas Carlot
> > <ni...@chronopost.fr.invalid> wrote:
> >
> > > Hello,
> > >
> > > I have a situation where when creating a new topic it stays in the
> > > following state, having not a single ISR, no leader for any partition,
> > > nothing.
> > > Of course I cannot neitherconsume not produce from/to this topic.
> > >
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 0 Leader: none Replicas: 1,2,3 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 1 Leader: none Replicas: 2,3,1 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 2 Leader: none Replicas: 3,1,2 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 3 Leader: none Replicas: 1,3,2 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 4 Leader: none Replicas: 2,1,3 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 5 Leader: none Replicas: 3,2,1 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 6 Leader: none Replicas: 1,2,3 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 7 Leader: none Replicas: 2,3,1 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 8 Leader: none Replicas: 3,1,2 Isr:
> > > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > > Partition: 9 Leader: none Replicas: 1,3,2 Isr:
> > >
> > > Any idea of what may be happening ?
> > >
> > > --
> > > [image: img] <https://www.chronopost.fr/fr?xtatc=INT-149>
> > >
> > >
> > > *Nicolas Carlot*
> > > *Lead dev*Direction des Systèmes d'Information
> > >
> > >
> > > 3 boulevard Romain Rolland
> > > 75014 Paris
> > > [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> > > [image:
> > > img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> > > img]
> > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
> > >
> > > [image: img]
> > > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3B3PQ-K1pRPQ>
> > >
> > > [image: img]
> > >
> >
>
>
> --
> [image: img] <https://www.chronopost.fr/fr?xtatc=INT-149>
>
>
> *Nicolas Carlot*
> *Lead dev*Direction des Systèmes d'Information
>
>
> 3 boulevard Romain Rolland
> 75014 Paris
> [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> [image:
> img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
>
> [image: img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3B3PQ-K1pRPQ>
>
> [image: img]
>
Re: Topic without Leader / ISR
Posted by Nicolas Carlot <ni...@chronopost.fr.INVALID>.
Not a single line regarding this topic, but at creation time :/
[2022-05-04 14:42:28,091] INFO Creating topic
douane.request.amendment.traitementdecla-to-custmestranss with
configuration {delete.retention.ms=604800000, retention.ms=604800000,
cleanup.policy=delete, compression.type=lz4} and initial partition
assignment HashMap(0 -> ArrayBuffer(3, 1, 2), 1 -> ArrayBuffer(1, 2, 3), 2
-> ArrayBuffer(2, 3, 1), 3 -> ArrayBuffer(3, 2, 1), 4 -> ArrayBuffer(1, 3,
2), 5 -> ArrayBuffer(2, 1, 3), 6 -> ArrayBuffer(3, 1, 2), 7 ->
ArrayBuffer(1, 2, 3), 8 -> ArrayBuffer(2, 3, 1), 9 -> ArrayBuffer(3, 2, 1))
(kafka.zk.AdminZkClient)
Le mar. 10 mai 2022 à 05:15, Luke Chen <sh...@gmail.com> a écrit :
> Hi Nicolas,
>
> Could you check the logs in broker side?
> There must be some errors while electing leaders or something.
>
> Thank you.
> Luke
>
> On Mon, May 9, 2022 at 11:31 PM Nicolas Carlot
> <ni...@chronopost.fr.invalid> wrote:
>
> > Hello,
> >
> > I have a situation where when creating a new topic it stays in the
> > following state, having not a single ISR, no leader for any partition,
> > nothing.
> > Of course I cannot neitherconsume not produce from/to this topic.
> >
> > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > Partition: 0 Leader: none Replicas: 1,2,3 Isr:
> > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > Partition: 1 Leader: none Replicas: 2,3,1 Isr:
> > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > Partition: 2 Leader: none Replicas: 3,1,2 Isr:
> > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > Partition: 3 Leader: none Replicas: 1,3,2 Isr:
> > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > Partition: 4 Leader: none Replicas: 2,1,3 Isr:
> > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > Partition: 5 Leader: none Replicas: 3,2,1 Isr:
> > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > Partition: 6 Leader: none Replicas: 1,2,3 Isr:
> > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > Partition: 7 Leader: none Replicas: 2,3,1 Isr:
> > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > Partition: 8 Leader: none Replicas: 3,1,2 Isr:
> > Topic: douane.request.amendment.traitementdecla-to-custmestrans
> > Partition: 9 Leader: none Replicas: 1,3,2 Isr:
> >
> > Any idea of what may be happening ?
> >
> > --
> > [image: img] <https://www.chronopost.fr/fr?xtatc=INT-149>
> >
> >
> > *Nicolas Carlot*
> > *Lead dev*Direction des Systèmes d'Information
> >
> >
> > 3 boulevard Romain Rolland
> > 75014 Paris
> > [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> > [image:
> > img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> > img]
> > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
> >
> > [image: img]
> > <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3B3PQ-K1pRPQ>
> >
> > [image: img]
> >
>
--
[image: img] <https://www.chronopost.fr/fr?xtatc=INT-149>
*Nicolas Carlot*
*Lead dev*Direction des Systèmes d'Information
3 boulevard Romain Rolland
75014 Paris
[image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ> [image:
img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
[image: img]
<https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3B3PQ-K1pRPQ>
[image: img]
Re: Topic without Leader / ISR
Posted by Luke Chen <sh...@gmail.com>.
Hi Nicolas,
Could you check the logs in broker side?
There must be some errors while electing leaders or something.
Thank you.
Luke
On Mon, May 9, 2022 at 11:31 PM Nicolas Carlot
<ni...@chronopost.fr.invalid> wrote:
> Hello,
>
> I have a situation where when creating a new topic it stays in the
> following state, having not a single ISR, no leader for any partition,
> nothing.
> Of course I cannot neitherconsume not produce from/to this topic.
>
> Topic: douane.request.amendment.traitementdecla-to-custmestrans
> Partition: 0 Leader: none Replicas: 1,2,3 Isr:
> Topic: douane.request.amendment.traitementdecla-to-custmestrans
> Partition: 1 Leader: none Replicas: 2,3,1 Isr:
> Topic: douane.request.amendment.traitementdecla-to-custmestrans
> Partition: 2 Leader: none Replicas: 3,1,2 Isr:
> Topic: douane.request.amendment.traitementdecla-to-custmestrans
> Partition: 3 Leader: none Replicas: 1,3,2 Isr:
> Topic: douane.request.amendment.traitementdecla-to-custmestrans
> Partition: 4 Leader: none Replicas: 2,1,3 Isr:
> Topic: douane.request.amendment.traitementdecla-to-custmestrans
> Partition: 5 Leader: none Replicas: 3,2,1 Isr:
> Topic: douane.request.amendment.traitementdecla-to-custmestrans
> Partition: 6 Leader: none Replicas: 1,2,3 Isr:
> Topic: douane.request.amendment.traitementdecla-to-custmestrans
> Partition: 7 Leader: none Replicas: 2,3,1 Isr:
> Topic: douane.request.amendment.traitementdecla-to-custmestrans
> Partition: 8 Leader: none Replicas: 3,1,2 Isr:
> Topic: douane.request.amendment.traitementdecla-to-custmestrans
> Partition: 9 Leader: none Replicas: 1,3,2 Isr:
>
> Any idea of what may be happening ?
>
> --
> [image: img] <https://www.chronopost.fr/fr?xtatc=INT-149>
>
>
> *Nicolas Carlot*
> *Lead dev*Direction des Systèmes d'Information
>
>
> 3 boulevard Romain Rolland
> 75014 Paris
> [image: img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3A4PQ>
> [image:
> img] <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pzPQ> [image:
> img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pRPQ> [image: img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L1pVPQ>
>
> [image: img]
> <https://mailsign.chronopost.fr/linkc/K0ppSHlnPT0-L3B3PQ-K1pRPQ>
>
> [image: img]
>