ISR differs between Kafka Metadata and Zookeeper

[Florian Weingarten <fl...@shopify.com>]

Hi list,

I am trying to debug a strange issue we are seeing. We are using
"Sarama" [1], our own Go implementation of the Kafka API.

Somehow, we either found a bug in Kafka, have a bug in our own code, or
got our cluster into a weird state. If we use our library to query Kafka
for metadata about any partition, it often (but not always!) returns an
ISR that is a strict subset of the replica set.

However, if I use the "bin/kafka-list-topic.sh" which comes with Kafka,
it shows the two sets as equal. I believe that tool gets its data from
Zookeeper. If I look at Zookeeper directly, I also see matching sets.

Does anybody have any idea what could cause this? What's the most
reliable way to get this data? Kafka or Zookeeper?

Any idea why we are seeing this? And if it's not a bug in the reporting,
what could have caused the replicas to not be in-sync (and how do we
trigger them to catch up? They don't seem to do it automatically).

Thanks!

Flo

[1]: https://github.com/Shopify/sarama

Re: ISR differs between Kafka Metadata and Zookeeper

[Joel Koshy <jj...@gmail.com>]

This may be due to https://issues.apache.org/jira/browse/KAFKA-1367

On Thu, Oct 03, 2013 at 04:17:45PM -0400, Florian Weingarten wrote:
> Hi list,
> 
> I am trying to debug a strange issue we are seeing. We are using
> "Sarama" [1], our own Go implementation of the Kafka API.
> 
> Somehow, we either found a bug in Kafka, have a bug in our own code, or
> got our cluster into a weird state. If we use our library to query Kafka
> for metadata about any partition, it often (but not always!) returns an
> ISR that is a strict subset of the replica set.
> 
> However, if I use the "bin/kafka-list-topic.sh" which comes with Kafka,
> it shows the two sets as equal. I believe that tool gets its data from
> Zookeeper. If I look at Zookeeper directly, I also see matching sets.
> 
> Does anybody have any idea what could cause this? What's the most
> reliable way to get this data? Kafka or Zookeeper?
> 
> Any idea why we are seeing this? And if it's not a bug in the reporting,
> what could have caused the replicas to not be in-sync (and how do we
> trigger them to catch up? They don't seem to do it automatically).
> 
> Thanks!
> 
> Flo
> 
> [1]: https://github.com/Shopify/sarama

Re: Restore zookeeper metadata

[Neha Narkhede <ne...@gmail.com>]

There is no easy way to move zookeeper data other than stopping all kafka
servers and clients, moving the zookeeper data manually and restarting the
server and clients with the new zookeeper url.

Thanks,
Neha
On Oct 7, 2013 10:12 PM, "Snehalata Nagaje" <
snehalata.nagaje@harbingergroup.com> wrote:

> Hi ,
>
> We are using central standalone zookeeper for kafka and HBase .
>
> Due to some problem, we have installed new zookeeper on different machine,
> but since we do not have old metadata available in Zookeeper required for
> kafka.
>
> We are not able to read previous topic messages.
>
> How we can restore old data to restore all metadata?
>
>
>
> Thanks,
> Snehalata
>
>
> Disclaimer:
> This e-mail may contain Privileged/Confidential information and is
> intended only for the individual(s) named. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance upon
> this information by persons or entities other than the intended recipient
> is prohibited. Please notify the sender, if you have received this e-mail
> by mistake and delete it from your system. Information in this message that
> does not relate to the official business of the company shall be understood
> as neither given nor endorsed by it. E-mail transmission cannot be
> guaranteed to be secure or error-free. The sender does not accept liability
> for any errors or omissions in the contents of this message which arise as
> a result of e-mail transmission. If verification is required please request
> a hard-copy version. Visit us at http://www.harbingergroup.com/
>

Re: Restore zookeeper metadata

[Jun Rao <ju...@gmail.com>]

Are you running Kafka 0.7 or 0.8?

Thanks,

Jun


On Mon, Oct 7, 2013 at 10:09 PM, Snehalata Nagaje <
snehalata.nagaje@harbingergroup.com> wrote:

> Hi ,
>
> We are using central standalone zookeeper for kafka and HBase .
>
> Due to some problem, we have installed new zookeeper on different machine,
> but since we do not have old metadata available in Zookeeper required for
> kafka.
>
> We are not able to read previous topic messages.
>
> How we can restore old data to restore all metadata?
>
>
>
> Thanks,
> Snehalata
>
>
> Disclaimer:
> This e-mail may contain Privileged/Confidential information and is
> intended only for the individual(s) named. Any review, retransmission,
> dissemination or other use of, or taking of any action in reliance upon
> this information by persons or entities other than the intended recipient
> is prohibited. Please notify the sender, if you have received this e-mail
> by mistake and delete it from your system. Information in this message that
> does not relate to the official business of the company shall be understood
> as neither given nor endorsed by it. E-mail transmission cannot be
> guaranteed to be secure or error-free. The sender does not accept liability
> for any errors or omissions in the contents of this message which arise as
> a result of e-mail transmission. If verification is required please request
> a hard-copy version. Visit us at http://www.harbingergroup.com/
>

Restore zookeeper metadata

[Snehalata Nagaje <sn...@harbingergroup.com>]

Hi ,

We are using central standalone zookeeper for kafka and HBase .

Due to some problem, we have installed new zookeeper on different machine, 
but since we do not have old metadata available in Zookeeper required for 
kafka.

We are not able to read previous topic messages.

How we can restore old data to restore all metadata?



Thanks,
Snehalata
 

Disclaimer:
This e-mail may contain Privileged/Confidential information and is intended only for the individual(s) named. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. Please notify the sender, if you have received this e-mail by mistake and delete it from your system. Information in this message that does not relate to the official business of the company shall be understood as neither given nor endorsed by it. E-mail transmission cannot be guaranteed to be secure or error-free. The sender does not accept liability for any errors or omissions in the contents of this message which arise as a result of e-mail transmission. If verification is required please request a hard-copy version. Visit us at http://www.harbingergroup.com/

Re: ISR differs between Kafka Metadata and Zookeeper

[Andrew Otto <ot...@wikimedia.org>]

I am seeing this behavior using librdkafka, as is another user.

Listing the topic metadata with the tool provided with Kafka
(kafka-topic.sh) shows all replicas in the ISR.  However, using kafkacat[1]
(built with librdkafka) shows that many ISRs are missing some replicas.  I
talked with Magnus (author of librdkafka), and he asked me to get a tcpdump
of the response returned by the controller broker to kafkacat.  The tcpdump
showed that the broker itself was returning the ISRs missing some replicas.

More information about this can be found here:
https://github.com/edenhill/librdkafka/issues/147

This doesn't seem to be a transient problem, as the results returned by
kafkacat are consistent.

[1] https://github.com/edenhill/kafkacat



On Thu, Oct 3, 2013 at 5:43 PM, Neha Narkhede <ne...@gmail.com>
wrote:

> When you issue a metadata request to any broker, it responds with the
> cluster metadata in memory. The broker gets it through an UpdateMetadata
> request issued by the controller broker. If there are state changes in
> flight, there may be a delay in the UpdateMetadata request propagation to
> the broker serving the TopicMetadataRequest. However, this behavior is
> transient and in steady state TopicMetadataRequest should always return
> up-to-date information.
>
> Thanks,
> Neha
>
>
> On Thu, Oct 3, 2013 at 1:17 PM, Florian Weingarten <fl...@shopify.com>
> wrote:
>
> > Hi list,
> >
> > I am trying to debug a strange issue we are seeing. We are using
> > "Sarama" [1], our own Go implementation of the Kafka API.
> >
> > Somehow, we either found a bug in Kafka, have a bug in our own code, or
> > got our cluster into a weird state. If we use our library to query Kafka
> > for metadata about any partition, it often (but not always!) returns an
> > ISR that is a strict subset of the replica set.
> >
> > However, if I use the "bin/kafka-list-topic.sh" which comes with Kafka,
> > it shows the two sets as equal. I believe that tool gets its data from
> > Zookeeper. If I look at Zookeeper directly, I also see matching sets.
> >
> > Does anybody have any idea what could cause this? What's the most
> > reliable way to get this data? Kafka or Zookeeper?
> >
> > Any idea why we are seeing this? And if it's not a bug in the reporting,
> > what could have caused the replicas to not be in-sync (and how do we
> > trigger them to catch up? They don't seem to do it automatically).
> >
> > Thanks!
> >
> > Flo
> >
> > [1]: https://github.com/Shopify/sarama
> >
>

Re: ISR differs between Kafka Metadata and Zookeeper

[Neha Narkhede <ne...@gmail.com>]

When you issue a metadata request to any broker, it responds with the
cluster metadata in memory. The broker gets it through an UpdateMetadata
request issued by the controller broker. If there are state changes in
flight, there may be a delay in the UpdateMetadata request propagation to
the broker serving the TopicMetadataRequest. However, this behavior is
transient and in steady state TopicMetadataRequest should always return
up-to-date information.

Thanks,
Neha


On Thu, Oct 3, 2013 at 1:17 PM, Florian Weingarten <fl...@shopify.com> wrote:

> Hi list,
>
> I am trying to debug a strange issue we are seeing. We are using
> "Sarama" [1], our own Go implementation of the Kafka API.
>
> Somehow, we either found a bug in Kafka, have a bug in our own code, or
> got our cluster into a weird state. If we use our library to query Kafka
> for metadata about any partition, it often (but not always!) returns an
> ISR that is a strict subset of the replica set.
>
> However, if I use the "bin/kafka-list-topic.sh" which comes with Kafka,
> it shows the two sets as equal. I believe that tool gets its data from
> Zookeeper. If I look at Zookeeper directly, I also see matching sets.
>
> Does anybody have any idea what could cause this? What's the most
> reliable way to get this data? Kafka or Zookeeper?
>
> Any idea why we are seeing this? And if it's not a bug in the reporting,
> what could have caused the replicas to not be in-sync (and how do we
> trigger them to catch up? They don't seem to do it automatically).
>
> Thanks!
>
> Flo
>
> [1]: https://github.com/Shopify/sarama
>