Re: ISR differs between Kafka Metadata and Zookeeper

[Andrew Otto <ot...@wikimedia.org>]

I am seeing this behavior using librdkafka, as is another user.

Listing the topic metadata with the tool provided with Kafka
(kafka-topic.sh) shows all replicas in the ISR.  However, using kafkacat[1]
(built with librdkafka) shows that many ISRs are missing some replicas.  I
talked with Magnus (author of librdkafka), and he asked me to get a tcpdump
of the response returned by the controller broker to kafkacat.  The tcpdump
showed that the broker itself was returning the ISRs missing some replicas.

More information about this can be found here:
https://github.com/edenhill/librdkafka/issues/147

This doesn't seem to be a transient problem, as the results returned by
kafkacat are consistent.

[1] https://github.com/edenhill/kafkacat



On Thu, Oct 3, 2013 at 5:43 PM, Neha Narkhede <ne...@gmail.com>
wrote:

> When you issue a metadata request to any broker, it responds with the
> cluster metadata in memory. The broker gets it through an UpdateMetadata
> request issued by the controller broker. If there are state changes in
> flight, there may be a delay in the UpdateMetadata request propagation to
> the broker serving the TopicMetadataRequest. However, this behavior is
> transient and in steady state TopicMetadataRequest should always return
> up-to-date information.
>
> Thanks,
> Neha
>
>
> On Thu, Oct 3, 2013 at 1:17 PM, Florian Weingarten <fl...@shopify.com>
> wrote:
>
> > Hi list,
> >
> > I am trying to debug a strange issue we are seeing. We are using
> > "Sarama" [1], our own Go implementation of the Kafka API.
> >
> > Somehow, we either found a bug in Kafka, have a bug in our own code, or
> > got our cluster into a weird state. If we use our library to query Kafka
> > for metadata about any partition, it often (but not always!) returns an
> > ISR that is a strict subset of the replica set.
> >
> > However, if I use the "bin/kafka-list-topic.sh" which comes with Kafka,
> > it shows the two sets as equal. I believe that tool gets its data from
> > Zookeeper. If I look at Zookeeper directly, I also see matching sets.
> >
> > Does anybody have any idea what could cause this? What's the most
> > reliable way to get this data? Kafka or Zookeeper?
> >
> > Any idea why we are seeing this? And if it's not a bug in the reporting,
> > what could have caused the replicas to not be in-sync (and how do we
> > trigger them to catch up? They don't seem to do it automatically).
> >
> > Thanks!
> >
> > Flo
> >
> > [1]: https://github.com/Shopify/sarama
> >
>