API to query cluster metadata on-demand

[Simon Cooper <si...@featurespace.co.uk>]

Is there a basic interface in the new client APIs to get the list of topics on a cluster, and get information on the topics (offsets, sizes, etc), without having to deal with a producer or consumer? I just want a basic synchronous API to query the metadata as-is. Does this exist in some form?

Thanks,
Simon

Re: API to query cluster metadata on-demand

[Todd Palino <tp...@gmail.com>]

What Gwen said :)

We developed a python web service internally called Menagerie that provides
this functionality for both Kafka and Zookeeper. We use it to drive a web
dashboard for stats, our (old style) lag checking, and some other CLI
tools. Unfortunately it ties into too much internal LinkedIn tooling for us
to open source.

That's one of the reasons we released Burrow (
https://github.com/linkedin/Burrow). The primary use is to do lag checking
for consumers as a service. But I'm also moving functionality from
Menagerie into it. Right now you can use it to fetch topic lists, partition
counts, and broker offsets. You can also get information for consumers (as
long as they are committing offsets to Kafka and not ZK).

If it looks useful and there's some bit of info you'd like it to provide,
you can submit a github issue and I'll take a look at it.

-Todd

On Thursday, September 3, 2015, Andrew Otto <ao...@wikimedia.org> wrote:

> If you don’t mind doing it with a C CLI:
>
> https://github.com/edenhill/kafkacat
>
> $ kafkacat -L -b mybroker
>
> But, uhhh, you probably want a something in the Java API.
>
> :)
>
>
> > On Sep 3, 2015, at 13:58, Gwen Shapira <gwen@confluent.io <javascript:;>>
> wrote:
> >
> > Ah, I wish.
> >
> > We are working on it :)
> >
> > On Thu, Sep 3, 2015 at 9:10 AM, Simon Cooper <
> > simon.cooper@featurespace.co.uk <javascript:;>> wrote:
> >
> >> Is there a basic interface in the new client APIs to get the list of
> >> topics on a cluster, and get information on the topics (offsets, sizes,
> >> etc), without having to deal with a producer or consumer? I just want a
> >> basic synchronous API to query the metadata as-is. Does this exist in
> some
> >> form?
> >>
> >> Thanks,
> >> Simon
> >>
>
>

Re: API to query cluster metadata on-demand

[Andrew Otto <ao...@wikimedia.org>]

If you don’t mind doing it with a C CLI:

https://github.com/edenhill/kafkacat

$ kafkacat -L -b mybroker

But, uhhh, you probably want a something in the Java API.

:)


> On Sep 3, 2015, at 13:58, Gwen Shapira <gw...@confluent.io> wrote:
> 
> Ah, I wish.
> 
> We are working on it :)
> 
> On Thu, Sep 3, 2015 at 9:10 AM, Simon Cooper <
> simon.cooper@featurespace.co.uk> wrote:
> 
>> Is there a basic interface in the new client APIs to get the list of
>> topics on a cluster, and get information on the topics (offsets, sizes,
>> etc), without having to deal with a producer or consumer? I just want a
>> basic synchronous API to query the metadata as-is. Does this exist in some
>> form?
>> 
>> Thanks,
>> Simon
>> 

Re: API to query cluster metadata on-demand

[Gwen Shapira <gw...@confluent.io>]

Ah, I wish.

We are working on it :)

On Thu, Sep 3, 2015 at 9:10 AM, Simon Cooper <
simon.cooper@featurespace.co.uk> wrote:

> Is there a basic interface in the new client APIs to get the list of
> topics on a cluster, and get information on the topics (offsets, sizes,
> etc), without having to deal with a producer or consumer? I just want a
> basic synchronous API to query the metadata as-is. Does this exist in some
> form?
>
> Thanks,
> Simon
>