Reg: Need info on Kafka Brokers

[Sumit Maheshwari <su...@gmail.com>]

Hi,

I am looking to get information about individual brokers in kafka cluster.
The information that I am looking for is:

   - List of topics in a broker
   - Partitions for each topic in a broker
   - Metrics like BytesIn/Out Per min, Messages In/Min per topic
   - ...

I have tried looking into the JMX metrics seems but they does not provide
the above information.
Is it possible to get these information from the broker itself without
contacting zookeeper?

Thanks,
Sumit

Re: Reg: Need info on Kafka Brokers

[Sreejith S <sr...@gmail.com>]

Hi Sumit,

JMX Metrics will give you lot of in depth information on Kafka.

Just try this.
https://github.com/srijiths/kafka-connectors/tree/master/kafka-connect-jmx

If you use above , then you should have a custom UI to show the metrics.

Also you can try open source kafka monitoring tool

https://github.com/yahoo/kafka-manager

You can find lot of other open source kafka monitoring tools as well.

Regards,
Srijith

On Mon, Jan 2, 2017 at 6:13 PM, Stevo Slavić <ss...@gmail.com> wrote:

> Hello Sumit,
>
> Not yet. AFAIK few topic management requests got introduced in broker side
> (see https://issues.apache.org/jira/browse/KAFKA-2229 ) but not yet in
> client APIs. Querying/listing topics request doesn't seem even to be
> planned yet.
>
> AdminUtils which talks with ZooKeeper via ZkClient is Kafka client API one
> can use to retrieve some information, if you're building your own
> monitoring tools or reporting custom metrics. There are already multiple
> OSS and closed source commercial products for monitoring Kafka which have
> this functionality.
>
> Kind regards,
> Stevo Slavic.
>
> On Mon, Jan 2, 2017 at 1:23 PM, Sumit Maheshwari <su...@gmail.com>
> wrote:
>
> > Hi,
> >
> > I am looking to get information about individual brokers in kafka
> cluster.
> > The information that I am looking for is:
> >
> >    - List of topics in a broker
> >    - Partitions for each topic in a broker
> >    - Metrics like BytesIn/Out Per min, Messages In/Min per topic
> >    - ...
> >
> > I have tried looking into the JMX metrics seems but they does not provide
> > the above information.
> > Is it possible to get these information from the broker itself without
> > contacting zookeeper?
> >
> > Thanks,
> > Sumit
> >
>



-- 


*Sreejith.S*
https://github.com/srijiths/
http://srijiths.wordpress.com/
tweet2sree@twitter <http://tweet2Sree>

Re: Reg: Need info on Kafka Brokers

[Stevo Slavić <ss...@gmail.com>]

Hello Sumit,

Not yet. AFAIK few topic management requests got introduced in broker side
(see https://issues.apache.org/jira/browse/KAFKA-2229 ) but not yet in
client APIs. Querying/listing topics request doesn't seem even to be
planned yet.

AdminUtils which talks with ZooKeeper via ZkClient is Kafka client API one
can use to retrieve some information, if you're building your own
monitoring tools or reporting custom metrics. There are already multiple
OSS and closed source commercial products for monitoring Kafka which have
this functionality.

Kind regards,
Stevo Slavic.

On Mon, Jan 2, 2017 at 1:23 PM, Sumit Maheshwari <su...@gmail.com>
wrote:

> Hi,
>
> I am looking to get information about individual brokers in kafka cluster.
> The information that I am looking for is:
>
>    - List of topics in a broker
>    - Partitions for each topic in a broker
>    - Metrics like BytesIn/Out Per min, Messages In/Min per topic
>    - ...
>
> I have tried looking into the JMX metrics seems but they does not provide
> the above information.
> Is it possible to get these information from the broker itself without
> contacting zookeeper?
>
> Thanks,
> Sumit
>