You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Phillip Mann <pm...@trulia.com> on 2016/08/02 17:36:33 UTC
Opening up Kafka JMX port for Kafka Consumer in Kafka Streams app
Hello all,
This is a bit of a convoluted title but we are trying to set up monitoring on our Kafka Cluster and Kafka Streams app. I currently have JMX port open on our Kafka cluster across our brokers. I am able to use a Java JMX client to get certain metrics that are helpful to us. However, the data we want to monitor the most is consumer level JMX metrics. This is made complicated because there is no documentation for what we are trying to do. Essentially we need to expose the JMX port for the Kafka Streams Consumer. Each Kafka Streams app contains a producer and consumer (group?). We want to get the offset lag metrics for the Kafka Streams consumer. Is this possible? If so, how do we do it? Thanks for the help!!
Phillip
Re: Opening up Kafka JMX port for Kafka Consumer in Kafka Streams app
Posted by Guozhang Wang <wa...@gmail.com>.
Hello Phillip,
You are right that a Kafka Streams instance has one producer and consumer
client behind the scene, and hence you can just monitor it like you are
monitoring a normal producer / consumer client.
More specifically, you can find the producer metric names as in:
http://docs.confluent.io/3.0.0/kafka/monitoring.html
And Kafka Streams has a few more metrics for its own in addition to the
consumer and producer clients, for which we are yet to add the docs for.
Guozhang
On Tue, Aug 2, 2016 at 11:49 AM, David Garcia <da...@spiceworks.com> wrote:
> Have you looked at kafka manager: https://github.com/yahoo/kafka-manager
> It provides consumer level metrics.
>
> -David
>
> On 8/2/16, 12:36 PM, "Phillip Mann" <pm...@trulia.com> wrote:
>
> Hello all,
>
> This is a bit of a convoluted title but we are trying to set up
> monitoring on our Kafka Cluster and Kafka Streams app. I currently have
> JMX port open on our Kafka cluster across our brokers. I am able to use a
> Java JMX client to get certain metrics that are helpful to us. However,
> the data we want to monitor the most is consumer level JMX metrics. This
> is made complicated because there is no documentation for what we are
> trying to do. Essentially we need to expose the JMX port for the Kafka
> Streams Consumer. Each Kafka Streams app contains a producer and consumer
> (group?). We want to get the offset lag metrics for the Kafka Streams
> consumer. Is this possible? If so, how do we do it? Thanks for the help!!
>
> Phillip
>
>
>
--
-- Guozhang
Re: Opening up Kafka JMX port for Kafka Consumer in Kafka Streams
app
Posted by David Garcia <da...@spiceworks.com>.
Have you looked at kafka manager: https://github.com/yahoo/kafka-manager
It provides consumer level metrics.
-David
On 8/2/16, 12:36 PM, "Phillip Mann" <pm...@trulia.com> wrote:
Hello all,
This is a bit of a convoluted title but we are trying to set up monitoring on our Kafka Cluster and Kafka Streams app. I currently have JMX port open on our Kafka cluster across our brokers. I am able to use a Java JMX client to get certain metrics that are helpful to us. However, the data we want to monitor the most is consumer level JMX metrics. This is made complicated because there is no documentation for what we are trying to do. Essentially we need to expose the JMX port for the Kafka Streams Consumer. Each Kafka Streams app contains a producer and consumer (group?). We want to get the offset lag metrics for the Kafka Streams consumer. Is this possible? If so, how do we do it? Thanks for the help!!
Phillip