You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Pankaj Mittal <pa...@livestream.com> on 2014/01/21 10:54:28 UTC

Want some Metrics

Hi,

We are using spark with KafkaDstream. Now How can we monitor spark and get
some metrics like rate at which messages are being received or processed
w.r.t. a worker or cluster wide ?

Thanks
Pankaj

Re: Want some Metrics

Posted by Pankaj Mittal <pa...@livestream.com>.

Hi,

We enabled JmxSink in metric.properties, but we see minimal metrics like
memory usage and all. How can we introduce metrics at application level
like throughput (message processed / sec).
We tried to create MetricRegistry and increment counter in Map function but
that doesn't help.
Or is there any other way to benchmark the our application against spark ?

Thanks
Pankaj

On Tue, Jan 21, 2014 at 3:24 PM, Pankaj Mittal <pankaj.mittal@livestream.com
> wrote:

> Hi,
>
> We are using spark with KafkaDstream. Now How can we monitor spark and get
> some metrics like rate at which messages are being received or processed
> w.r.t. a worker or cluster wide ?
>
> Thanks
> Pankaj
>