You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by ravi kanth <ra...@gmail.com> on 2020/06/03 00:55:41 UTC
Hadoop monitoring using Prometheus
Hi Everyone,
We have a production-ready cluster with 35 nodes that we are currently
using. We are currently using System metrics using Prometheus + Grafana to
visualize server. However, we are more interested in visualizing the Hadoop
& Yarn service level metrics.
I came across hadoop JMX port which exposes all the needed metrics from the
service. However, I remained unsuccessful in tagging these metrics to
prometheus jmx agent.
Is there anyone who successfully got the JMX monitoring of Hadoop
components work with Prometheus? Any help is greatly appreciated.
Currently, we have scripts to parse the meaningful values out of JMX end
point of the namenode & datanodes.
Thanks In advance,
Ravi
Re: Hadoop monitoring using Prometheus
Posted by "संजीव (Sanjeev Tripurari)" <sa...@gmail.com>.
Hi,
We are collecting jmx metrics from the web api, and pushing it to
prometheus,
but for yarn applications its very difficult to collect data and push, same
issue with oozie.
Is anybody collecting application level or job level YARN and OOZIE metrics
for prometheus?
Regards
-Sanjeev
On Wed, 3 Jun 2020 at 06:26, ravi kanth <ra...@gmail.com> wrote:
> Hi Everyone,
>
> We have a production-ready cluster with 35 nodes that we are currently
> using. We are currently using System metrics using Prometheus + Grafana to
> visualize server. However, we are more interested in visualizing the Hadoop
> & Yarn service level metrics.
>
> I came across hadoop JMX port which exposes all the needed metrics from
> the service. However, I remained unsuccessful in tagging these metrics to
> prometheus jmx agent.
>
> Is there anyone who successfully got the JMX monitoring of Hadoop
> components work with Prometheus? Any help is greatly appreciated.
>
> Currently, we have scripts to parse the meaningful values out of JMX end
> point of the namenode & datanodes.
>
> Thanks In advance,
>
> Ravi
>
>
Re: Hadoop monitoring using Prometheus
Posted by Venkatesulu Guntakindapalli <ve...@media.net>.
Hi,
You can use JMX exporter that gives you a protheus target, this
https://github.com/prometheus/jmx_exporter. will help in your case.
On Wed, Jun 3, 2020 at 3:33 PM Sanel Zukan <sa...@gmail.com> wrote:
> ravi kanth <ra...@gmail.com> writes:
> > Wei-Chiu,
> >
> > Thanks for the response. Are there any workarounds for Hadoop 3.1.0? As
> > this is a critical production cluster it needs immediate attention.
>
> My approach was a bit different.
>
> Instead of using Prometheus to pull the metrics, I used collectd [1] to
> collect system and JMX metrics, send that to Riemann [2] and store it in
> database (in my case OpenTSDB, but changing it to Prometheus pushgateway
> address would be couple of lines of code in Riemann).
>
> Inside Riemann, you can tag, shape and calculate metrics the way you
> like.
>
> There is also standalone app [3] you can run to watch for multiple JMX
> connections, but I haven't tried that.
>
> [1] https://collectd.org/
> [2] http://riemann.io/
> [3] https://github.com/twosigma/riemann-jmx
>
> > Thanks,
> > Ravi
>
> Best,
> Sanel
>
> > On Tue, Jun 2, 2020 at 5:59 PM Wei-Chiu Chuang <we...@cloudera.com>
> wrote:
> >
> >> Check out HADOOP-16398
> >> <https://issues.apache.org/jira/browse/HADOOP-16398>
> >> It's a new feature in Hadoop 3.3.0
> >>
> >> Akira might be able to help.
> >>
> >> On Tue, Jun 2, 2020 at 5:56 PM ravi kanth <ra...@gmail.com> wrote:
> >>
> >>> Hi Everyone,
> >>>
> >>> We have a production-ready cluster with 35 nodes that we are currently
> >>> using. We are currently using System metrics using Prometheus +
> Grafana to
> >>> visualize server. However, we are more interested in visualizing the
> Hadoop
> >>> & Yarn service level metrics.
> >>>
> >>> I came across hadoop JMX port which exposes all the needed metrics from
> >>> the service. However, I remained unsuccessful in tagging these metrics
> to
> >>> prometheus jmx agent.
> >>>
> >>> Is there anyone who successfully got the JMX monitoring of Hadoop
> >>> components work with Prometheus? Any help is greatly appreciated.
> >>>
> >>> Currently, we have scripts to parse the meaningful values out of JMX
> end
> >>> point of the namenode & datanodes.
> >>>
> >>> Thanks In advance,
> >>>
> >>> Ravi
> >>>
> >>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: user-help@hadoop.apache.org
>
>
--
Thanks & Regards,
Venkatesh
SRE, Media.net (Data Platform)
Flock Id- venkatesulu.g@media.net
Contact - 9949522101
Re: Hadoop monitoring using Prometheus
Posted by Sanel Zukan <sa...@gmail.com>.
ravi kanth <ra...@gmail.com> writes:
> Wei-Chiu,
>
> Thanks for the response. Are there any workarounds for Hadoop 3.1.0? As
> this is a critical production cluster it needs immediate attention.
My approach was a bit different.
Instead of using Prometheus to pull the metrics, I used collectd [1] to
collect system and JMX metrics, send that to Riemann [2] and store it in
database (in my case OpenTSDB, but changing it to Prometheus pushgateway
address would be couple of lines of code in Riemann).
Inside Riemann, you can tag, shape and calculate metrics the way you
like.
There is also standalone app [3] you can run to watch for multiple JMX
connections, but I haven't tried that.
[1] https://collectd.org/
[2] http://riemann.io/
[3] https://github.com/twosigma/riemann-jmx
> Thanks,
> Ravi
Best,
Sanel
> On Tue, Jun 2, 2020 at 5:59 PM Wei-Chiu Chuang <we...@cloudera.com> wrote:
>
>> Check out HADOOP-16398
>> <https://issues.apache.org/jira/browse/HADOOP-16398>
>> It's a new feature in Hadoop 3.3.0
>>
>> Akira might be able to help.
>>
>> On Tue, Jun 2, 2020 at 5:56 PM ravi kanth <ra...@gmail.com> wrote:
>>
>>> Hi Everyone,
>>>
>>> We have a production-ready cluster with 35 nodes that we are currently
>>> using. We are currently using System metrics using Prometheus + Grafana to
>>> visualize server. However, we are more interested in visualizing the Hadoop
>>> & Yarn service level metrics.
>>>
>>> I came across hadoop JMX port which exposes all the needed metrics from
>>> the service. However, I remained unsuccessful in tagging these metrics to
>>> prometheus jmx agent.
>>>
>>> Is there anyone who successfully got the JMX monitoring of Hadoop
>>> components work with Prometheus? Any help is greatly appreciated.
>>>
>>> Currently, we have scripts to parse the meaningful values out of JMX end
>>> point of the namenode & datanodes.
>>>
>>> Thanks In advance,
>>>
>>> Ravi
>>>
>>>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@hadoop.apache.org
For additional commands, e-mail: user-help@hadoop.apache.org
Re: Hadoop monitoring using Prometheus
Posted by ravi kanth <ra...@gmail.com>.
Wei-Chiu,
Thanks for the response. Are there any workarounds for Hadoop 3.1.0? As
this is a critical production cluster it needs immediate attention.
Thanks,
Ravi
On Tue, Jun 2, 2020 at 5:59 PM Wei-Chiu Chuang <we...@cloudera.com> wrote:
> Check out HADOOP-16398
> <https://issues.apache.org/jira/browse/HADOOP-16398>
> It's a new feature in Hadoop 3.3.0
>
> Akira might be able to help.
>
> On Tue, Jun 2, 2020 at 5:56 PM ravi kanth <ra...@gmail.com> wrote:
>
>> Hi Everyone,
>>
>> We have a production-ready cluster with 35 nodes that we are currently
>> using. We are currently using System metrics using Prometheus + Grafana to
>> visualize server. However, we are more interested in visualizing the Hadoop
>> & Yarn service level metrics.
>>
>> I came across hadoop JMX port which exposes all the needed metrics from
>> the service. However, I remained unsuccessful in tagging these metrics to
>> prometheus jmx agent.
>>
>> Is there anyone who successfully got the JMX monitoring of Hadoop
>> components work with Prometheus? Any help is greatly appreciated.
>>
>> Currently, we have scripts to parse the meaningful values out of JMX end
>> point of the namenode & datanodes.
>>
>> Thanks In advance,
>>
>> Ravi
>>
>>
Re: Hadoop monitoring using Prometheus
Posted by Wei-Chiu Chuang <we...@cloudera.com.INVALID>.
Check out HADOOP-16398 <https://issues.apache.org/jira/browse/HADOOP-16398>
It's a new feature in Hadoop 3.3.0
Akira might be able to help.
On Tue, Jun 2, 2020 at 5:56 PM ravi kanth <ra...@gmail.com> wrote:
> Hi Everyone,
>
> We have a production-ready cluster with 35 nodes that we are currently
> using. We are currently using System metrics using Prometheus + Grafana to
> visualize server. However, we are more interested in visualizing the Hadoop
> & Yarn service level metrics.
>
> I came across hadoop JMX port which exposes all the needed metrics from
> the service. However, I remained unsuccessful in tagging these metrics to
> prometheus jmx agent.
>
> Is there anyone who successfully got the JMX monitoring of Hadoop
> components work with Prometheus? Any help is greatly appreciated.
>
> Currently, we have scripts to parse the meaningful values out of JMX end
> point of the namenode & datanodes.
>
> Thanks In advance,
>
> Ravi
>
>