You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@hadoop.apache.org by Benyi Wang <be...@gmail.com> on 2013/10/24 01:21:29 UTC

Questions about hadoop-metrics2.properties

1. Does hadoop metrics2 only support File and Ganglia sink?
2. Can I expose metrics as JMX, especially for customized metrics? I
created some  metrics in my mapreduce job and could successfully output
them using a FileSink. But if I use jconsole to access YARN nodemanager, I
can only see hadoop metrics e.g Hadoop/NodeManager/NodeManagerMetrices
etc.,  not mine with prefix maptask. How to setup to see maptask/reducetask
prefix metrics?
3. Is there an example using jmx? I could not find

The configuration syntax is:

  [prefix].[source|sink|jmx|].[instance].[option]

http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html

Re: Questions about hadoop-metrics2.properties

Posted by Benyi Wang <be...@gmail.com>.

Thanks, Luke. Your answer is really helpful.


On Wed, Oct 23, 2013 at 11:51 PM, Luke Lu <ll...@apache.org> wrote:

> 1. File and Ganglia are the only bundled sinks, though there are
> socket/json (for chukwa) and graphite sinks patches in the works.
> 2. Hadoop metrics (and metrics2) is mostly designed for system/process
> metrics, which means you'll need to attach jconsole to your map/reduce task
> processes to see your task metrics instrumented via metrics. What you
> actually want is probably custom job counters.
> 3. You don't need any configuration to use JMX to access metrics2, as JMX
> is currently on by default. The configuration in hadoop-metrics2.properties
> is mostly for optional sink configuration and metrics filtering.
>
> __Luke
>
>
>
> On Wed, Oct 23, 2013 at 4:21 PM, Benyi Wang <be...@gmail.com> wrote:
>
>> 1. Does hadoop metrics2 only support File and Ganglia sink?
>> 2. Can I expose metrics as JMX, especially for customized metrics? I
>> created some  metrics in my mapreduce job and could successfully output
>> them using a FileSink. But if I use jconsole to access YARN nodemanager, I
>> can only see hadoop metrics e.g Hadoop/NodeManager/NodeManagerMetrices
>> etc.,  not mine with prefix maptask. How to setup to see maptask/reducetask
>> prefix metrics?
>> 3. Is there an example using jmx? I could not find
>>
>> The configuration syntax is:
>>
>>   [prefix].[source|sink|jmx|].[instance].[option]
>>
>>
>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html
>>
>
>

Re: Questions about hadoop-metrics2.properties

Posted by Benyi Wang <be...@gmail.com>.

Thanks, Luke. Your answer is really helpful.


On Wed, Oct 23, 2013 at 11:51 PM, Luke Lu <ll...@apache.org> wrote:

> 1. File and Ganglia are the only bundled sinks, though there are
> socket/json (for chukwa) and graphite sinks patches in the works.
> 2. Hadoop metrics (and metrics2) is mostly designed for system/process
> metrics, which means you'll need to attach jconsole to your map/reduce task
> processes to see your task metrics instrumented via metrics. What you
> actually want is probably custom job counters.
> 3. You don't need any configuration to use JMX to access metrics2, as JMX
> is currently on by default. The configuration in hadoop-metrics2.properties
> is mostly for optional sink configuration and metrics filtering.
>
> __Luke
>
>
>
> On Wed, Oct 23, 2013 at 4:21 PM, Benyi Wang <be...@gmail.com> wrote:
>
>> 1. Does hadoop metrics2 only support File and Ganglia sink?
>> 2. Can I expose metrics as JMX, especially for customized metrics? I
>> created some  metrics in my mapreduce job and could successfully output
>> them using a FileSink. But if I use jconsole to access YARN nodemanager, I
>> can only see hadoop metrics e.g Hadoop/NodeManager/NodeManagerMetrices
>> etc.,  not mine with prefix maptask. How to setup to see maptask/reducetask
>> prefix metrics?
>> 3. Is there an example using jmx? I could not find
>>
>> The configuration syntax is:
>>
>>   [prefix].[source|sink|jmx|].[instance].[option]
>>
>>
>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html
>>
>
>

Re: Questions about hadoop-metrics2.properties

Posted by Benyi Wang <be...@gmail.com>.

Thanks, Luke. Your answer is really helpful.


On Wed, Oct 23, 2013 at 11:51 PM, Luke Lu <ll...@apache.org> wrote:

> 1. File and Ganglia are the only bundled sinks, though there are
> socket/json (for chukwa) and graphite sinks patches in the works.
> 2. Hadoop metrics (and metrics2) is mostly designed for system/process
> metrics, which means you'll need to attach jconsole to your map/reduce task
> processes to see your task metrics instrumented via metrics. What you
> actually want is probably custom job counters.
> 3. You don't need any configuration to use JMX to access metrics2, as JMX
> is currently on by default. The configuration in hadoop-metrics2.properties
> is mostly for optional sink configuration and metrics filtering.
>
> __Luke
>
>
>
> On Wed, Oct 23, 2013 at 4:21 PM, Benyi Wang <be...@gmail.com> wrote:
>
>> 1. Does hadoop metrics2 only support File and Ganglia sink?
>> 2. Can I expose metrics as JMX, especially for customized metrics? I
>> created some  metrics in my mapreduce job and could successfully output
>> them using a FileSink. But if I use jconsole to access YARN nodemanager, I
>> can only see hadoop metrics e.g Hadoop/NodeManager/NodeManagerMetrices
>> etc.,  not mine with prefix maptask. How to setup to see maptask/reducetask
>> prefix metrics?
>> 3. Is there an example using jmx? I could not find
>>
>> The configuration syntax is:
>>
>>   [prefix].[source|sink|jmx|].[instance].[option]
>>
>>
>> http://hadoop.apache.org/docs/current/api/org/apache/hadoop/metrics2/package-summary.html
>>
>
>