You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Jason Venner <ja...@attributor.com> on 2008/03/06 23:03:28 UTC
Question about using the metrics framework.
We have started our first attempt at this, and do not see the metrics
being reported.
Our first cut simply is trying to report the counters at the end of the job.
A theory is that the job is exiting before the metrics are flushed.
This code is in the driver for our map/reduce task, and is called after
the JobClient.runJob() method returns.
private ContextFactory contextFactory = null;
private MetricsContext myContext = null;
private MetricsRecord randomizeRecord = null;
{
try {
contextFactory = ContextFactory.getFactory();
myContext = contextFactory.getContext("myContext");
randomizeRecord = myContext.createRecord( "MyContextName" );
} catch( Exception e ) {
LOG.error( "Unable to initialize metrics context factory", e );
}
}
void reportCounterMetrics( Counters counters ) {
for( String groupName : counters.getGroupNames() ) {
Group group = counters.getGroup( groupName );
for( String counterName : group.getCounterNames() ) {
randomizeRecord.setMetric( counterName,
group.getCounter(counterName) );
LOG.info( "reporting counter " + counterName + " " +
group.getCounter(counterName) );
}
}
randomizeRecord.update();
}
Re: Question about using the metrics framework.
Posted by Chris K Wensel <ch...@wensel.net>.
never mind on the jvm. just found the typo.. <frown>
On Mar 6, 2008, at 3:19 PM, Chris K Wensel wrote:
> actually, I don't see any jvm metrics across the cluster.
>
> any idea how to get a local gmond to gmetad to report local
> statistics? it is also accumulating slave stats just fine (minus jvm).
>
> On Mar 6, 2008, at 3:09 PM, Jason Venner wrote:
>
>> We have tweeked our metrics properties to cause all nodes to report
>> to the master node, and we see the map/reduce metrics also see the
>> jvm metrics from all nodes.
>>
>> The other thing is that this only runs on the master node, as it is
>> only run from the driver which currently runs on the master node.
>>
>> As I look deeper, I believe the metrics are not being flushed when
>> the context is shut down, and the flush methods are not implemented
>> for the ganglia context.
>>
>>
>> Chris K Wensel wrote:
>>> I have ganglia up on my cluster, and I definitely see some metrics
>>> from the map/reduce tasks. But I don't see anything from the JVM
>>> context for ganglia except from the master node.
>>>
>>> but i have gmetad running on the master and reporting from a local
>>> gmond seems flakey.
>>>
>>> On Mar 6, 2008, at 2:03 PM, Jason Venner wrote:
>>>
>>>> We have started our first attempt at this, and do not see the
>>>> metrics being reported.
>>>>
>>>> Our first cut simply is trying to report the counters at the end
>>>> of the job.
>>>>
>>>> A theory is that the job is exiting before the metrics are flushed.
>>>>
>>>> This code is in the driver for our map/reduce task, and is called
>>>> after the JobClient.runJob() method returns.
>>>>
>>>> private ContextFactory contextFactory = null;
>>>> private MetricsContext myContext = null;
>>>> private MetricsRecord randomizeRecord = null;
>>>> {
>>>> try {
>>>> contextFactory = ContextFactory.getFactory();
>>>> myContext = contextFactory.getContext("myContext");
>>>> randomizeRecord =
>>>> myContext.createRecord( "MyContextName" );
>>>> } catch( Exception e ) {
>>>> LOG.error( "Unable to initialize metrics context
>>>> factory", e );
>>>> }
>>>> }
>>>>
>>>> void reportCounterMetrics( Counters counters ) {
>>>> for( String groupName : counters.getGroupNames() ) {
>>>> Group group = counters.getGroup( groupName );
>>>> for( String counterName : group.getCounterNames() ) {
>>>> randomizeRecord.setMetric( counterName,
>>>> group.getCounter(counterName) );
>>>> LOG.info( "reporting counter " + counterName + " " +
>>>> group.getCounter(counterName) );
>>>> }
>>>> }
>>>> randomizeRecord.update();
>>>> }
>>>>
>>>
>>> Chris K Wensel
>>> chris@wensel.net
>>> http://chris.wensel.net/
>>>
>>>
>>>
>
> Chris K Wensel
> chris@wensel.net
> http://chris.wensel.net/
>
>
>
Chris K Wensel
chris@wensel.net
http://chris.wensel.net/
Re: Question about using the metrics framework.
Posted by Chris K Wensel <ch...@wensel.net>.
actually, I don't see any jvm metrics across the cluster.
any idea how to get a local gmond to gmetad to report local
statistics? it is also accumulating slave stats just fine (minus jvm).
On Mar 6, 2008, at 3:09 PM, Jason Venner wrote:
> We have tweeked our metrics properties to cause all nodes to report
> to the master node, and we see the map/reduce metrics also see the
> jvm metrics from all nodes.
>
> The other thing is that this only runs on the master node, as it is
> only run from the driver which currently runs on the master node.
>
> As I look deeper, I believe the metrics are not being flushed when
> the context is shut down, and the flush methods are not implemented
> for the ganglia context.
>
>
> Chris K Wensel wrote:
>> I have ganglia up on my cluster, and I definitely see some metrics
>> from the map/reduce tasks. But I don't see anything from the JVM
>> context for ganglia except from the master node.
>>
>> but i have gmetad running on the master and reporting from a local
>> gmond seems flakey.
>>
>> On Mar 6, 2008, at 2:03 PM, Jason Venner wrote:
>>
>>> We have started our first attempt at this, and do not see the
>>> metrics being reported.
>>>
>>> Our first cut simply is trying to report the counters at the end
>>> of the job.
>>>
>>> A theory is that the job is exiting before the metrics are flushed.
>>>
>>> This code is in the driver for our map/reduce task, and is called
>>> after the JobClient.runJob() method returns.
>>>
>>> private ContextFactory contextFactory = null;
>>> private MetricsContext myContext = null;
>>> private MetricsRecord randomizeRecord = null;
>>> {
>>> try {
>>> contextFactory = ContextFactory.getFactory();
>>> myContext = contextFactory.getContext("myContext");
>>> randomizeRecord =
>>> myContext.createRecord( "MyContextName" );
>>> } catch( Exception e ) {
>>> LOG.error( "Unable to initialize metrics context
>>> factory", e );
>>> }
>>> }
>>>
>>> void reportCounterMetrics( Counters counters ) {
>>> for( String groupName : counters.getGroupNames() ) {
>>> Group group = counters.getGroup( groupName );
>>> for( String counterName : group.getCounterNames() ) {
>>> randomizeRecord.setMetric( counterName,
>>> group.getCounter(counterName) );
>>> LOG.info( "reporting counter " + counterName + " " +
>>> group.getCounter(counterName) );
>>> }
>>> }
>>> randomizeRecord.update();
>>> }
>>>
>>
>> Chris K Wensel
>> chris@wensel.net
>> http://chris.wensel.net/
>>
>>
>>
Chris K Wensel
chris@wensel.net
http://chris.wensel.net/
Re: Question about using the metrics framework.
Posted by Jason Venner <ja...@attributor.com>.
We have tweeked our metrics properties to cause all nodes to report to
the master node, and we see the map/reduce metrics also see the jvm
metrics from all nodes.
The other thing is that this only runs on the master node, as it is only
run from the driver which currently runs on the master node.
As I look deeper, I believe the metrics are not being flushed when the
context is shut down, and the flush methods are not implemented for the
ganglia context.
Chris K Wensel wrote:
> I have ganglia up on my cluster, and I definitely see some metrics
> from the map/reduce tasks. But I don't see anything from the JVM
> context for ganglia except from the master node.
>
> but i have gmetad running on the master and reporting from a local
> gmond seems flakey.
>
> On Mar 6, 2008, at 2:03 PM, Jason Venner wrote:
>
>> We have started our first attempt at this, and do not see the metrics
>> being reported.
>>
>> Our first cut simply is trying to report the counters at the end of
>> the job.
>>
>> A theory is that the job is exiting before the metrics are flushed.
>>
>> This code is in the driver for our map/reduce task, and is called
>> after the JobClient.runJob() method returns.
>>
>> private ContextFactory contextFactory = null;
>> private MetricsContext myContext = null;
>> private MetricsRecord randomizeRecord = null;
>> {
>> try {
>> contextFactory = ContextFactory.getFactory();
>> myContext = contextFactory.getContext("myContext");
>> randomizeRecord = myContext.createRecord( "MyContextName" );
>> } catch( Exception e ) {
>> LOG.error( "Unable to initialize metrics context factory",
>> e );
>> }
>> }
>>
>> void reportCounterMetrics( Counters counters ) {
>> for( String groupName : counters.getGroupNames() ) {
>> Group group = counters.getGroup( groupName );
>> for( String counterName : group.getCounterNames() ) {
>> randomizeRecord.setMetric( counterName,
>> group.getCounter(counterName) );
>> LOG.info( "reporting counter " + counterName + " " +
>> group.getCounter(counterName) );
>> }
>> }
>> randomizeRecord.update();
>> }
>>
>
> Chris K Wensel
> chris@wensel.net
> http://chris.wensel.net/
>
>
>
Re: Question about using the metrics framework.
Posted by Chris K Wensel <ch...@wensel.net>.
I have ganglia up on my cluster, and I definitely see some metrics
from the map/reduce tasks. But I don't see anything from the JVM
context for ganglia except from the master node.
but i have gmetad running on the master and reporting from a local
gmond seems flakey.
On Mar 6, 2008, at 2:03 PM, Jason Venner wrote:
> We have started our first attempt at this, and do not see the
> metrics being reported.
>
> Our first cut simply is trying to report the counters at the end of
> the job.
>
> A theory is that the job is exiting before the metrics are flushed.
>
> This code is in the driver for our map/reduce task, and is called
> after the JobClient.runJob() method returns.
>
> private ContextFactory contextFactory = null;
> private MetricsContext myContext = null;
> private MetricsRecord randomizeRecord = null;
> {
> try {
> contextFactory = ContextFactory.getFactory();
> myContext = contextFactory.getContext("myContext");
> randomizeRecord = myContext.createRecord( "MyContextName" );
> } catch( Exception e ) {
> LOG.error( "Unable to initialize metrics context factory",
> e );
> }
> }
>
> void reportCounterMetrics( Counters counters ) {
> for( String groupName : counters.getGroupNames() ) {
> Group group = counters.getGroup( groupName );
> for( String counterName : group.getCounterNames() ) {
> randomizeRecord.setMetric( counterName,
> group.getCounter(counterName) );
> LOG.info( "reporting counter " + counterName + " " +
> group.getCounter(counterName) );
> }
> }
> randomizeRecord.update();
> }
>
Chris K Wensel
chris@wensel.net
http://chris.wensel.net/