How can I use counters in Hadoop

[CloudyEye <su...@gmail.com>]

Hi, I am new newbie to Hadoop. I would be thankful if you help me.

I've read that I can use the Reporter class to increase counters. this way:
reporter.incrCounter(Enum args, long arg1);

How can I get the values of those counters ?

My aim is to count the total inputs to the mappers , then i want to override
the:
public void configure(JobConf job) {} method of the reducer to get the
counter value before reducing any key,values.

Regards,
-- 
View this message in context: http://www.nabble.com/How-can-I-use-counters-in-Hadoop-tp16702607p16702607.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: How can I use counters in Hadoop

[CloudyEye <su...@gmail.com>]

Thank you Mr. Udatny. Your code was helpful.

Cheers,

-- 
View this message in context: http://www.nabble.com/How-can-I-use-counters-in-Hadoop-tp16702607p16722296.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: How can I use counters in Hadoop

[ru...@rosa.com]

i guess you cannot access the counter values from within your reducer.

as i know, the counter values could be accessed outside the mapred job, 
where you run it:

                 JobClient jb = new JobClient(jobConf); 
                 RunningJob rj = jb.submitJob(jobConf);

                 while (!rj.isComplete()) { 
                         Thread.sleep(2000); 
                         LOG.debug("job running");
                         } 

                 counters =  rj.getCounters();
                 counters.log(LOG);

maybe somebody knows better.





CloudyEye <su...@gmail.com> 
04/16/2008 10:52 AM
Please respond to
core-user@hadoop.apache.org


To
core-user@hadoop.apache.org
cc

Subject
Re: How can I use counters in Hadoop







Thank you. I tried a while with some code but I did not succeed in getting
the value of counters. Please check this code:

public class MyMapper extends MapReduceBase implements Mapper {
                 private static enum myCounters {NUMBER_OF_ROWS}

                 public void map(WritableComparable key, Writable values,
                                                 OutputCollector output, 
Reporter reporter) throws IOException {

 reporter.incrCounter(myCounters.NUMBER_OF_ROWS, 1);  
                 }
}

And here is my simple reducer:
public class MyReduce extends MapReduceBase implements Reducer {
                 private static enum myCounters {NUMBER_OF_ROWS}
                 private static long numOfRows;
 
                 public void configure(JobConf job) {
                                 super.configure(job);
 
                                 this.numOfRows= ????
 
                 }
}

How can I assign the value of counter: NUMBER_OF_ROWS to the long variable
numOfRows  ??

Best regards,




stack-3 wrote:
> 
> https://issues.apache.org/jira/browse/HBASE-559 has an example. Ignore 
> the HBase stuff.  Whats important is the ENUM at head of the MR job 
> class, the calls to Reporter inside in tasks, and the properties file -- 

> both how its named and that it ends up in the generated job jar.
> St.Ack
> 
> 

-- 
View this message in context: 
http://www.nabble.com/How-can-I-use-counters-in-Hadoop-tp16702607p16719201.html

Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: How can I use counters in Hadoop

[CloudyEye <su...@gmail.com>]

Thank you. I tried a while with some code but I did not succeed in getting
the value of counters. Please check this code:

public class MyMapper extends MapReduceBase implements Mapper {
	private static enum myCounters {NUMBER_OF_ROWS}

	public void map(WritableComparable key, Writable values,
			OutputCollector output, Reporter reporter) throws IOException {

		reporter.incrCounter(myCounters.NUMBER_OF_ROWS, 1);		
	}
}

And here is my simple reducer:
public class MyReduce extends MapReduceBase implements Reducer {
	private static enum myCounters {NUMBER_OF_ROWS}
	private static long numOfRows;
	
	public void configure(JobConf job) {
		super.configure(job);
		
		this.numOfRows= ????
		
	}
}

How can I assign the value of counter: NUMBER_OF_ROWS to the long variable
numOfRows  ??

Best regards,




stack-3 wrote:
> 
> https://issues.apache.org/jira/browse/HBASE-559 has an example. Ignore 
> the HBase stuff.  Whats important is the ENUM at head of the MR job 
> class, the calls to Reporter inside in tasks, and the properties file -- 
> both how its named and that it ends up in the generated job jar.
> St.Ack
> 
> 

-- 
View this message in context: http://www.nabble.com/How-can-I-use-counters-in-Hadoop-tp16702607p16719201.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

Re: How can I use counters in Hadoop

[stack <st...@duboce.net>]

https://issues.apache.org/jira/browse/HBASE-559 has an example. Ignore 
the HBase stuff.  Whats important is the ENUM at head of the MR job 
class, the calls to Reporter inside in tasks, and the properties file -- 
both how its named and that it ends up in the generated job jar.
St.Ack


CloudyEye wrote:
> Hi, I am new newbie to Hadoop. I would be thankful if you help me.
>
> I've read that I can use the Reporter class to increase counters. this way:
> reporter.incrCounter(Enum args, long arg1);
>
> How can I get the values of those counters ?
>
> My aim is to count the total inputs to the mappers , then i want to override
> the:
> public void configure(JobConf job) {} method of the reducer to get the
> counter value before reducing any key,values.
>
> Regards,
>