You are viewing a plain text version of this content. The canonical link for it is here.
Posted to general@hadoop.apache.org by Michael Segel <mi...@hotmail.com> on 2010/07/19 20:22:02 UTC
Dynamic counters in Hadoop M/R jobs...
Hi,
In looking at "The Definitive Guide" pgs 211-218 looking at the documentation for counters have a question about how to use the
dynamic (String, String) methods to create/access counters.
Looking at the IdentityTableMap class, the following map() method exists():
public void map(ImmutableBytesWritable key,
RowResult value,
org.apache.hadoop.mapred.OutputCollector<ImmutableBytesWritable,RowResult> output,
org.apache.hadoop.mapred.Reporter reporter)
throws IOExceptionThis method is deprecated in 20.5
So how can you get the Reporter so you can use the 'reporter.incrCounter(String group,String counter, long amount)'
method. (See pg 214 code fragment)
I must be missing something.
Thx
-Mike
_________________________________________________________________
The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail.
http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4
RE: Dynamic counters in Hadoop M/R jobs...
Posted by "Segel, Mike" <ms...@navteq.com>.
Thanks for the reply.
Got a couple of questions...
I saw that I could get the StatusReporter from the Mapper.Context.
But the API doesn't have a way to get the StatusReport from the Mapper.Context.
If you're going to go to the Mapper.Context you can get the counters.
And you have a getCounter(String, String). Then you can always increment it.
But this kind of begs a question which isn't documented...
If I have a mapper.Context and I say getCounter("Foo","Foobar Count");
What happens if there isn't a counter ("Foo","Foobar Count") ?
What I mean is how do you add a dynamic counter or if I ask for a counter that doesn't already exist, will it create it?
(This isn't documented... and its not safe to assume anything.)
Sorry, but dynamic counters isn't really well documented.
-Mike
-----Original Message-----
From: anthony.urso@gmail.com [mailto:anthony.urso@gmail.com] On Behalf Of Anthony Urso
Sent: Monday, July 19, 2010 7:11 PM
To: general@hadoop.apache.org
Subject: Re: Dynamic counters in Hadoop M/R jobs...
The new Mapper class passes a Map.Context object to the map() method.
From this you can get a StatusReporter object which can produce a
named Counter object and increment it.
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Mapper.Context.html
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/StatusReporter.html
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Counter.html
Alternatively, the old APIs are supposed to be temporarily
de-deprecated in the next stable release, so you can use them as
described by that book.
On Mon, Jul 19, 2010 at 11:22 AM, Michael Segel
<mi...@hotmail.com> wrote:
>
> Hi,
>
> In looking at "The Definitive Guide" pgs 211-218 looking at the documentation for counters have a question about how to use the
>
> dynamic (String, String) methods to create/access counters.
>
> Looking at the IdentityTableMap class, the following map() method exists():
> public void map(ImmutableBytesWritable key,
> RowResult value,
> org.apache.hadoop.mapred.OutputCollector<ImmutableBytesWritable,RowResult> output,
> org.apache.hadoop.mapred.Reporter reporter)
> throws IOExceptionThis method is deprecated in 20.5
>
> So how can you get the Reporter so you can use the 'reporter.incrCounter(String group,String counter, long amount)'
> method. (See pg 214 code fragment)
>
> I must be missing something.
>
> Thx
>
> -Mike
>
>
>
> _________________________________________________________________
> The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail.
> http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4
The information contained in this communication may be CONFIDENTIAL and is intended only for the use of the recipient(s) named above. If you are not the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please notify the sender and delete/destroy the original message and any copy of it from your computer or paper files.
Re: Dynamic counters in Hadoop M/R jobs...
Posted by Anthony Urso <an...@cs.ucla.edu>.
The new Mapper class passes a Map.Context object to the map() method.
>From this you can get a StatusReporter object which can produce a
named Counter object and increment it.
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Mapper.Context.html
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/StatusReporter.html
http://hadoop.apache.org/common/docs/current/api/org/apache/hadoop/mapreduce/Counter.html
Alternatively, the old APIs are supposed to be temporarily
de-deprecated in the next stable release, so you can use them as
described by that book.
On Mon, Jul 19, 2010 at 11:22 AM, Michael Segel
<mi...@hotmail.com> wrote:
>
> Hi,
>
> In looking at "The Definitive Guide" pgs 211-218 looking at the documentation for counters have a question about how to use the
>
> dynamic (String, String) methods to create/access counters.
>
> Looking at the IdentityTableMap class, the following map() method exists():
> public void map(ImmutableBytesWritable key,
> RowResult value,
> org.apache.hadoop.mapred.OutputCollector<ImmutableBytesWritable,RowResult> output,
> org.apache.hadoop.mapred.Reporter reporter)
> throws IOExceptionThis method is deprecated in 20.5
>
> So how can you get the Reporter so you can use the 'reporter.incrCounter(String group,String counter, long amount)'
> method. (See pg 214 code fragment)
>
> I must be missing something.
>
> Thx
>
> -Mike
>
>
>
> _________________________________________________________________
> The New Busy is not the too busy. Combine all your e-mail accounts with Hotmail.
> http://www.windowslive.com/campaign/thenewbusy?tile=multiaccount&ocid=PID28326::T:WLMTAGL:ON:WL:en-US:WM_HMP:042010_4