You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by some speed <sp...@gmail.com> on 2009/02/05 11:09:45 UTC
Problem with Counters
Hi,
Can someone help me with the usage of counters please? I am incrementing a
counter in Reduce method but I am unable to collect the counter value after
the job is completed.
Its something like this:
public static class Reduce extends MapReduceBase implements Reducer<Text,
FloatWritable, Text, FloatWritable>
{
static enum MyCounter{ct_key1};
public void reduce(..............) throws IOException
{
reporter.incrCounter(MyCounter.ct_key1, 1);
output.collect(......);
}
}
-----main method----
{
RunningJob running = null;
running=JobClient.runJob(conf);
Counters ct = running.getCounters();
/********* How do I Collect the ct_key1 value ????***********/
long res = ct.getCounter(MyCounter.ct_key1);
}
Thanks,
Sharath
Re: Problem with Counters
Posted by some speed <sp...@gmail.com>.
Thank you all so much.
That works. I made a stupid mistake with the naming of a local variable. so
the error. :(
On Thu, Feb 5, 2009 at 9:49 AM, Tom White <to...@cloudera.com> wrote:
> Try moving the enum to inside the top level class (as you already did)
> and then use getCounter() passing the enum value:
>
> public class MyJob {
>
> static enum MyCounter{ct_key1};
>
> // Mapper and Reducer defined here
>
> public static void main(String[] args) throws IOException {
> // ...
> RunningJob running =JobClient.runJob(conf);
> Counters ct = running.getCounters();
> long res = ct.getCounter(MyCounter.ct_key1);
> // ...
> }
>
> }
>
> BTW org.apache.hadoop.mapred.Task$Counter is a built-in MapReduce
> counter, so that won't help you retrieve your custom counter.
>
> Cheers,
>
> Tom
>
> On Thu, Feb 5, 2009 at 2:22 PM, Rasit OZDAS <ra...@gmail.com> wrote:
> > Sharath,
> >
> > You're using reporter.incrCounter(enumVal, intVal); to increment
> counter,
> > I think method to get should also be similar.
> >
> > Try to use findCounter(enumVal).getCounter() or getCounter(enumVal).
> >
> > Hope this helps,
> > Rasit
> >
> > 2009/2/5 some speed <sp...@gmail.com>:
> >> In fact I put the enum in my Reduce method as the following link (from
> >> Yahoo) says so:
> >>
> >>
> http://public.yahoo.com/gogate/hadoop-tutorial/html/module5.html#metrics
> >> --->Look at the section under Reporting Custom Metrics.
> >>
> >> 2009/2/5 some speed <sp...@gmail.com>
> >>
> >>> Thanks Rasit.
> >>>
> >>> I did as you said.
> >>>
> >>> 1) Put the static enum MyCounter{ct_key1} just above main()
> >>>
> >>> 2) Changed result =
> >>> ct.findCounter("org.apache.hadoop.mapred.Task$Counter", 1,
> >>> "Reduce.MyCounter").getCounter();
> >>>
> >>> Still is doesnt seem to help. It throws a null pointer exception.Its
> not
> >>> able to find the Counter.
> >>>
> >>>
> >>>
> >>> Thanks,
> >>>
> >>> Sharath
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Feb 5, 2009 at 8:04 AM, Rasit OZDAS <ra...@gmail.com>
> wrote:
> >>>
> >>>> Forgot to say, value "0" means that the requested counter does not
> exist.
> >>>>
> >>>> 2009/2/5 Rasit OZDAS <ra...@gmail.com>:
> >>>> > Sharath,
> >>>> > I think the static enum definition should be out of Reduce class.
> >>>> > Hadoop probably tries to find it elsewhere with "MyCounter", but
> it's
> >>>> > actually "Reduce.MyCounter" in your example.
> >>>> >
> >>>> > Hope this helps,
> >>>> > Rasit
> >>>> >
> >>>> > 2009/2/5 some speed <sp...@gmail.com>:
> >>>> >> I Tried the following...It gets compiled but the value of result
> seems
> >>>> to be
> >>>> >> 0 always.
> >>>> >>
> >>>> >> RunningJob running = JobClient.runJob(conf);
> >>>> >>
> >>>> >> Counters ct = new Counters();
> >>>> >> ct = running.getCounters();
> >>>> >>
> >>>> >> long result =
> >>>> >> ct.findCounter("org.apache.hadoop.mapred.Task$Counter", 0,
> >>>> >> "*MyCounter*").getCounter();
> >>>> >> //even tried MyCounter.Key1
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> Does anyone know whay that is happening?
> >>>> >>
> >>>> >> Thanks,
> >>>> >>
> >>>> >> Sharath
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> On Thu, Feb 5, 2009 at 5:59 AM, some speed <sp...@gmail.com>
> >>>> wrote:
> >>>> >>
> >>>> >>> Hi Tom,
> >>>> >>>
> >>>> >>> I get the error :
> >>>> >>>
> >>>> >>> Cannot find Symbol* "**MyCounter.ct_key1 " *
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>>
> >>>> >>> On Thu, Feb 5, 2009 at 5:51 AM, Tom White <to...@cloudera.com>
> wrote:
> >>>> >>>
> >>>> >>>> Hi Sharath,
> >>>> >>>>
> >>>> >>>> The code you posted looks right to me. Counters#getCounter() will
> >>>> >>>> return the counter's value. What error are you getting?
> >>>> >>>>
> >>>> >>>> Tom
> >>>> >>>>
> >>>> >>>> On Thu, Feb 5, 2009 at 10:09 AM, some speed <
> speed.some@gmail.com>
> >>>> wrote:
> >>>> >>>> > Hi,
> >>>> >>>> >
> >>>> >>>> > Can someone help me with the usage of counters please? I am
> >>>> incrementing
> >>>> >>>> a
> >>>> >>>> > counter in Reduce method but I am unable to collect the counter
> >>>> value
> >>>> >>>> after
> >>>> >>>> > the job is completed.
> >>>> >>>> >
> >>>> >>>> > Its something like this:
> >>>> >>>> >
> >>>> >>>> > public static class Reduce extends MapReduceBase implements
> >>>> >>>> Reducer<Text,
> >>>> >>>> > FloatWritable, Text, FloatWritable>
> >>>> >>>> > {
> >>>> >>>> > static enum MyCounter{ct_key1};
> >>>> >>>> >
> >>>> >>>> > public void reduce(..............) throws IOException
> >>>> >>>> > {
> >>>> >>>> >
> >>>> >>>> > reporter.incrCounter(MyCounter.ct_key1, 1);
> >>>> >>>> >
> >>>> >>>> > output.collect(......);
> >>>> >>>> >
> >>>> >>>> > }
> >>>> >>>> > }
> >>>> >>>> >
> >>>> >>>> > -----main method----
> >>>> >>>> > {
> >>>> >>>> > RunningJob running = null;
> >>>> >>>> > running=JobClient.runJob(conf);
> >>>> >>>> >
> >>>> >>>> > Counters ct = running.getCounters();
> >>>> >>>> > /********* How do I Collect the ct_key1 value ????***********/
> >>>> >>>> > long res = ct.getCounter(MyCounter.ct_key1);
> >>>> >>>> >
> >>>> >>>> > }
> >>>> >>>> >
> >>>> >>>> >
> >>>> >>>> >
> >>>> >>>> >
> >>>> >>>> >
> >>>> >>>> > Thanks,
> >>>> >>>> >
> >>>> >>>> > Sharath
> >>>> >>>> >
> >>>> >>>>
> >>>> >>>
> >>>> >>>
> >>>> >>
> >>>> >
> >>>> >
> >>>> >
> >>>> > --
> >>>> > M. Raşit ÖZDAŞ
> >>>> >
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> M. Raşit ÖZDAŞ
> >>>>
> >>>
> >>>
> >>
> >
> >
> >
> > --
> > M. Raşit ÖZDAŞ
> >
>
Re: Problem with Counters
Posted by Tom White <to...@cloudera.com>.
Try moving the enum to inside the top level class (as you already did)
and then use getCounter() passing the enum value:
public class MyJob {
static enum MyCounter{ct_key1};
// Mapper and Reducer defined here
public static void main(String[] args) throws IOException {
// ...
RunningJob running =JobClient.runJob(conf);
Counters ct = running.getCounters();
long res = ct.getCounter(MyCounter.ct_key1);
// ...
}
}
BTW org.apache.hadoop.mapred.Task$Counter is a built-in MapReduce
counter, so that won't help you retrieve your custom counter.
Cheers,
Tom
On Thu, Feb 5, 2009 at 2:22 PM, Rasit OZDAS <ra...@gmail.com> wrote:
> Sharath,
>
> You're using reporter.incrCounter(enumVal, intVal); to increment counter,
> I think method to get should also be similar.
>
> Try to use findCounter(enumVal).getCounter() or getCounter(enumVal).
>
> Hope this helps,
> Rasit
>
> 2009/2/5 some speed <sp...@gmail.com>:
>> In fact I put the enum in my Reduce method as the following link (from
>> Yahoo) says so:
>>
>> http://public.yahoo.com/gogate/hadoop-tutorial/html/module5.html#metrics
>> --->Look at the section under Reporting Custom Metrics.
>>
>> 2009/2/5 some speed <sp...@gmail.com>
>>
>>> Thanks Rasit.
>>>
>>> I did as you said.
>>>
>>> 1) Put the static enum MyCounter{ct_key1} just above main()
>>>
>>> 2) Changed result =
>>> ct.findCounter("org.apache.hadoop.mapred.Task$Counter", 1,
>>> "Reduce.MyCounter").getCounter();
>>>
>>> Still is doesnt seem to help. It throws a null pointer exception.Its not
>>> able to find the Counter.
>>>
>>>
>>>
>>> Thanks,
>>>
>>> Sharath
>>>
>>>
>>>
>>>
>>> On Thu, Feb 5, 2009 at 8:04 AM, Rasit OZDAS <ra...@gmail.com> wrote:
>>>
>>>> Forgot to say, value "0" means that the requested counter does not exist.
>>>>
>>>> 2009/2/5 Rasit OZDAS <ra...@gmail.com>:
>>>> > Sharath,
>>>> > I think the static enum definition should be out of Reduce class.
>>>> > Hadoop probably tries to find it elsewhere with "MyCounter", but it's
>>>> > actually "Reduce.MyCounter" in your example.
>>>> >
>>>> > Hope this helps,
>>>> > Rasit
>>>> >
>>>> > 2009/2/5 some speed <sp...@gmail.com>:
>>>> >> I Tried the following...It gets compiled but the value of result seems
>>>> to be
>>>> >> 0 always.
>>>> >>
>>>> >> RunningJob running = JobClient.runJob(conf);
>>>> >>
>>>> >> Counters ct = new Counters();
>>>> >> ct = running.getCounters();
>>>> >>
>>>> >> long result =
>>>> >> ct.findCounter("org.apache.hadoop.mapred.Task$Counter", 0,
>>>> >> "*MyCounter*").getCounter();
>>>> >> //even tried MyCounter.Key1
>>>> >>
>>>> >>
>>>> >>
>>>> >> Does anyone know whay that is happening?
>>>> >>
>>>> >> Thanks,
>>>> >>
>>>> >> Sharath
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Thu, Feb 5, 2009 at 5:59 AM, some speed <sp...@gmail.com>
>>>> wrote:
>>>> >>
>>>> >>> Hi Tom,
>>>> >>>
>>>> >>> I get the error :
>>>> >>>
>>>> >>> Cannot find Symbol* "**MyCounter.ct_key1 " *
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> On Thu, Feb 5, 2009 at 5:51 AM, Tom White <to...@cloudera.com> wrote:
>>>> >>>
>>>> >>>> Hi Sharath,
>>>> >>>>
>>>> >>>> The code you posted looks right to me. Counters#getCounter() will
>>>> >>>> return the counter's value. What error are you getting?
>>>> >>>>
>>>> >>>> Tom
>>>> >>>>
>>>> >>>> On Thu, Feb 5, 2009 at 10:09 AM, some speed <sp...@gmail.com>
>>>> wrote:
>>>> >>>> > Hi,
>>>> >>>> >
>>>> >>>> > Can someone help me with the usage of counters please? I am
>>>> incrementing
>>>> >>>> a
>>>> >>>> > counter in Reduce method but I am unable to collect the counter
>>>> value
>>>> >>>> after
>>>> >>>> > the job is completed.
>>>> >>>> >
>>>> >>>> > Its something like this:
>>>> >>>> >
>>>> >>>> > public static class Reduce extends MapReduceBase implements
>>>> >>>> Reducer<Text,
>>>> >>>> > FloatWritable, Text, FloatWritable>
>>>> >>>> > {
>>>> >>>> > static enum MyCounter{ct_key1};
>>>> >>>> >
>>>> >>>> > public void reduce(..............) throws IOException
>>>> >>>> > {
>>>> >>>> >
>>>> >>>> > reporter.incrCounter(MyCounter.ct_key1, 1);
>>>> >>>> >
>>>> >>>> > output.collect(......);
>>>> >>>> >
>>>> >>>> > }
>>>> >>>> > }
>>>> >>>> >
>>>> >>>> > -----main method----
>>>> >>>> > {
>>>> >>>> > RunningJob running = null;
>>>> >>>> > running=JobClient.runJob(conf);
>>>> >>>> >
>>>> >>>> > Counters ct = running.getCounters();
>>>> >>>> > /********* How do I Collect the ct_key1 value ????***********/
>>>> >>>> > long res = ct.getCounter(MyCounter.ct_key1);
>>>> >>>> >
>>>> >>>> > }
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> >
>>>> >>>> > Thanks,
>>>> >>>> >
>>>> >>>> > Sharath
>>>> >>>> >
>>>> >>>>
>>>> >>>
>>>> >>>
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > M. Raşit ÖZDAŞ
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> M. Raşit ÖZDAŞ
>>>>
>>>
>>>
>>
>
>
>
> --
> M. Raşit ÖZDAŞ
>
Re: Problem with Counters
Posted by Rasit OZDAS <ra...@gmail.com>.
Sharath,
You're using reporter.incrCounter(enumVal, intVal); to increment counter,
I think method to get should also be similar.
Try to use findCounter(enumVal).getCounter() or getCounter(enumVal).
Hope this helps,
Rasit
2009/2/5 some speed <sp...@gmail.com>:
> In fact I put the enum in my Reduce method as the following link (from
> Yahoo) says so:
>
> http://public.yahoo.com/gogate/hadoop-tutorial/html/module5.html#metrics
> --->Look at the section under Reporting Custom Metrics.
>
> 2009/2/5 some speed <sp...@gmail.com>
>
>> Thanks Rasit.
>>
>> I did as you said.
>>
>> 1) Put the static enum MyCounter{ct_key1} just above main()
>>
>> 2) Changed result =
>> ct.findCounter("org.apache.hadoop.mapred.Task$Counter", 1,
>> "Reduce.MyCounter").getCounter();
>>
>> Still is doesnt seem to help. It throws a null pointer exception.Its not
>> able to find the Counter.
>>
>>
>>
>> Thanks,
>>
>> Sharath
>>
>>
>>
>>
>> On Thu, Feb 5, 2009 at 8:04 AM, Rasit OZDAS <ra...@gmail.com> wrote:
>>
>>> Forgot to say, value "0" means that the requested counter does not exist.
>>>
>>> 2009/2/5 Rasit OZDAS <ra...@gmail.com>:
>>> > Sharath,
>>> > I think the static enum definition should be out of Reduce class.
>>> > Hadoop probably tries to find it elsewhere with "MyCounter", but it's
>>> > actually "Reduce.MyCounter" in your example.
>>> >
>>> > Hope this helps,
>>> > Rasit
>>> >
>>> > 2009/2/5 some speed <sp...@gmail.com>:
>>> >> I Tried the following...It gets compiled but the value of result seems
>>> to be
>>> >> 0 always.
>>> >>
>>> >> RunningJob running = JobClient.runJob(conf);
>>> >>
>>> >> Counters ct = new Counters();
>>> >> ct = running.getCounters();
>>> >>
>>> >> long result =
>>> >> ct.findCounter("org.apache.hadoop.mapred.Task$Counter", 0,
>>> >> "*MyCounter*").getCounter();
>>> >> //even tried MyCounter.Key1
>>> >>
>>> >>
>>> >>
>>> >> Does anyone know whay that is happening?
>>> >>
>>> >> Thanks,
>>> >>
>>> >> Sharath
>>> >>
>>> >>
>>> >>
>>> >> On Thu, Feb 5, 2009 at 5:59 AM, some speed <sp...@gmail.com>
>>> wrote:
>>> >>
>>> >>> Hi Tom,
>>> >>>
>>> >>> I get the error :
>>> >>>
>>> >>> Cannot find Symbol* "**MyCounter.ct_key1 " *
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>>
>>> >>> On Thu, Feb 5, 2009 at 5:51 AM, Tom White <to...@cloudera.com> wrote:
>>> >>>
>>> >>>> Hi Sharath,
>>> >>>>
>>> >>>> The code you posted looks right to me. Counters#getCounter() will
>>> >>>> return the counter's value. What error are you getting?
>>> >>>>
>>> >>>> Tom
>>> >>>>
>>> >>>> On Thu, Feb 5, 2009 at 10:09 AM, some speed <sp...@gmail.com>
>>> wrote:
>>> >>>> > Hi,
>>> >>>> >
>>> >>>> > Can someone help me with the usage of counters please? I am
>>> incrementing
>>> >>>> a
>>> >>>> > counter in Reduce method but I am unable to collect the counter
>>> value
>>> >>>> after
>>> >>>> > the job is completed.
>>> >>>> >
>>> >>>> > Its something like this:
>>> >>>> >
>>> >>>> > public static class Reduce extends MapReduceBase implements
>>> >>>> Reducer<Text,
>>> >>>> > FloatWritable, Text, FloatWritable>
>>> >>>> > {
>>> >>>> > static enum MyCounter{ct_key1};
>>> >>>> >
>>> >>>> > public void reduce(..............) throws IOException
>>> >>>> > {
>>> >>>> >
>>> >>>> > reporter.incrCounter(MyCounter.ct_key1, 1);
>>> >>>> >
>>> >>>> > output.collect(......);
>>> >>>> >
>>> >>>> > }
>>> >>>> > }
>>> >>>> >
>>> >>>> > -----main method----
>>> >>>> > {
>>> >>>> > RunningJob running = null;
>>> >>>> > running=JobClient.runJob(conf);
>>> >>>> >
>>> >>>> > Counters ct = running.getCounters();
>>> >>>> > /********* How do I Collect the ct_key1 value ????***********/
>>> >>>> > long res = ct.getCounter(MyCounter.ct_key1);
>>> >>>> >
>>> >>>> > }
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> >
>>> >>>> > Thanks,
>>> >>>> >
>>> >>>> > Sharath
>>> >>>> >
>>> >>>>
>>> >>>
>>> >>>
>>> >>
>>> >
>>> >
>>> >
>>> > --
>>> > M. Raşit ÖZDAŞ
>>> >
>>>
>>>
>>>
>>> --
>>> M. Raşit ÖZDAŞ
>>>
>>
>>
>
--
M. Raşit ÖZDAŞ
Re: Problem with Counters
Posted by some speed <sp...@gmail.com>.
In fact I put the enum in my Reduce method as the following link (from
Yahoo) says so:
http://public.yahoo.com/gogate/hadoop-tutorial/html/module5.html#metrics
--->Look at the section under Reporting Custom Metrics.
2009/2/5 some speed <sp...@gmail.com>
> Thanks Rasit.
>
> I did as you said.
>
> 1) Put the static enum MyCounter{ct_key1} just above main()
>
> 2) Changed result =
> ct.findCounter("org.apache.hadoop.mapred.Task$Counter", 1,
> "Reduce.MyCounter").getCounter();
>
> Still is doesnt seem to help. It throws a null pointer exception.Its not
> able to find the Counter.
>
>
>
> Thanks,
>
> Sharath
>
>
>
>
> On Thu, Feb 5, 2009 at 8:04 AM, Rasit OZDAS <ra...@gmail.com> wrote:
>
>> Forgot to say, value "0" means that the requested counter does not exist.
>>
>> 2009/2/5 Rasit OZDAS <ra...@gmail.com>:
>> > Sharath,
>> > I think the static enum definition should be out of Reduce class.
>> > Hadoop probably tries to find it elsewhere with "MyCounter", but it's
>> > actually "Reduce.MyCounter" in your example.
>> >
>> > Hope this helps,
>> > Rasit
>> >
>> > 2009/2/5 some speed <sp...@gmail.com>:
>> >> I Tried the following...It gets compiled but the value of result seems
>> to be
>> >> 0 always.
>> >>
>> >> RunningJob running = JobClient.runJob(conf);
>> >>
>> >> Counters ct = new Counters();
>> >> ct = running.getCounters();
>> >>
>> >> long result =
>> >> ct.findCounter("org.apache.hadoop.mapred.Task$Counter", 0,
>> >> "*MyCounter*").getCounter();
>> >> //even tried MyCounter.Key1
>> >>
>> >>
>> >>
>> >> Does anyone know whay that is happening?
>> >>
>> >> Thanks,
>> >>
>> >> Sharath
>> >>
>> >>
>> >>
>> >> On Thu, Feb 5, 2009 at 5:59 AM, some speed <sp...@gmail.com>
>> wrote:
>> >>
>> >>> Hi Tom,
>> >>>
>> >>> I get the error :
>> >>>
>> >>> Cannot find Symbol* "**MyCounter.ct_key1 " *
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> On Thu, Feb 5, 2009 at 5:51 AM, Tom White <to...@cloudera.com> wrote:
>> >>>
>> >>>> Hi Sharath,
>> >>>>
>> >>>> The code you posted looks right to me. Counters#getCounter() will
>> >>>> return the counter's value. What error are you getting?
>> >>>>
>> >>>> Tom
>> >>>>
>> >>>> On Thu, Feb 5, 2009 at 10:09 AM, some speed <sp...@gmail.com>
>> wrote:
>> >>>> > Hi,
>> >>>> >
>> >>>> > Can someone help me with the usage of counters please? I am
>> incrementing
>> >>>> a
>> >>>> > counter in Reduce method but I am unable to collect the counter
>> value
>> >>>> after
>> >>>> > the job is completed.
>> >>>> >
>> >>>> > Its something like this:
>> >>>> >
>> >>>> > public static class Reduce extends MapReduceBase implements
>> >>>> Reducer<Text,
>> >>>> > FloatWritable, Text, FloatWritable>
>> >>>> > {
>> >>>> > static enum MyCounter{ct_key1};
>> >>>> >
>> >>>> > public void reduce(..............) throws IOException
>> >>>> > {
>> >>>> >
>> >>>> > reporter.incrCounter(MyCounter.ct_key1, 1);
>> >>>> >
>> >>>> > output.collect(......);
>> >>>> >
>> >>>> > }
>> >>>> > }
>> >>>> >
>> >>>> > -----main method----
>> >>>> > {
>> >>>> > RunningJob running = null;
>> >>>> > running=JobClient.runJob(conf);
>> >>>> >
>> >>>> > Counters ct = running.getCounters();
>> >>>> > /********* How do I Collect the ct_key1 value ????***********/
>> >>>> > long res = ct.getCounter(MyCounter.ct_key1);
>> >>>> >
>> >>>> > }
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > Thanks,
>> >>>> >
>> >>>> > Sharath
>> >>>> >
>> >>>>
>> >>>
>> >>>
>> >>
>> >
>> >
>> >
>> > --
>> > M. Raşit ÖZDAŞ
>> >
>>
>>
>>
>> --
>> M. Raşit ÖZDAŞ
>>
>
>
Re: Problem with Counters
Posted by some speed <sp...@gmail.com>.
Thanks Rasit.
I did as you said.
1) Put the static enum MyCounter{ct_key1} just above main()
2) Changed result = ct.findCounter("org.apache.hadoop.mapred.Task$Counter",
1, "Reduce.MyCounter").getCounter();
Still is doesnt seem to help. It throws a null pointer exception.Its not
able to find the Counter.
Thanks,
Sharath
On Thu, Feb 5, 2009 at 8:04 AM, Rasit OZDAS <ra...@gmail.com> wrote:
> Forgot to say, value "0" means that the requested counter does not exist.
>
> 2009/2/5 Rasit OZDAS <ra...@gmail.com>:
> > Sharath,
> > I think the static enum definition should be out of Reduce class.
> > Hadoop probably tries to find it elsewhere with "MyCounter", but it's
> > actually "Reduce.MyCounter" in your example.
> >
> > Hope this helps,
> > Rasit
> >
> > 2009/2/5 some speed <sp...@gmail.com>:
> >> I Tried the following...It gets compiled but the value of result seems
> to be
> >> 0 always.
> >>
> >> RunningJob running = JobClient.runJob(conf);
> >>
> >> Counters ct = new Counters();
> >> ct = running.getCounters();
> >>
> >> long result =
> >> ct.findCounter("org.apache.hadoop.mapred.Task$Counter", 0,
> >> "*MyCounter*").getCounter();
> >> //even tried MyCounter.Key1
> >>
> >>
> >>
> >> Does anyone know whay that is happening?
> >>
> >> Thanks,
> >>
> >> Sharath
> >>
> >>
> >>
> >> On Thu, Feb 5, 2009 at 5:59 AM, some speed <sp...@gmail.com>
> wrote:
> >>
> >>> Hi Tom,
> >>>
> >>> I get the error :
> >>>
> >>> Cannot find Symbol* "**MyCounter.ct_key1 " *
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Thu, Feb 5, 2009 at 5:51 AM, Tom White <to...@cloudera.com> wrote:
> >>>
> >>>> Hi Sharath,
> >>>>
> >>>> The code you posted looks right to me. Counters#getCounter() will
> >>>> return the counter's value. What error are you getting?
> >>>>
> >>>> Tom
> >>>>
> >>>> On Thu, Feb 5, 2009 at 10:09 AM, some speed <sp...@gmail.com>
> wrote:
> >>>> > Hi,
> >>>> >
> >>>> > Can someone help me with the usage of counters please? I am
> incrementing
> >>>> a
> >>>> > counter in Reduce method but I am unable to collect the counter
> value
> >>>> after
> >>>> > the job is completed.
> >>>> >
> >>>> > Its something like this:
> >>>> >
> >>>> > public static class Reduce extends MapReduceBase implements
> >>>> Reducer<Text,
> >>>> > FloatWritable, Text, FloatWritable>
> >>>> > {
> >>>> > static enum MyCounter{ct_key1};
> >>>> >
> >>>> > public void reduce(..............) throws IOException
> >>>> > {
> >>>> >
> >>>> > reporter.incrCounter(MyCounter.ct_key1, 1);
> >>>> >
> >>>> > output.collect(......);
> >>>> >
> >>>> > }
> >>>> > }
> >>>> >
> >>>> > -----main method----
> >>>> > {
> >>>> > RunningJob running = null;
> >>>> > running=JobClient.runJob(conf);
> >>>> >
> >>>> > Counters ct = running.getCounters();
> >>>> > /********* How do I Collect the ct_key1 value ????***********/
> >>>> > long res = ct.getCounter(MyCounter.ct_key1);
> >>>> >
> >>>> > }
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > Thanks,
> >>>> >
> >>>> > Sharath
> >>>> >
> >>>>
> >>>
> >>>
> >>
> >
> >
> >
> > --
> > M. Raşit ÖZDAŞ
> >
>
>
>
> --
> M. Raşit ÖZDAŞ
>
Re: Problem with Counters
Posted by Rasit OZDAS <ra...@gmail.com>.
Forgot to say, value "0" means that the requested counter does not exist.
2009/2/5 Rasit OZDAS <ra...@gmail.com>:
> Sharath,
> I think the static enum definition should be out of Reduce class.
> Hadoop probably tries to find it elsewhere with "MyCounter", but it's
> actually "Reduce.MyCounter" in your example.
>
> Hope this helps,
> Rasit
>
> 2009/2/5 some speed <sp...@gmail.com>:
>> I Tried the following...It gets compiled but the value of result seems to be
>> 0 always.
>>
>> RunningJob running = JobClient.runJob(conf);
>>
>> Counters ct = new Counters();
>> ct = running.getCounters();
>>
>> long result =
>> ct.findCounter("org.apache.hadoop.mapred.Task$Counter", 0,
>> "*MyCounter*").getCounter();
>> //even tried MyCounter.Key1
>>
>>
>>
>> Does anyone know whay that is happening?
>>
>> Thanks,
>>
>> Sharath
>>
>>
>>
>> On Thu, Feb 5, 2009 at 5:59 AM, some speed <sp...@gmail.com> wrote:
>>
>>> Hi Tom,
>>>
>>> I get the error :
>>>
>>> Cannot find Symbol* "**MyCounter.ct_key1 " *
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Thu, Feb 5, 2009 at 5:51 AM, Tom White <to...@cloudera.com> wrote:
>>>
>>>> Hi Sharath,
>>>>
>>>> The code you posted looks right to me. Counters#getCounter() will
>>>> return the counter's value. What error are you getting?
>>>>
>>>> Tom
>>>>
>>>> On Thu, Feb 5, 2009 at 10:09 AM, some speed <sp...@gmail.com> wrote:
>>>> > Hi,
>>>> >
>>>> > Can someone help me with the usage of counters please? I am incrementing
>>>> a
>>>> > counter in Reduce method but I am unable to collect the counter value
>>>> after
>>>> > the job is completed.
>>>> >
>>>> > Its something like this:
>>>> >
>>>> > public static class Reduce extends MapReduceBase implements
>>>> Reducer<Text,
>>>> > FloatWritable, Text, FloatWritable>
>>>> > {
>>>> > static enum MyCounter{ct_key1};
>>>> >
>>>> > public void reduce(..............) throws IOException
>>>> > {
>>>> >
>>>> > reporter.incrCounter(MyCounter.ct_key1, 1);
>>>> >
>>>> > output.collect(......);
>>>> >
>>>> > }
>>>> > }
>>>> >
>>>> > -----main method----
>>>> > {
>>>> > RunningJob running = null;
>>>> > running=JobClient.runJob(conf);
>>>> >
>>>> > Counters ct = running.getCounters();
>>>> > /********* How do I Collect the ct_key1 value ????***********/
>>>> > long res = ct.getCounter(MyCounter.ct_key1);
>>>> >
>>>> > }
>>>> >
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > Thanks,
>>>> >
>>>> > Sharath
>>>> >
>>>>
>>>
>>>
>>
>
>
>
> --
> M. Raşit ÖZDAŞ
>
--
M. Raşit ÖZDAŞ
Re: Problem with Counters
Posted by Rasit OZDAS <ra...@gmail.com>.
Sharath,
I think the static enum definition should be out of Reduce class.
Hadoop probably tries to find it elsewhere with "MyCounter", but it's
actually "Reduce.MyCounter" in your example.
Hope this helps,
Rasit
2009/2/5 some speed <sp...@gmail.com>:
> I Tried the following...It gets compiled but the value of result seems to be
> 0 always.
>
> RunningJob running = JobClient.runJob(conf);
>
> Counters ct = new Counters();
> ct = running.getCounters();
>
> long result =
> ct.findCounter("org.apache.hadoop.mapred.Task$Counter", 0,
> "*MyCounter*").getCounter();
> //even tried MyCounter.Key1
>
>
>
> Does anyone know whay that is happening?
>
> Thanks,
>
> Sharath
>
>
>
> On Thu, Feb 5, 2009 at 5:59 AM, some speed <sp...@gmail.com> wrote:
>
>> Hi Tom,
>>
>> I get the error :
>>
>> Cannot find Symbol* "**MyCounter.ct_key1 " *
>>
>>
>>
>>
>>
>>
>> On Thu, Feb 5, 2009 at 5:51 AM, Tom White <to...@cloudera.com> wrote:
>>
>>> Hi Sharath,
>>>
>>> The code you posted looks right to me. Counters#getCounter() will
>>> return the counter's value. What error are you getting?
>>>
>>> Tom
>>>
>>> On Thu, Feb 5, 2009 at 10:09 AM, some speed <sp...@gmail.com> wrote:
>>> > Hi,
>>> >
>>> > Can someone help me with the usage of counters please? I am incrementing
>>> a
>>> > counter in Reduce method but I am unable to collect the counter value
>>> after
>>> > the job is completed.
>>> >
>>> > Its something like this:
>>> >
>>> > public static class Reduce extends MapReduceBase implements
>>> Reducer<Text,
>>> > FloatWritable, Text, FloatWritable>
>>> > {
>>> > static enum MyCounter{ct_key1};
>>> >
>>> > public void reduce(..............) throws IOException
>>> > {
>>> >
>>> > reporter.incrCounter(MyCounter.ct_key1, 1);
>>> >
>>> > output.collect(......);
>>> >
>>> > }
>>> > }
>>> >
>>> > -----main method----
>>> > {
>>> > RunningJob running = null;
>>> > running=JobClient.runJob(conf);
>>> >
>>> > Counters ct = running.getCounters();
>>> > /********* How do I Collect the ct_key1 value ????***********/
>>> > long res = ct.getCounter(MyCounter.ct_key1);
>>> >
>>> > }
>>> >
>>> >
>>> >
>>> >
>>> >
>>> > Thanks,
>>> >
>>> > Sharath
>>> >
>>>
>>
>>
>
--
M. Raşit ÖZDAŞ
Re: Problem with Counters
Posted by some speed <sp...@gmail.com>.
I Tried the following...It gets compiled but the value of result seems to be
0 always.
RunningJob running = JobClient.runJob(conf);
Counters ct = new Counters();
ct = running.getCounters();
long result =
ct.findCounter("org.apache.hadoop.mapred.Task$Counter", 0,
"*MyCounter*").getCounter();
//even tried MyCounter.Key1
Does anyone know whay that is happening?
Thanks,
Sharath
On Thu, Feb 5, 2009 at 5:59 AM, some speed <sp...@gmail.com> wrote:
> Hi Tom,
>
> I get the error :
>
> Cannot find Symbol* "**MyCounter.ct_key1 " *
>
>
>
>
>
>
> On Thu, Feb 5, 2009 at 5:51 AM, Tom White <to...@cloudera.com> wrote:
>
>> Hi Sharath,
>>
>> The code you posted looks right to me. Counters#getCounter() will
>> return the counter's value. What error are you getting?
>>
>> Tom
>>
>> On Thu, Feb 5, 2009 at 10:09 AM, some speed <sp...@gmail.com> wrote:
>> > Hi,
>> >
>> > Can someone help me with the usage of counters please? I am incrementing
>> a
>> > counter in Reduce method but I am unable to collect the counter value
>> after
>> > the job is completed.
>> >
>> > Its something like this:
>> >
>> > public static class Reduce extends MapReduceBase implements
>> Reducer<Text,
>> > FloatWritable, Text, FloatWritable>
>> > {
>> > static enum MyCounter{ct_key1};
>> >
>> > public void reduce(..............) throws IOException
>> > {
>> >
>> > reporter.incrCounter(MyCounter.ct_key1, 1);
>> >
>> > output.collect(......);
>> >
>> > }
>> > }
>> >
>> > -----main method----
>> > {
>> > RunningJob running = null;
>> > running=JobClient.runJob(conf);
>> >
>> > Counters ct = running.getCounters();
>> > /********* How do I Collect the ct_key1 value ????***********/
>> > long res = ct.getCounter(MyCounter.ct_key1);
>> >
>> > }
>> >
>> >
>> >
>> >
>> >
>> > Thanks,
>> >
>> > Sharath
>> >
>>
>
>
Re: Problem with Counters
Posted by some speed <sp...@gmail.com>.
Hi Tom,
I get the error :
Cannot find Symbol* "**MyCounter.ct_key1 " *
On Thu, Feb 5, 2009 at 5:51 AM, Tom White <to...@cloudera.com> wrote:
> Hi Sharath,
>
> The code you posted looks right to me. Counters#getCounter() will
> return the counter's value. What error are you getting?
>
> Tom
>
> On Thu, Feb 5, 2009 at 10:09 AM, some speed <sp...@gmail.com> wrote:
> > Hi,
> >
> > Can someone help me with the usage of counters please? I am incrementing
> a
> > counter in Reduce method but I am unable to collect the counter value
> after
> > the job is completed.
> >
> > Its something like this:
> >
> > public static class Reduce extends MapReduceBase implements Reducer<Text,
> > FloatWritable, Text, FloatWritable>
> > {
> > static enum MyCounter{ct_key1};
> >
> > public void reduce(..............) throws IOException
> > {
> >
> > reporter.incrCounter(MyCounter.ct_key1, 1);
> >
> > output.collect(......);
> >
> > }
> > }
> >
> > -----main method----
> > {
> > RunningJob running = null;
> > running=JobClient.runJob(conf);
> >
> > Counters ct = running.getCounters();
> > /********* How do I Collect the ct_key1 value ????***********/
> > long res = ct.getCounter(MyCounter.ct_key1);
> >
> > }
> >
> >
> >
> >
> >
> > Thanks,
> >
> > Sharath
> >
>
Re: Problem with Counters
Posted by Tom White <to...@cloudera.com>.
Hi Sharath,
The code you posted looks right to me. Counters#getCounter() will
return the counter's value. What error are you getting?
Tom
On Thu, Feb 5, 2009 at 10:09 AM, some speed <sp...@gmail.com> wrote:
> Hi,
>
> Can someone help me with the usage of counters please? I am incrementing a
> counter in Reduce method but I am unable to collect the counter value after
> the job is completed.
>
> Its something like this:
>
> public static class Reduce extends MapReduceBase implements Reducer<Text,
> FloatWritable, Text, FloatWritable>
> {
> static enum MyCounter{ct_key1};
>
> public void reduce(..............) throws IOException
> {
>
> reporter.incrCounter(MyCounter.ct_key1, 1);
>
> output.collect(......);
>
> }
> }
>
> -----main method----
> {
> RunningJob running = null;
> running=JobClient.runJob(conf);
>
> Counters ct = running.getCounters();
> /********* How do I Collect the ct_key1 value ????***********/
> long res = ct.getCounter(MyCounter.ct_key1);
>
> }
>
>
>
>
>
> Thanks,
>
> Sharath
>