You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Steve Schlosser <sw...@gmail.com> on 2007/10/03 16:20:32 UTC

Read counters in main?

Hello all

What is the procedure for reading counters after a job has run?  I
want to be able to run my job using JobClient.runJob(), then read out
some counters to make a decision about whether to run another
reduction or exit.  I can think of some hacky ways to do it, but there
must be some correct way to read the counters.

Thanks!

-steve

Re: Read counters in main?

Posted by Steve Schlosser <sw...@gmail.com>.

Thanks for the help.  This seems to do what I want, but now I'm having
another problem.  I get a null pointer exception in JobClient.getFs(),
which is called by submitJob().

Is there something else I need to set in the jobConf object I pass to
submitJob()?  It works fine when I use JobClient.runJob() instead of
submitJob()...

Here is the stack from when the exception is thrown:

java.lang.NullPointerException
        at org.apache.hadoop.mapred.JobClient.getFs(JobClient.java:236)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:272)
        at net.intelresearch.cvmHadoop.KeyedByLocationalCode$ReduceOnlyDriver.main(Unknown
Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
        at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:143)
        at net.intelresearch.cvmHadoop.Usage.main(Unknown Source)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:155)


On 10/3/07, Arun C Murthy <ar...@yahoo-inc.com> wrote:
> On Wed, Oct 03, 2007 at 10:20:32AM -0400, Steve Schlosser wrote:
> >Hello all
> >
> >What is the procedure for reading counters after a job has run?  I
> >want to be able to run my job using JobClient.runJob(), then read out
> >some counters to make a decision about whether to run another
> >reduction or exit.  I can think of some hacky ways to do it, but there
> >must be some correct way to read the counters.
> >
>
> Take a look at: http://lucene.apache.org/hadoop/api/org/apache/hadoop/mapred/JobClient.html#submitJob(org.apache.hadoop.mapred.JobConf).
>
> That returns a RunningJob, then use http://lucene.apache.org/hadoop/api/org/apache/hadoop/mapred/RunningJob.html#getCounters()
>
> Arun
>
> >Thanks!
> >
> >-steve
>

Re: Read counters in main?

Posted by Arun C Murthy <ar...@yahoo-inc.com>.

On Wed, Oct 03, 2007 at 10:20:32AM -0400, Steve Schlosser wrote:
>Hello all
>
>What is the procedure for reading counters after a job has run?  I
>want to be able to run my job using JobClient.runJob(), then read out
>some counters to make a decision about whether to run another
>reduction or exit.  I can think of some hacky ways to do it, but there
>must be some correct way to read the counters.
>

Take a look at: http://lucene.apache.org/hadoop/api/org/apache/hadoop/mapred/JobClient.html#submitJob(org.apache.hadoop.mapred.JobConf). 

That returns a RunningJob, then use http://lucene.apache.org/hadoop/api/org/apache/hadoop/mapred/RunningJob.html#getCounters()

Arun

>Thanks!
>
>-steve