You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by ArunKumar <ar...@gmail.com> on 2011/12/11 07:44:52 UTC
Accessing Job counters displayed in WEB GUI in Hadoop Code
Hai guys !
Can i access the Job counters displayed in WEB GUI in Hadoop code when the
job finished their execution ?
If so, how can i access the values like "average task run time" and counters
"FILE/HDFS BYTES READ/WRITTEN" immediately after the job has completed in
JobQueueTaskScheduler Code or in some other code file ?
Thanks,
Arun
--
View this message in context: http://lucene.472066.n3.nabble.com/Accessing-Job-counters-displayed-in-WEB-GUI-in-Hadoop-Code-tp3576925p3576925.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: Accessing Job counters displayed in WEB GUI in Hadoop Code
Posted by ArunKumar <ar...@gmail.com>.
Guys !
How can i access Average map/reduce task run time for a job in Job Client
Code ?
Arun
--
View this message in context: http://lucene.472066.n3.nabble.com/Accessing-Job-counters-displayed-in-WEB-GUI-in-Hadoop-Code-tp3576925p3578967.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
Re: Accessing Job counters displayed in WEB GUI in Hadoop Code
Posted by Harsh J <ha...@cloudera.com>.
Hej again,
You can get task data via the completion events call from your client:
http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/mapreduce/Job.html#getTaskCompletionEvents(int)
They should carry the data you seek.
On Mon, Dec 12, 2011 at 4:47 AM, W.P. McNeill <bi...@gmail.com> wrote:
> You can read counter values from the Job.getCounters API
> (http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/mapreduce/Job.html).
> I'm not sure about the other information like execution times. I've
> been wondering that myself.
>
> On 12/10/11, ArunKumar <ar...@gmail.com> wrote:
>> Hai guys !
>>
>> Can i access the Job counters displayed in WEB GUI in Hadoop code when the
>> job finished their execution ?
>> If so, how can i access the values like "average task run time" and counters
>> "FILE/HDFS BYTES READ/WRITTEN" immediately after the job has completed in
>> JobQueueTaskScheduler Code or in some other code file ?
>>
>>
>> Thanks,
>> Arun
>>
>> --
>> View this message in context:
>> http://lucene.472066.n3.nabble.com/Accessing-Job-counters-displayed-in-WEB-GUI-in-Hadoop-Code-tp3576925p3576925.html
>> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>>
--
Harsh J
Re: Accessing Job counters displayed in WEB GUI in Hadoop Code
Posted by "W.P. McNeill" <bi...@gmail.com>.
You can read counter values from the Job.getCounters API
(http://hadoop.apache.org/common/docs/r0.20.1/api/org/apache/hadoop/mapreduce/Job.html).
I'm not sure about the other information like execution times. I've
been wondering that myself.
On 12/10/11, ArunKumar <ar...@gmail.com> wrote:
> Hai guys !
>
> Can i access the Job counters displayed in WEB GUI in Hadoop code when the
> job finished their execution ?
> If so, how can i access the values like "average task run time" and counters
> "FILE/HDFS BYTES READ/WRITTEN" immediately after the job has completed in
> JobQueueTaskScheduler Code or in some other code file ?
>
>
> Thanks,
> Arun
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Accessing-Job-counters-displayed-in-WEB-GUI-in-Hadoop-Code-tp3576925p3576925.html
> Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
>
RE: Accessing Job counters displayed in WEB GUI in Hadoop Code
Posted by Devaraj K <de...@huawei.com>.
Hi Arun,
You can get the Counters object from the job after completion and
then find the counter whichever you want in Counters object using
findCounter api. Please find the sample snippet for accessing the counter
after Job completion
Configuration conf = new Configuration();
Cluster cluster = new Cluster(conf);
Job job = Job.getInstance(cluster,conf);
result = job.waitForCompletion(true);
.....
.....
Counters counters = job.getCounters();
Counter c1 = counters.findCounter(CustomCOUNTER.NOOFBADRECORDS);
System.out.println(c1.getDisplayName()+":"+c1.getValue());
Devaraj K
-----Original Message-----
From: ArunKumar [mailto:arunk786@gmail.com]
Sent: Sunday, December 11, 2011 12:15 PM
To: hadoop-user@lucene.apache.org
Subject: Accessing Job counters displayed in WEB GUI in Hadoop Code
Hai guys !
Can i access the Job counters displayed in WEB GUI in Hadoop code when the
job finished their execution ?
If so, how can i access the values like "average task run time" and counters
"FILE/HDFS BYTES READ/WRITTEN" immediately after the job has completed in
JobQueueTaskScheduler Code or in some other code file ?
Thanks,
Arun
--
View this message in context:
http://lucene.472066.n3.nabble.com/Accessing-Job-counters-displayed-in-WEB-G
UI-in-Hadoop-Code-tp3576925p3576925.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.
RE: Accessing Job counters displayed in WEB GUI in Hadoop Code
Posted by Devaraj K <de...@huawei.com>.
Hi Arun,
You can get the Counters object from the job after completion and
then find the counter whichever you want in Counters object using
findCounter api. Please find the sample snippet for accessing the counter
after Job completion
Configuration conf = new Configuration();
Cluster cluster = new Cluster(conf);
Job job = Job.getInstance(cluster,conf);
result = job.waitForCompletion(true);
.....
.....
Counters counters = job.getCounters();
Counter c1 = counters.findCounter(CustomCOUNTER.NOOFBADRECORDS);
System.out.println(c1.getDisplayName()+":"+c1.getValue());
Devaraj K
-----Original Message-----
From: ArunKumar [mailto:arunk786@gmail.com]
Sent: Sunday, December 11, 2011 12:15 PM
To: hadoop-user@lucene.apache.org
Subject: Accessing Job counters displayed in WEB GUI in Hadoop Code
Hai guys !
Can i access the Job counters displayed in WEB GUI in Hadoop code when the
job finished their execution ?
If so, how can i access the values like "average task run time" and counters
"FILE/HDFS BYTES READ/WRITTEN" immediately after the job has completed in
JobQueueTaskScheduler Code or in some other code file ?
Thanks,
Arun
--
View this message in context:
http://lucene.472066.n3.nabble.com/Accessing-Job-counters-displayed-in-WEB-G
UI-in-Hadoop-Code-tp3576925p3576925.html
Sent from the Hadoop lucene-users mailing list archive at Nabble.com.