You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hadoop.apache.org by Jeff LI <un...@gmail.com> on 2012/11/20 17:13:58 UTC
Start time, end time, and task tracker of individual tasks of a job
Hello,
Is there a way to obtain the information of each individual task of a
map-reduce job, including start time, end time, which task tracker runs
this task and so on?
I know this information can be found through the web interface running on
the jobtracter. But is it possible to redirect the information to a nicely
formatted log file for each job?
By the way, I'm running hadoop 0.20.2-cdh3u5.
Thanks advance for the help.
Cheers
Jeff
Re: Start time, end time, and task tracker of individual tasks of a job
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Most of this information is already available in the JobHistory files. And there are parsers to read from these files.
HTH
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Nov 20, 2012, at 8:13 AM, Jeff LI wrote:
> Hello,
>
> Is there a way to obtain the information of each individual task of a map-reduce job, including start time, end time, which task tracker runs this task and so on?
>
> I know this information can be found through the web interface running on the jobtracter. But is it possible to redirect the information to a nicely formatted log file for each job?
>
> By the way, I'm running hadoop 0.20.2-cdh3u5.
>
> Thanks advance for the help.
>
> Cheers
>
> Jeff
>
Re: Start time, end time, and task tracker of individual tasks of a job
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Most of this information is already available in the JobHistory files. And there are parsers to read from these files.
HTH
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Nov 20, 2012, at 8:13 AM, Jeff LI wrote:
> Hello,
>
> Is there a way to obtain the information of each individual task of a map-reduce job, including start time, end time, which task tracker runs this task and so on?
>
> I know this information can be found through the web interface running on the jobtracter. But is it possible to redirect the information to a nicely formatted log file for each job?
>
> By the way, I'm running hadoop 0.20.2-cdh3u5.
>
> Thanks advance for the help.
>
> Cheers
>
> Jeff
>
Re: Start time, end time, and task tracker of individual tasks of a job
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Most of this information is already available in the JobHistory files. And there are parsers to read from these files.
HTH
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Nov 20, 2012, at 8:13 AM, Jeff LI wrote:
> Hello,
>
> Is there a way to obtain the information of each individual task of a map-reduce job, including start time, end time, which task tracker runs this task and so on?
>
> I know this information can be found through the web interface running on the jobtracter. But is it possible to redirect the information to a nicely formatted log file for each job?
>
> By the way, I'm running hadoop 0.20.2-cdh3u5.
>
> Thanks advance for the help.
>
> Cheers
>
> Jeff
>
Re: Start time, end time, and task tracker of individual tasks of a job
Posted by Harsh J <ha...@cloudera.com>.
Hey Jeff,
Yes, we expose some information for each task completion event..
For Old API, use RunningJob, specifically:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/RunningJob.html#getTaskCompletionEvents(int)
For New API, use Job, specifically:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getTaskCompletionEvents(int)
On Tue, Nov 20, 2012 at 9:43 PM, Jeff LI <un...@gmail.com> wrote:
> Hello,
>
> Is there a way to obtain the information of each individual task of a
> map-reduce job, including start time, end time, which task tracker runs this
> task and so on?
>
> I know this information can be found through the web interface running on
> the jobtracter. But is it possible to redirect the information to a nicely
> formatted log file for each job?
>
> By the way, I'm running hadoop 0.20.2-cdh3u5.
>
> Thanks advance for the help.
>
> Cheers
>
> Jeff
>
--
Harsh J
Re: Start time, end time, and task tracker of individual tasks of a job
Posted by Vinod Kumar Vavilapalli <vi...@hortonworks.com>.
Most of this information is already available in the JobHistory files. And there are parsers to read from these files.
HTH
+Vinod Kumar Vavilapalli
Hortonworks Inc.
http://hortonworks.com/
On Nov 20, 2012, at 8:13 AM, Jeff LI wrote:
> Hello,
>
> Is there a way to obtain the information of each individual task of a map-reduce job, including start time, end time, which task tracker runs this task and so on?
>
> I know this information can be found through the web interface running on the jobtracter. But is it possible to redirect the information to a nicely formatted log file for each job?
>
> By the way, I'm running hadoop 0.20.2-cdh3u5.
>
> Thanks advance for the help.
>
> Cheers
>
> Jeff
>
Re: Start time, end time, and task tracker of individual tasks of a job
Posted by Harsh J <ha...@cloudera.com>.
Hey Jeff,
Yes, we expose some information for each task completion event..
For Old API, use RunningJob, specifically:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/RunningJob.html#getTaskCompletionEvents(int)
For New API, use Job, specifically:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getTaskCompletionEvents(int)
On Tue, Nov 20, 2012 at 9:43 PM, Jeff LI <un...@gmail.com> wrote:
> Hello,
>
> Is there a way to obtain the information of each individual task of a
> map-reduce job, including start time, end time, which task tracker runs this
> task and so on?
>
> I know this information can be found through the web interface running on
> the jobtracter. But is it possible to redirect the information to a nicely
> formatted log file for each job?
>
> By the way, I'm running hadoop 0.20.2-cdh3u5.
>
> Thanks advance for the help.
>
> Cheers
>
> Jeff
>
--
Harsh J
Re: Start time, end time, and task tracker of individual tasks of a job
Posted by Harsh J <ha...@cloudera.com>.
Hey Jeff,
Yes, we expose some information for each task completion event..
For Old API, use RunningJob, specifically:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/RunningJob.html#getTaskCompletionEvents(int)
For New API, use Job, specifically:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getTaskCompletionEvents(int)
On Tue, Nov 20, 2012 at 9:43 PM, Jeff LI <un...@gmail.com> wrote:
> Hello,
>
> Is there a way to obtain the information of each individual task of a
> map-reduce job, including start time, end time, which task tracker runs this
> task and so on?
>
> I know this information can be found through the web interface running on
> the jobtracter. But is it possible to redirect the information to a nicely
> formatted log file for each job?
>
> By the way, I'm running hadoop 0.20.2-cdh3u5.
>
> Thanks advance for the help.
>
> Cheers
>
> Jeff
>
--
Harsh J
Re: Start time, end time, and task tracker of individual tasks of a job
Posted by Harsh J <ha...@cloudera.com>.
Hey Jeff,
Yes, we expose some information for each task completion event..
For Old API, use RunningJob, specifically:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/RunningJob.html#getTaskCompletionEvents(int)
For New API, use Job, specifically:
http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapreduce/Job.html#getTaskCompletionEvents(int)
On Tue, Nov 20, 2012 at 9:43 PM, Jeff LI <un...@gmail.com> wrote:
> Hello,
>
> Is there a way to obtain the information of each individual task of a
> map-reduce job, including start time, end time, which task tracker runs this
> task and so on?
>
> I know this information can be found through the web interface running on
> the jobtracter. But is it possible to redirect the information to a nicely
> formatted log file for each job?
>
> By the way, I'm running hadoop 0.20.2-cdh3u5.
>
> Thanks advance for the help.
>
> Cheers
>
> Jeff
>
--
Harsh J