You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Hadoop Learner <ha...@gmail.com> on 2013/01/06 16:00:20 UTC
Reg: Fetching TaskAttempt Details from a RunningJob
Hi All,
Working on a requirement of hadoop Job Monitoring. Requirement is to get
every Task attempt details of a Running Job. Details are following :
Task Attempt Start Time
Task Tracker Name where Task Attempt is executing
Errors or Exceptions in a running Task Attempt
I have Implemented JobClient, RunningJob, JobStatus, TaskReport APIs to get
the Task Attempt IDs( I have the task Attempt IDs for which I need required
details). By using these APIs I was unable to get Task Attempt details as
per my requirement mentioned above. Can you please help me with some ways
to get required details using Java APIs.
Any pointers will be helpful.
(Note : I have tried using TaskInProgress API, But not sure how to get
TaskInProgress instance from a task ID. Also noticed TaskInProgress class
is not visible to my classes)
Thanks and Regards,
Shyam
Re: Reg: Fetching TaskAttempt Details from a RunningJob
Posted by Hadoop Learner <ha...@gmail.com>.
Any Suggestions?
Thanks and Regards,
Shyam
On Sun, Jan 6, 2013 at 3:00 PM, Hadoop Learner
<ha...@gmail.com>wrote:
> Hi All,
>
> Working on a requirement of hadoop Job Monitoring. Requirement is to get
> every Task attempt details of a Running Job. Details are following :
>
> Task Attempt Start Time
> Task Tracker Name where Task Attempt is executing
> Errors or Exceptions in a running Task Attempt
>
> I have Implemented JobClient, RunningJob, JobStatus, TaskReport APIs to
> get the Task Attempt IDs( I have the task Attempt IDs for which I need
> required details). By using these APIs I was unable to get Task Attempt
> details as per my requirement mentioned above. Can you please help me with
> some ways to get required details using Java APIs.
>
> Any pointers will be helpful.
>
> (Note : I have tried using TaskInProgress API, But not sure how to get
> TaskInProgress instance from a task ID. Also noticed TaskInProgress class
> is not visible to my classes)
>
> Thanks and Regards,
> Shyam
>
Re: Reg: Fetching TaskAttempt Details from a RunningJob
Posted by Hadoop Learner <ha...@gmail.com>.
Any Suggestions?
Thanks and Regards,
Shyam
On Sun, Jan 6, 2013 at 3:00 PM, Hadoop Learner
<ha...@gmail.com>wrote:
> Hi All,
>
> Working on a requirement of hadoop Job Monitoring. Requirement is to get
> every Task attempt details of a Running Job. Details are following :
>
> Task Attempt Start Time
> Task Tracker Name where Task Attempt is executing
> Errors or Exceptions in a running Task Attempt
>
> I have Implemented JobClient, RunningJob, JobStatus, TaskReport APIs to
> get the Task Attempt IDs( I have the task Attempt IDs for which I need
> required details). By using these APIs I was unable to get Task Attempt
> details as per my requirement mentioned above. Can you please help me with
> some ways to get required details using Java APIs.
>
> Any pointers will be helpful.
>
> (Note : I have tried using TaskInProgress API, But not sure how to get
> TaskInProgress instance from a task ID. Also noticed TaskInProgress class
> is not visible to my classes)
>
> Thanks and Regards,
> Shyam
>
Re: Reg: Fetching TaskAttempt Details from a RunningJob
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Hi,
In Hadoop 1.0, I don't think this information is exposed. The
TaskInProgress is an internal class and hence cannot / should not be used
from client applications. The only way out seems to be to screen scrape the
information from the Jobtracker web UI.
If you can live with completed events, then there is something called
TaskCompletionEvents that seem to provide some of this information. You
could look at JobClient.getTaskCompletionEvents.
Please note that in Hadoop 2.0 where mapreduce has been re-architected into
YARN, there are JSON APIs that seem to expose the information you require:
http://hadoop.apache.org/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html#Response_Examples
Look here for "taskAttempts"
Thanks
hemanth
On Sun, Jan 6, 2013 at 8:30 PM, Hadoop Learner
<ha...@gmail.com>wrote:
> Hi All,
>
> Working on a requirement of hadoop Job Monitoring. Requirement is to get
> every Task attempt details of a Running Job. Details are following :
>
> Task Attempt Start Time
> Task Tracker Name where Task Attempt is executing
> Errors or Exceptions in a running Task Attempt
>
> I have Implemented JobClient, RunningJob, JobStatus, TaskReport APIs to
> get the Task Attempt IDs( I have the task Attempt IDs for which I need
> required details). By using these APIs I was unable to get Task Attempt
> details as per my requirement mentioned above. Can you please help me with
> some ways to get required details using Java APIs.
>
> Any pointers will be helpful.
>
> (Note : I have tried using TaskInProgress API, But not sure how to get
> TaskInProgress instance from a task ID. Also noticed TaskInProgress class
> is not visible to my classes)
>
> Thanks and Regards,
> Shyam
>
Re: Reg: Fetching TaskAttempt Details from a RunningJob
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Hi,
In Hadoop 1.0, I don't think this information is exposed. The
TaskInProgress is an internal class and hence cannot / should not be used
from client applications. The only way out seems to be to screen scrape the
information from the Jobtracker web UI.
If you can live with completed events, then there is something called
TaskCompletionEvents that seem to provide some of this information. You
could look at JobClient.getTaskCompletionEvents.
Please note that in Hadoop 2.0 where mapreduce has been re-architected into
YARN, there are JSON APIs that seem to expose the information you require:
http://hadoop.apache.org/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html#Response_Examples
Look here for "taskAttempts"
Thanks
hemanth
On Sun, Jan 6, 2013 at 8:30 PM, Hadoop Learner
<ha...@gmail.com>wrote:
> Hi All,
>
> Working on a requirement of hadoop Job Monitoring. Requirement is to get
> every Task attempt details of a Running Job. Details are following :
>
> Task Attempt Start Time
> Task Tracker Name where Task Attempt is executing
> Errors or Exceptions in a running Task Attempt
>
> I have Implemented JobClient, RunningJob, JobStatus, TaskReport APIs to
> get the Task Attempt IDs( I have the task Attempt IDs for which I need
> required details). By using these APIs I was unable to get Task Attempt
> details as per my requirement mentioned above. Can you please help me with
> some ways to get required details using Java APIs.
>
> Any pointers will be helpful.
>
> (Note : I have tried using TaskInProgress API, But not sure how to get
> TaskInProgress instance from a task ID. Also noticed TaskInProgress class
> is not visible to my classes)
>
> Thanks and Regards,
> Shyam
>
Re: Reg: Fetching TaskAttempt Details from a RunningJob
Posted by Hadoop Learner <ha...@gmail.com>.
Any Suggestions?
Thanks and Regards,
Shyam
On Sun, Jan 6, 2013 at 3:00 PM, Hadoop Learner
<ha...@gmail.com>wrote:
> Hi All,
>
> Working on a requirement of hadoop Job Monitoring. Requirement is to get
> every Task attempt details of a Running Job. Details are following :
>
> Task Attempt Start Time
> Task Tracker Name where Task Attempt is executing
> Errors or Exceptions in a running Task Attempt
>
> I have Implemented JobClient, RunningJob, JobStatus, TaskReport APIs to
> get the Task Attempt IDs( I have the task Attempt IDs for which I need
> required details). By using these APIs I was unable to get Task Attempt
> details as per my requirement mentioned above. Can you please help me with
> some ways to get required details using Java APIs.
>
> Any pointers will be helpful.
>
> (Note : I have tried using TaskInProgress API, But not sure how to get
> TaskInProgress instance from a task ID. Also noticed TaskInProgress class
> is not visible to my classes)
>
> Thanks and Regards,
> Shyam
>
Re: Reg: Fetching TaskAttempt Details from a RunningJob
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Hi,
In Hadoop 1.0, I don't think this information is exposed. The
TaskInProgress is an internal class and hence cannot / should not be used
from client applications. The only way out seems to be to screen scrape the
information from the Jobtracker web UI.
If you can live with completed events, then there is something called
TaskCompletionEvents that seem to provide some of this information. You
could look at JobClient.getTaskCompletionEvents.
Please note that in Hadoop 2.0 where mapreduce has been re-architected into
YARN, there are JSON APIs that seem to expose the information you require:
http://hadoop.apache.org/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html#Response_Examples
Look here for "taskAttempts"
Thanks
hemanth
On Sun, Jan 6, 2013 at 8:30 PM, Hadoop Learner
<ha...@gmail.com>wrote:
> Hi All,
>
> Working on a requirement of hadoop Job Monitoring. Requirement is to get
> every Task attempt details of a Running Job. Details are following :
>
> Task Attempt Start Time
> Task Tracker Name where Task Attempt is executing
> Errors or Exceptions in a running Task Attempt
>
> I have Implemented JobClient, RunningJob, JobStatus, TaskReport APIs to
> get the Task Attempt IDs( I have the task Attempt IDs for which I need
> required details). By using these APIs I was unable to get Task Attempt
> details as per my requirement mentioned above. Can you please help me with
> some ways to get required details using Java APIs.
>
> Any pointers will be helpful.
>
> (Note : I have tried using TaskInProgress API, But not sure how to get
> TaskInProgress instance from a task ID. Also noticed TaskInProgress class
> is not visible to my classes)
>
> Thanks and Regards,
> Shyam
>
Re: Reg: Fetching TaskAttempt Details from a RunningJob
Posted by Hemanth Yamijala <yh...@thoughtworks.com>.
Hi,
In Hadoop 1.0, I don't think this information is exposed. The
TaskInProgress is an internal class and hence cannot / should not be used
from client applications. The only way out seems to be to screen scrape the
information from the Jobtracker web UI.
If you can live with completed events, then there is something called
TaskCompletionEvents that seem to provide some of this information. You
could look at JobClient.getTaskCompletionEvents.
Please note that in Hadoop 2.0 where mapreduce has been re-architected into
YARN, there are JSON APIs that seem to expose the information you require:
http://hadoop.apache.org/docs/r2.0.0-alpha/hadoop-yarn/hadoop-yarn-site/WebServicesIntro.html#Response_Examples
Look here for "taskAttempts"
Thanks
hemanth
On Sun, Jan 6, 2013 at 8:30 PM, Hadoop Learner
<ha...@gmail.com>wrote:
> Hi All,
>
> Working on a requirement of hadoop Job Monitoring. Requirement is to get
> every Task attempt details of a Running Job. Details are following :
>
> Task Attempt Start Time
> Task Tracker Name where Task Attempt is executing
> Errors or Exceptions in a running Task Attempt
>
> I have Implemented JobClient, RunningJob, JobStatus, TaskReport APIs to
> get the Task Attempt IDs( I have the task Attempt IDs for which I need
> required details). By using these APIs I was unable to get Task Attempt
> details as per my requirement mentioned above. Can you please help me with
> some ways to get required details using Java APIs.
>
> Any pointers will be helpful.
>
> (Note : I have tried using TaskInProgress API, But not sure how to get
> TaskInProgress instance from a task ID. Also noticed TaskInProgress class
> is not visible to my classes)
>
> Thanks and Regards,
> Shyam
>
Re: Reg: Fetching TaskAttempt Details from a RunningJob
Posted by Hadoop Learner <ha...@gmail.com>.
Any Suggestions?
Thanks and Regards,
Shyam
On Sun, Jan 6, 2013 at 3:00 PM, Hadoop Learner
<ha...@gmail.com>wrote:
> Hi All,
>
> Working on a requirement of hadoop Job Monitoring. Requirement is to get
> every Task attempt details of a Running Job. Details are following :
>
> Task Attempt Start Time
> Task Tracker Name where Task Attempt is executing
> Errors or Exceptions in a running Task Attempt
>
> I have Implemented JobClient, RunningJob, JobStatus, TaskReport APIs to
> get the Task Attempt IDs( I have the task Attempt IDs for which I need
> required details). By using these APIs I was unable to get Task Attempt
> details as per my requirement mentioned above. Can you please help me with
> some ways to get required details using Java APIs.
>
> Any pointers will be helpful.
>
> (Note : I have tried using TaskInProgress API, But not sure how to get
> TaskInProgress instance from a task ID. Also noticed TaskInProgress class
> is not visible to my classes)
>
> Thanks and Regards,
> Shyam
>