You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Devaraj Das (JIRA)" <ji...@apache.org> on 2008/12/09 08:21:44 UTC

[jira] Created: (HADOOP-4807) Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.

Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------

                 Key: HADOOP-4807
                 URL: https://issues.apache.org/jira/browse/HADOOP-4807
             Project: Hadoop Core
          Issue Type: Sub-task
          Components: mapred
            Reporter: Devaraj Das
            Assignee: Devaraj Das
             Fix For: 0.20.0


Add framework hooks to get the IDs of running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker. These are required to inject failures.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-4807) Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-4807:
--------------------------------

    Status: Patch Available  (was: Open)

> Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4807
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4807
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4807.patch
>
>
> Add framework hooks to get the IDs of running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker. These are required to inject failures.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-4807) Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-4807:
--------------------------------

    Attachment: 4807.patch

Attached patch provides the functionality. The patch adds a new class TaskInProgressStatus that is passed along in the status report (like when the method JobSubmissionProtocol.getMapTaskReports() is invoked) to the client. Also, the ClusterStatus now has the list of active trackers.
Provided the following new command line options in the JobClient:
1) -list-trackers : display the list of active trackers in the cluster
2) -list-attempt-ids <jobId> <task-type> <task-state> : displays the list of tasks for a given job of a give type (like map or reduce) currently in a particular state (running or completed).


> Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4807
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4807
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4807.patch
>
>
> Add framework hooks to get the IDs of running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker. These are required to inject failures.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-4807) Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655520#action_12655520 ] 

Arun C Murthy commented on HADOOP-4807:
---------------------------------------

Super minor nit: I'd prefer to rename TaskInProgressStatusType to just Status! *smile*

> Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4807
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4807
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4807.patch
>
>
> Add framework hooks to get the IDs of running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker. These are required to inject failures.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-4807) Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.

Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Arun C Murthy updated HADOOP-4807:
----------------------------------

    Status: Open  (was: Patch Available)

Minor comments:

# I'd suggest we deprecate both the current ClusterStatus constructors and use only the new one - this helps with the complication in write and readFields too.
# Rather than introduce {set|is}{Failed|Pending|Completed|Running|Killed} methods in TaskInProgressStatus I'd suggest a single {get|set}Status which takes/returns the enum which is called with appropriate values from TaskInProgress.generateSingleReport.
# There are some debugging statements left over.

Long term: It would be really great to use the TaskInProgressStatus.TaskInProgressStatusType to maintain the TIP's state rather than all the booleans...

> Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4807
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4807
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4807.patch
>
>
> Add framework hooks to get the IDs of running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker. These are required to inject failures.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HADOOP-4807) Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das updated HADOOP-4807:
--------------------------------

    Attachment: 4807.3.patch

Thanks Arun/Sharad for looking at this. I have incorporated the comments. 
1) Made TaskInProgressStatusType a separate class all by itself and called it TIPStatus
2) Factored TaskInProgressStatus functionality into TaskReport since that is where it is needed
3) Added a new getClusterStatus method that takes a boolean argument and depending on whether it is true, sets the task tracker names as well. If false, only counts of blacklisted/active trackers are set (as is the behavior today). I did this since I wanted to avoid iterations over the tasktracker list for every call to assignTasks in CapacityScheduler, for example.

> Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4807
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4807
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4807.3.patch, 4807.patch
>
>
> Add framework hooks to get the IDs of running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker. These are required to inject failures.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-4807) Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.

Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655532#action_12655532 ] 

Sharad Agarwal commented on HADOOP-4807:
----------------------------------------

Clients may be interested in names of blacklisted trackers as well. I think -list-trackers should list all by default and perhaps take an argument to list blacklisted/active ones ?

> Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4807
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4807
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4807.patch
>
>
> Add framework hooks to get the IDs of running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker. These are required to inject failures.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HADOOP-4807) Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.

Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HADOOP-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12655896#action_12655896 ] 

Sharad Agarwal commented on HADOOP-4807:
----------------------------------------

looks good. 
Please take a note to document the new commands to the commands manual.

> Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4807
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4807
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4807.3.patch, 4807.patch
>
>
> Add framework hooks to get the IDs of running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker. These are required to inject failures.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (HADOOP-4807) Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.

Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HADOOP-4807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Devaraj Das resolved HADOOP-4807.
---------------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]

All test cases and test-patch passed. I committed this.

> Add framework hooks to get the running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker.
> -----------------------------------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-4807
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4807
>             Project: Hadoop Core
>          Issue Type: Sub-task
>          Components: mapred
>            Reporter: Devaraj Das
>            Assignee: Devaraj Das
>             Fix For: 0.20.0
>
>         Attachments: 4807.3.patch, 4807.patch
>
>
> Add framework hooks to get the IDs of running/completed/pending tasks for a given job. Add a way to query the list of currently active tasktrackers from the JobTracker. These are required to inject failures.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.