You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Hong Tang (JIRA)" <ji...@apache.org> on 2009/04/23 20:59:30 UTC
[jira] Created: (HADOOP-5733) Add map/reduce slot capacity and lost
map/reduce slot capacity to JobTracker metrics
Add map/reduce slot capacity and lost map/reduce slot capacity to JobTracker metrics
------------------------------------------------------------------------------------
Key: HADOOP-5733
URL: https://issues.apache.org/jira/browse/HADOOP-5733
Project: Hadoop Core
Issue Type: Improvement
Components: mapred, metrics
Reporter: Hong Tang
It would be nice to have the actual map/reduce slot capacity and the lost map/reduce slot capacity (# of blacklisted nodes * map-slot-per-node or reduce-slot-per-node). This information can be used to calculate a JT view of slot utilization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5733) Add map/reduce slot capacity and lost
map/reduce slot capacity to JobTracker metrics
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated HADOOP-5733:
-------------------------------------------
Attachment: hadoop-5733-2.patch
Attaching patch incorporating the comment:
* Changed map slot and reduce slot metric from {{incrMetric}} to {{setMetric}}
* Changed the field holding, map slots and reduce slots to volatile, so the setters need not be synchronized.
* The maps and reduce slot is set in {{updateTaskTrackerStatus}} in {{JobTracker}}
* The setters for black listed slots have been made synchronized.
> Add map/reduce slot capacity and lost map/reduce slot capacity to JobTracker metrics
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5733
> URL: https://issues.apache.org/jira/browse/HADOOP-5733
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, metrics
> Reporter: Hong Tang
> Attachments: hadoop-5733-1.patch, hadoop-5733-2.patch
>
>
> It would be nice to have the actual map/reduce slot capacity and the lost map/reduce slot capacity (# of blacklisted nodes * map-slot-per-node or reduce-slot-per-node). This information can be used to calculate a JT view of slot utilization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5733) Add map/reduce slot capacity and
lost map/reduce slot capacity to JobTracker metrics
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12706154#action_12706154 ]
Hudson commented on HADOOP-5733:
--------------------------------
Integrated in Hadoop-trunk #827 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/827/])
. Add map/reduce slot capacity and blacklisted capacity to JobTracker metrics. Contributed by Sreekanth Ramakrishnan
> Add map/reduce slot capacity and lost map/reduce slot capacity to JobTracker metrics
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5733
> URL: https://issues.apache.org/jira/browse/HADOOP-5733
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, metrics
> Reporter: Hong Tang
> Assignee: Sreekanth Ramakrishnan
> Fix For: 0.21.0
>
> Attachments: hadoop-5733-1.patch, hadoop-5733-2.patch, hadoop-5733-3.patch, hadoop-5733-4.patch
>
>
> It would be nice to have the actual map/reduce slot capacity and the lost map/reduce slot capacity (# of blacklisted nodes * map-slot-per-node or reduce-slot-per-node). This information can be used to calculate a JT view of slot utilization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5733) Add map/reduce slot capacity and lost
map/reduce slot capacity to JobTracker metrics
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-5733:
----------------------------------
Attachment: hadoop-5733-4.patch
Merged with trunk, as conflicts with HADOOP-5738. Also re-added the blacklist metric resets {{to doUpdates}}, since {{incrMetric}} semantics still require it there.
{noformat}
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no tests are needed for this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 Eclipse classpath. The patch retains Eclipse classpath integrity.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
{noformat}
> Add map/reduce slot capacity and lost map/reduce slot capacity to JobTracker metrics
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5733
> URL: https://issues.apache.org/jira/browse/HADOOP-5733
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, metrics
> Reporter: Hong Tang
> Attachments: hadoop-5733-1.patch, hadoop-5733-2.patch, hadoop-5733-3.patch, hadoop-5733-4.patch
>
>
> It would be nice to have the actual map/reduce slot capacity and the lost map/reduce slot capacity (# of blacklisted nodes * map-slot-per-node or reduce-slot-per-node). This information can be used to calculate a JT view of slot utilization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5733) Add map/reduce slot capacity and lost
map/reduce slot capacity to JobTracker metrics
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated HADOOP-5733:
-------------------------------------------
Attachment: hadoop-5733-3.patch
*Not resetting the metrics field during {{doUpdates}}
* Setting of the slots from add/removeHostCapacity have been removed because, in previous patches case, the map and reduce slots were incremental fields so, when ever the capacities were added/removed it was adjusted. Now, since it is static it is set whenever the TT statuses have updated the JT's internal capacity fields. But it is retained when tracker is marked blacklisted we increment/decrement in add/removeHostCapacity.
> Add map/reduce slot capacity and lost map/reduce slot capacity to JobTracker metrics
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5733
> URL: https://issues.apache.org/jira/browse/HADOOP-5733
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, metrics
> Reporter: Hong Tang
> Attachments: hadoop-5733-1.patch, hadoop-5733-2.patch, hadoop-5733-3.patch
>
>
> It would be nice to have the actual map/reduce slot capacity and the lost map/reduce slot capacity (# of blacklisted nodes * map-slot-per-node or reduce-slot-per-node). This information can be used to calculate a JT view of slot utilization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5733) Add map/reduce slot capacity and
lost map/reduce slot capacity to JobTracker metrics
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705808#action_12705808 ]
Chris Douglas commented on HADOOP-5733:
---------------------------------------
bq. Setting of the slots from add/removeHostCapacity have been removed because, in previous patches case, the map and reduce slots were incremental fields so, when ever the capacities were added/removed it was adjusted. Now, since it is static it is set whenever the TT statuses have updated the JT's internal capacity fields. But it is retained when tracker is marked blacklisted we increment/decrement in add/removeHostCapacity.
Sorry, I forgot to acknowledge this. Thanks for the explanation
> Add map/reduce slot capacity and lost map/reduce slot capacity to JobTracker metrics
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5733
> URL: https://issues.apache.org/jira/browse/HADOOP-5733
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, metrics
> Reporter: Hong Tang
> Assignee: Sreekanth Ramakrishnan
> Fix For: 0.21.0
>
> Attachments: hadoop-5733-1.patch, hadoop-5733-2.patch, hadoop-5733-3.patch, hadoop-5733-4.patch
>
>
> It would be nice to have the actual map/reduce slot capacity and the lost map/reduce slot capacity (# of blacklisted nodes * map-slot-per-node or reduce-slot-per-node). This information can be used to calculate a JT view of slot utilization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5733) Add map/reduce slot capacity and
lost map/reduce slot capacity to JobTracker metrics
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705523#action_12705523 ]
Chris Douglas commented on HADOOP-5733:
---------------------------------------
* When using {{setMetric}}, doUpdates shouldn't reset the metric to 0
* set\*Slots doesn't need to be adjusted in add/removeHostCapacity, as in the original patch?
> Add map/reduce slot capacity and lost map/reduce slot capacity to JobTracker metrics
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5733
> URL: https://issues.apache.org/jira/browse/HADOOP-5733
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, metrics
> Reporter: Hong Tang
> Attachments: hadoop-5733-1.patch, hadoop-5733-2.patch
>
>
> It would be nice to have the actual map/reduce slot capacity and the lost map/reduce slot capacity (# of blacklisted nodes * map-slot-per-node or reduce-slot-per-node). This information can be used to calculate a JT view of slot utilization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5733) Add map/reduce slot capacity and lost
map/reduce slot capacity to JobTracker metrics
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-5733:
------------------------------------
Attachment: hadoop-5733-v20.patch
Attached example for earlier version not to be committed.
> Add map/reduce slot capacity and lost map/reduce slot capacity to JobTracker metrics
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5733
> URL: https://issues.apache.org/jira/browse/HADOOP-5733
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, metrics
> Reporter: Hong Tang
> Assignee: Sreekanth Ramakrishnan
> Fix For: 0.21.0
>
> Attachments: hadoop-5733-1.patch, hadoop-5733-2.patch, hadoop-5733-3.patch, hadoop-5733-4.patch, hadoop-5733-v20.patch
>
>
> It would be nice to have the actual map/reduce slot capacity and the lost map/reduce slot capacity (# of blacklisted nodes * map-slot-per-node or reduce-slot-per-node). This information can be used to calculate a JT view of slot utilization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-5733) Add map/reduce slot capacity and
lost map/reduce slot capacity to JobTracker metrics
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas resolved HADOOP-5733.
-----------------------------------
Resolution: Fixed
Fix Version/s: 0.21.0
Assignee: Sreekanth Ramakrishnan
Hadoop Flags: [Reviewed]
I committed this. Thanks, Sreekanth
> Add map/reduce slot capacity and lost map/reduce slot capacity to JobTracker metrics
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5733
> URL: https://issues.apache.org/jira/browse/HADOOP-5733
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, metrics
> Reporter: Hong Tang
> Assignee: Sreekanth Ramakrishnan
> Fix For: 0.21.0
>
> Attachments: hadoop-5733-1.patch, hadoop-5733-2.patch, hadoop-5733-3.patch, hadoop-5733-4.patch
>
>
> It would be nice to have the actual map/reduce slot capacity and the lost map/reduce slot capacity (# of blacklisted nodes * map-slot-per-node or reduce-slot-per-node). This information can be used to calculate a JT view of slot utilization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-5733) Add map/reduce slot capacity and
lost map/reduce slot capacity to JobTracker metrics
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12705493#action_12705493 ]
Chris Douglas commented on HADOOP-5733:
---------------------------------------
Looks good
For the map/reduce slots:
* Instead of \{add,dec\}\*Slots, consider adding set\*Slots to the instrumentation and update with total\*TaskCapacity (use MetricsRecord::setMetric)
* Updates can occur outside the synchronized block in addHostCapacity and removeHostCapacity. With get/set, the field in the metrics can be volatile and updated without synchronizing on the instrumentation
For the blacklisted slots:
* The add/dec methods should be synchronized; there's a race condition with doUpdate
> Add map/reduce slot capacity and lost map/reduce slot capacity to JobTracker metrics
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5733
> URL: https://issues.apache.org/jira/browse/HADOOP-5733
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, metrics
> Reporter: Hong Tang
> Attachments: hadoop-5733-1.patch
>
>
> It would be nice to have the actual map/reduce slot capacity and the lost map/reduce slot capacity (# of blacklisted nodes * map-slot-per-node or reduce-slot-per-node). This information can be used to calculate a JT view of slot utilization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-5733) Add map/reduce slot capacity and lost
map/reduce slot capacity to JobTracker metrics
Posted by "Sreekanth Ramakrishnan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-5733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sreekanth Ramakrishnan updated HADOOP-5733:
-------------------------------------------
Attachment: hadoop-5733-1.patch
Attaching patch addressing this issue:
Added following new fields:
* map_slots : Number of Map slots in Cluster
* reduce_slots : Number of reduce slots in cluster.
* blacklisted_maps : Number of maps slots black listed.
* blacklisted_reduces : Number of reduce slots black listed.
Made changes in {{JobTracker}} to publish these metrics.
> Add map/reduce slot capacity and lost map/reduce slot capacity to JobTracker metrics
> ------------------------------------------------------------------------------------
>
> Key: HADOOP-5733
> URL: https://issues.apache.org/jira/browse/HADOOP-5733
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred, metrics
> Reporter: Hong Tang
> Attachments: hadoop-5733-1.patch
>
>
> It would be nice to have the actual map/reduce slot capacity and the lost map/reduce slot capacity (# of blacklisted nodes * map-slot-per-node or reduce-slot-per-node). This information can be used to calculate a JT view of slot utilization.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.