You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org> on 2010/10/07 10:01:32 UTC
[jira] Created: (MAPREDUCE-2116) optimize getTasksToKill to reduce
JobTracker contention
optimize getTasksToKill to reduce JobTracker contention
-------------------------------------------------------
Key: MAPREDUCE-2116
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2116
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: jobtracker
Reporter: Joydeep Sen Sarma
getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
at java.util.TreeMap.getEntry(TreeMap.java:328)
at java.util.TreeMap.get(TreeMap.java:255)
at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
- locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2116) optimize getTasksToKill to
reduce JobTracker contention
Posted by "Scott Chen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920293#action_12920293 ]
Scott Chen commented on MAPREDUCE-2116:
---------------------------------------
Hey louli,
That's a nice observation. Nice graph too.
bq. what if we keep a ref in TaskAttemptID class to point to its TIP ref?
Joydeep has similar idea. But he created another class that keeps a pair. We are testing this internally.
> optimize getTasksToKill to reduce JobTracker contention
> -------------------------------------------------------
>
> Key: MAPREDUCE-2116
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2116
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Joydeep Sen Sarma
> Attachments: 2116.1.patch, getTaskToKill.JPG
>
>
> getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
> at java.util.TreeMap.getEntry(TreeMap.java:328)
> at java.util.TreeMap.get(TreeMap.java:255)
> at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
> at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
> locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
> this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2116) optimize getTasksToKill to
reduce JobTracker contention
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920109#action_12920109 ]
Joydeep Sen Sarma commented on MAPREDUCE-2116:
----------------------------------------------
yeah - we are running with a patch that switches from TreeMap to HashMap (for the tracker->attempt map) and keeps a pair of (attempt, tip) in the map. we haven't deployed this on the main cluster yet - so can't say for certain how much this will help. i can upload a patch for trunk later this week.
> optimize getTasksToKill to reduce JobTracker contention
> -------------------------------------------------------
>
> Key: MAPREDUCE-2116
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2116
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Joydeep Sen Sarma
> Attachments: getTaskToKill.JPG
>
>
> getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
> at java.util.TreeMap.getEntry(TreeMap.java:328)
> at java.util.TreeMap.get(TreeMap.java:255)
> at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
> at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
> locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
> this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2116) optimize getTasksToKill to reduce
JobTracker contention
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated MAPREDUCE-2116:
-----------------------------------------
Attachment: 2116.2.patch
added the optimization to shouldClose(). don't have any other jira tracking that.
> optimize getTasksToKill to reduce JobTracker contention
> -------------------------------------------------------
>
> Key: MAPREDUCE-2116
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2116
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Joydeep Sen Sarma
> Attachments: 2116.1.patch, 2116.2.patch, getTaskToKill.JPG
>
>
> getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
> at java.util.TreeMap.getEntry(TreeMap.java:328)
> at java.util.TreeMap.get(TreeMap.java:255)
> at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
> at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
> locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
> this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2116) optimize getTasksToKill to reduce
JobTracker contention
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated MAPREDUCE-2116:
-----------------------------------------
Status: Patch Available (was: Open)
> optimize getTasksToKill to reduce JobTracker contention
> -------------------------------------------------------
>
> Key: MAPREDUCE-2116
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2116
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Joydeep Sen Sarma
> Attachments: 2116.1.patch, getTaskToKill.JPG
>
>
> getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
> at java.util.TreeMap.getEntry(TreeMap.java:328)
> at java.util.TreeMap.get(TreeMap.java:255)
> at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
> at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
> locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
> this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2116) optimize getTasksToKill to reduce
JobTracker contention
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated MAPREDUCE-2116:
-----------------------------------------
Attachment: 2116.1.patch
> optimize getTasksToKill to reduce JobTracker contention
> -------------------------------------------------------
>
> Key: MAPREDUCE-2116
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2116
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Joydeep Sen Sarma
> Attachments: 2116.1.patch, getTaskToKill.JPG
>
>
> getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
> at java.util.TreeMap.getEntry(TreeMap.java:328)
> at java.util.TreeMap.get(TreeMap.java:255)
> at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
> at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
> locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
> this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2116) optimize getTasksToKill to
reduce JobTracker contention
Posted by "luoli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920098#action_12920098 ]
luoli commented on MAPREDUCE-2116:
----------------------------------
!getTaskToKill.JPG!
This is really the issue, we encounter the same problem. Here is the picture of heartbeat call profiling, In one of our busy cluster, the getTaskToKill call consume about 27/32 cpu time of a heartbeat process in jobtracker. And we found that the TreeMap's get operate consume the most of the cpu time. one of the TreeMap is the taskidToTIPMap in jobtracker which keeps all the TaskAttemptID -> TIP map and will grow huge when the cluster running lots of job and each job contain hundreds of thousands of tasks. Also we found that in the shouldClose of TIP, the TreeMap of taskStatuses's get also very slow , and I don't know why is that because this treemap will contain two entry at most.
For the taskidToTIPMap's consume, what if we keep a ref in TaskAttemptID class to point to its TIP ref? If so , the call of taskidToTIPMap.get will become unnecessary since we can get a TaskAttemptID's corresponding TIP from it's own object. Like TaskAttemptID.getTaskInProgress()
I will upload a patch base of trunk soon
any Thoughts?
> optimize getTasksToKill to reduce JobTracker contention
> -------------------------------------------------------
>
> Key: MAPREDUCE-2116
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2116
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Joydeep Sen Sarma
> Attachments: getTaskToKill.JPG
>
>
> getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
> at java.util.TreeMap.getEntry(TreeMap.java:328)
> at java.util.TreeMap.get(TreeMap.java:255)
> at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
> at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
> locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
> this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2116) optimize getTasksToKill to reduce
JobTracker contention
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated MAPREDUCE-2116:
-----------------------------------------
Description:
getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
at java.util.TreeMap.getEntry(TreeMap.java:328)
at java.util.TreeMap.get(TreeMap.java:255)
at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
was:
getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
at java.util.TreeMap.getEntry(TreeMap.java:328)
at java.util.TreeMap.get(TreeMap.java:255)
at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
- locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
> optimize getTasksToKill to reduce JobTracker contention
> -------------------------------------------------------
>
> Key: MAPREDUCE-2116
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2116
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Joydeep Sen Sarma
>
> getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
> at java.util.TreeMap.getEntry(TreeMap.java:328)
> at java.util.TreeMap.get(TreeMap.java:255)
> at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
> at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
> locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
> this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2116) optimize getTasksToKill to
reduce JobTracker contention
Posted by "luoli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920905#action_12920905 ]
luoli commented on MAPREDUCE-2116:
----------------------------------
bq. Joydeep has similar idea. But he created another class that keeps a pair. We are testing this internally.
en~, that can work too.
And now we found the reason why the taskStatuses.get in shouldClose() consume so much time. That's because in getTasksToKill(), the shouldClose() got called so many times. This is because for every TaskAttemptID in taskIds which got from
Set<TaskAttemptID> taskIds = trackerToTaskMap.get(taskTracker);
call, the taskIds will contains hundreds of entry if the tasktracker have finished lots of task attempts and those task's job all have not finished finally. So all those task attempts are contained in trackerToTaskMap's value set. In this case, even lots of tasks attempt have finished in the tasktracker, it gets iterated in every heartbeat of the tasktracker before the jobs which they belong to have finished. This is not good.
> optimize getTasksToKill to reduce JobTracker contention
> -------------------------------------------------------
>
> Key: MAPREDUCE-2116
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2116
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Joydeep Sen Sarma
> Attachments: 2116.1.patch, getTaskToKill.JPG
>
>
> getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
> at java.util.TreeMap.getEntry(TreeMap.java:328)
> at java.util.TreeMap.get(TreeMap.java:255)
> at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
> at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
> locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
> this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2116) optimize getTasksToKill to reduce
JobTracker contention
Posted by "luoli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
luoli updated MAPREDUCE-2116:
-----------------------------
Attachment: getTaskToKill.JPG
getTaskToKill cpu counte
> optimize getTasksToKill to reduce JobTracker contention
> -------------------------------------------------------
>
> Key: MAPREDUCE-2116
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2116
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Joydeep Sen Sarma
> Attachments: getTaskToKill.JPG
>
>
> getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
> at java.util.TreeMap.getEntry(TreeMap.java:328)
> at java.util.TreeMap.get(TreeMap.java:255)
> at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
> at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
> locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
> this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2116) optimize getTasksToKill to
reduce JobTracker contention
Posted by "Kang Xiao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12925738#action_12925738 ]
Kang Xiao commented on MAPREDUCE-2116:
--------------------------------------
bq. That's because in getTasksToKill(), the shouldClose() got called so many times.
Profiling shows that getTasksToSave() also calls taskidToTIPMap.get() but uses much less cpu. The main differences between getTasksToKill() and getTasksToSave() are:
* tip.shouldCommit(taskId) is simpler than tip.shouldClose(taskId), the patch has optimized tip.shouldClose(taskId)
* getTasksToKill() iterators on trackerToTaskMap.get(taskTracker) task attempt set while getTasksToSave() iterators on tasktrackerstatus.getTaskReports() set. The set iteratored by getTasksToSave() is smaller than getTasksToKill(). It still need to be optimized.
It may be not necessary for getTasksToKill() to iterators on trackerToTaskMap.get(taskTracker) which is all running/complted tasks of tasktracker:
# running tasks are included in tasktrackerstatus.getTaskReports()
# user killed task can not be completed, according to TaskInProgress.killTask()
# completed maps on tasktracker can be purged by KillJobAction upon job termination
# lostTaskTracker() clear tasks for tasktracker by calling trackerToTaskMap.remove(taskTracker)
> optimize getTasksToKill to reduce JobTracker contention
> -------------------------------------------------------
>
> Key: MAPREDUCE-2116
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2116
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Joydeep Sen Sarma
> Attachments: 2116.1.patch, 2116.2.patch, 2116.3.patch, getTaskToKill.JPG
>
>
> getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
> at java.util.TreeMap.getEntry(TreeMap.java:328)
> at java.util.TreeMap.get(TreeMap.java:255)
> at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
> at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
> locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
> this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-2116) optimize getTasksToKill to reduce
JobTracker contention
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma updated MAPREDUCE-2116:
-----------------------------------------
Attachment: 2116.3.patch
revised patch addressing findBugs warnings. here's the hadoopQA output:
[exec] BUILD SUCCESSFUL
[exec] Total time: 1 minute 25 seconds
[exec]
[exec]
[exec]
[exec]
[exec] -1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new or modified tests.
[exec] Please justify why no new tests are needed for this patch.
[exec] Also please list what manual steps were performed to verify this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec] +1 system tests framework. The patch passed system tests framework compile.
[exec]
[exec]
[exec]
[exec]
[exec] ======================================================================
[exec] ======================================================================
[exec] Finished build.
[exec] ======================================================================
[exec] ======================================================================
[exec]
> optimize getTasksToKill to reduce JobTracker contention
> -------------------------------------------------------
>
> Key: MAPREDUCE-2116
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2116
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Joydeep Sen Sarma
> Attachments: 2116.1.patch, 2116.2.patch, 2116.3.patch, getTaskToKill.JPG
>
>
> getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
> at java.util.TreeMap.getEntry(TreeMap.java:328)
> at java.util.TreeMap.get(TreeMap.java:255)
> at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
> at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
> locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
> this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2116) optimize getTasksToKill to
reduce JobTracker contention
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12921040#action_12921040 ]
Joydeep Sen Sarma commented on MAPREDUCE-2116:
----------------------------------------------
@luoli - shouldClose() had an expensive synchronized call that was not necessary. i have updated the patch with that change - it should help things (after we made this change - shouldClose() stopped appearing as one of the top calls in the profile).
beyond this - we are working on restructuring the locks so that the JT lock is not held for calls that are made at the JIP level. think about it - there's no reason why we should hold JT lock when trying to find out whether a taskattempt needs to be closed (it should suffice to hold the JIP lock at the most). we are tracking it via MAPREDUCE-2118.
> optimize getTasksToKill to reduce JobTracker contention
> -------------------------------------------------------
>
> Key: MAPREDUCE-2116
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2116
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: jobtracker
> Reporter: Joydeep Sen Sarma
> Attachments: 2116.1.patch, 2116.2.patch, getTaskToKill.JPG
>
>
> getTasksToKill shows up as one of the top routines holding the JT lock. Specifically, the translation from attemptid to tip is very expensive:
> at java.util.TreeMap.getEntry(TreeMap.java:328)
> at java.util.TreeMap.get(TreeMap.java:255)
> at org.apache.hadoop.mapred.TaskInProgress.shouldClose(TaskInProgress.java:500)
> at org.apache.hadoop.mapred.JobTracker.getTasksToKill(JobTracker.java:3464)
> locked <0x00002aab6ebb6640> (a org.apache.hadoop.mapred.JobTracker)
> at org.apache.hadoop.mapred.JobTracker.heartbeat(JobTracker.java:3181)
> this seems like an avoidable expense since the tip for a given attempt is fixed (and one should not need a map lookup to find the association). on a different note - not clear to me why TreeMaps are in use here (i didn't find any iteration over these maps). any background info on why things are arranged the way they are would be useful.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.