You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org> on 2008/12/08 18:16:46 UTC
[jira] Created: (HADOOP-4803) large pending jobs hog resources
large pending jobs hog resources
--------------------------------
Key: HADOOP-4803
URL: https://issues.apache.org/jira/browse/HADOOP-4803
Project: Hadoop Core
Issue Type: Bug
Components: contrib/fair-share
Reporter: Joydeep Sen Sarma
observing the cluster over the last day - one thing i noticed is that small jobs (single digit tasks) are not doing a good job competing against large jobs. what seems to happen is that:
- large job comes along and needs to wait for a while for other large jobs.
- slots are slowly transfered from one large job to another
- small tasks keep waiting forever.
is this an artifact of deficit based scheduling? it seems that long pending large jobs are out-scheduling small jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4803) large pending jobs hog resources
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12654472#action_12654472 ]
Joydeep Sen Sarma commented on HADOOP-4803:
-------------------------------------------
it's worse than the title would suggest. compute quotas are also not being honored.
i am observing our ETL pipeline not getting the configured number of map slots - whereas there's another jobs with no minimum slot guarantees that keeps hogging resources. this other task has a crazy looking deficit (from the advanced scheduler page):
Deficit: -17796533s
but regardless - i would have thought that honoring quotas would come before any form of fair scheduling (deficit based or not)
> large pending jobs hog resources
> --------------------------------
>
> Key: HADOOP-4803
> URL: https://issues.apache.org/jira/browse/HADOOP-4803
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Reporter: Joydeep Sen Sarma
>
> observing the cluster over the last day - one thing i noticed is that small jobs (single digit tasks) are not doing a good job competing against large jobs. what seems to happen is that:
> - large job comes along and needs to wait for a while for other large jobs.
> - slots are slowly transfered from one large job to another
> - small tasks keep waiting forever.
> is this an artifact of deficit based scheduling? it seems that long pending large jobs are out-scheduling small jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4803) large pending jobs hog resources
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671723#action_12671723 ]
Matei Zaharia commented on HADOOP-4803:
---------------------------------------
Yes exactly. The last time each pool was at its min / fair share is already being maintained by the preemption patch (HADOOP-4665), so it won't be much work. One other benefit of this change will be that jobs will tend to reuse the same slot more often, leading to more JVM reuse. This can be a bad thing if it leads to poor locality, but HADOOP-4667 will ensure that a job keeps using a node till it runs out of local blocks to read on that node, and then waits and switches to hopefully a node where it has more local data to process. This should give us the best of both JVM reuse and data locality. (When I talked to Arun and Owen about the use of deficits in the fair scheduler before, they were concerned that it may lead to less JVM reuse because jobs will jump between slots more often.)
> large pending jobs hog resources
> --------------------------------
>
> Key: HADOOP-4803
> URL: https://issues.apache.org/jira/browse/HADOOP-4803
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Reporter: Joydeep Sen Sarma
> Assignee: Matei Zaharia
>
> observing the cluster over the last day - one thing i noticed is that small jobs (single digit tasks) are not doing a good job competing against large jobs. what seems to happen is that:
> - large job comes along and needs to wait for a while for other large jobs.
> - slots are slowly transfered from one large job to another
> - small tasks keep waiting forever.
> is this an artifact of deficit based scheduling? it seems that long pending large jobs are out-scheduling small jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4803) large pending jobs hog resources
Posted by "Hemanth Yamijala (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671720#action_12671720 ]
Hemanth Yamijala commented on HADOOP-4803:
------------------------------------------
So, do you mean that you won't maintain job deficits at all, and instead you would maintain how much time pools (or jobs) have not got their min share, and sort by that ?
> large pending jobs hog resources
> --------------------------------
>
> Key: HADOOP-4803
> URL: https://issues.apache.org/jira/browse/HADOOP-4803
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Reporter: Joydeep Sen Sarma
> Assignee: Matei Zaharia
>
> observing the cluster over the last day - one thing i noticed is that small jobs (single digit tasks) are not doing a good job competing against large jobs. what seems to happen is that:
> - large job comes along and needs to wait for a while for other large jobs.
> - slots are slowly transfered from one large job to another
> - small tasks keep waiting forever.
> is this an artifact of deficit based scheduling? it seems that long pending large jobs are out-scheduling small jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4803) large pending jobs hog resources
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656686#action_12656686 ]
Joydeep Sen Sarma commented on HADOOP-4803:
-------------------------------------------
Ignore the previous comment (which may be due to a bug we are trying to diagnose separately). i would like to stick to the initial report - deficit based scheduling causes pending jobs to rise up in priority (effectively). especially when such jobs are large - they tend to hog the cluster once they can be scheduled.
i guess this does beg the question of why a large deficit was accumulated in the first place (and that maybe due to a bug) - but this does seem to call for some solution in any case.
one of the things that i think u had mentioned would make sense - instead of giving all incoming slots to a task with a large deficit - give it a large enough fraction so that it's 'catching up'. one way could be to assign additional weight multiplicator that's proportional to the deficit. this should leave some slots on an ongoing basis to new jobs without a lot of deficit. thoughts?
> large pending jobs hog resources
> --------------------------------
>
> Key: HADOOP-4803
> URL: https://issues.apache.org/jira/browse/HADOOP-4803
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Reporter: Joydeep Sen Sarma
> Assignee: Matei Zaharia
>
> observing the cluster over the last day - one thing i noticed is that small jobs (single digit tasks) are not doing a good job competing against large jobs. what seems to happen is that:
> - large job comes along and needs to wait for a while for other large jobs.
> - slots are slowly transfered from one large job to another
> - small tasks keep waiting forever.
> is this an artifact of deficit based scheduling? it seems that long pending large jobs are out-scheduling small jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4803) large pending jobs hog resources
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12656738#action_12656738 ]
Matei Zaharia commented on HADOOP-4803:
---------------------------------------
I agree, I think the "catching up" idea could help here. Basically the problem is the following - if you have a job with long tasks, when it makes it to the head of the queue (max deficit), it may grab a *lot* of slots and hold onto them for a while. Instead, we should give it a more moderate share - enough that it can "catch up" at a reasonable time. It may also be good to take into account task durations for the job in this equation - i.e. look further ahead for jobs with large tasks.
> large pending jobs hog resources
> --------------------------------
>
> Key: HADOOP-4803
> URL: https://issues.apache.org/jira/browse/HADOOP-4803
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Reporter: Joydeep Sen Sarma
> Assignee: Matei Zaharia
>
> observing the cluster over the last day - one thing i noticed is that small jobs (single digit tasks) are not doing a good job competing against large jobs. what seems to happen is that:
> - large job comes along and needs to wait for a while for other large jobs.
> - slots are slowly transfered from one large job to another
> - small tasks keep waiting forever.
> is this an artifact of deficit based scheduling? it seems that long pending large jobs are out-scheduling small jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4803) large pending jobs hog resources
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4803?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma reassigned HADOOP-4803:
-----------------------------------------
Assignee: Matei Zaharia
> large pending jobs hog resources
> --------------------------------
>
> Key: HADOOP-4803
> URL: https://issues.apache.org/jira/browse/HADOOP-4803
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Reporter: Joydeep Sen Sarma
> Assignee: Matei Zaharia
>
> observing the cluster over the last day - one thing i noticed is that small jobs (single digit tasks) are not doing a good job competing against large jobs. what seems to happen is that:
> - large job comes along and needs to wait for a while for other large jobs.
> - slots are slowly transfered from one large job to another
> - small tasks keep waiting forever.
> is this an artifact of deficit based scheduling? it seems that long pending large jobs are out-scheduling small jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4803) large pending jobs hog resources
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12671337#action_12671337 ]
Matei Zaharia commented on HADOOP-4803:
---------------------------------------
In fact upon further discussion with Joydeep and Dhruba, we may drop deficits altogether once we add preemption and just use a similar concept for guaranteed shares to make sure pools get their min share in order of how long they've been waiting for it. This will simplify the code and make the scheduler behavior easier to understand.
> large pending jobs hog resources
> --------------------------------
>
> Key: HADOOP-4803
> URL: https://issues.apache.org/jira/browse/HADOOP-4803
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Reporter: Joydeep Sen Sarma
> Assignee: Matei Zaharia
>
> observing the cluster over the last day - one thing i noticed is that small jobs (single digit tasks) are not doing a good job competing against large jobs. what seems to happen is that:
> - large job comes along and needs to wait for a while for other large jobs.
> - slots are slowly transfered from one large job to another
> - small tasks keep waiting forever.
> is this an artifact of deficit based scheduling? it seems that long pending large jobs are out-scheduling small jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4803) large pending jobs hog resources
Posted by "Matei Zaharia (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12716087#action_12716087 ]
Matei Zaharia commented on HADOOP-4803:
---------------------------------------
A note on progress for this issue: I have a tested patch that removes deficits and also adds support for FIFO pools, but I am waiting for HADOOP-4665 and HADOOP-4667 to be committed before posting it because it depends on those.
> large pending jobs hog resources
> --------------------------------
>
> Key: HADOOP-4803
> URL: https://issues.apache.org/jira/browse/HADOOP-4803
> Project: Hadoop Core
> Issue Type: Bug
> Components: contrib/fair-share
> Reporter: Joydeep Sen Sarma
> Assignee: Matei Zaharia
>
> observing the cluster over the last day - one thing i noticed is that small jobs (single digit tasks) are not doing a good job competing against large jobs. what seems to happen is that:
> - large job comes along and needs to wait for a while for other large jobs.
> - slots are slowly transfered from one large job to another
> - small tasks keep waiting forever.
> is this an artifact of deficit based scheduling? it seems that long pending large jobs are out-scheduling small jobs
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.