You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Jason Lowe (JIRA)" <ji...@apache.org> on 2017/07/18 20:59:00 UTC
[jira] [Commented] (MAPREDUCE-6916) History server scheduling tasks
at fixed rate can be problematic when those tasks are slow
[ https://issues.apache.org/jira/browse/MAPREDUCE-6916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16092179#comment-16092179 ]
Jason Lowe commented on MAPREDUCE-6916:
---------------------------------------
Slow or temporarily unavailable namenodes and slow datanodes are examples of things that can cause one or both of these tasks to execute more slowly than it should. We want to run these tasks at a fixed rate, but we don't want to "catch up" any missing task executions if we fall behind since there's no point to that. A better design would be to have the tasks that are actually scheduled with the executor to not do the full job but simply notify another thread that is responsible for doing the task. That way the executor tasks are quick and should never back up in practice, and we won't end up double-executing if we notify more than once while the task is still running.
> History server scheduling tasks at fixed rate can be problematic when those tasks are slow
> ------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-6916
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6916
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
> Affects Versions: 2.7.4
> Reporter: Jason Lowe
>
> The job history server currently schedules both the task of moving jobs from intermediate to done and the task of cleaning jobs at a fixed rate. If those tasks take longer than the rate period to execute then a backlog of to-be-scheduled tasks can build up and cause a long storm of them to execute later when the blockage clears.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org