[jira] [Commented] (MAPREDUCE-2636) Scheduling over disks horizontally

["Steve Loughran (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059852#comment-13059852 ] 

Steve Loughran commented on MAPREDUCE-2636:
-------------------------------------------

right now the JT is ignorant of where blocks live on a server, only that they are server-local. Indeed, the Namenode doesn't know either. It would be quite a large change to add this information, and if it took up more namenode memory, large-filesystem sites would be reluctant to adopt the improvement. 

You'd also have to take into account not just the source disk, but the output disks, and maybe the location of the any intermediate/overspill storage; the JT would need to know not just how many slots there were free, but which disks each active task is reading and writing, either by having this data pushed to it, or by checking prior to scheduling.

Like I said, a lot of work. Rather than rushing to do this, I'd recommend you come up with a way of measuring the conflict that is occurring. That way different clusters (with different #s of disks/server) could get data on how much of an issue this is -and whether adding more HDDs to a server improves things, or, as more tasks get executed on multicore CPUs, whether it gets worse.

> Scheduling over disks horizontally
> ----------------------------------
>
>                 Key: MAPREDUCE-2636
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2636
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: job submission
>            Reporter: Evert Lammerts
>            Priority: Minor
>
> Based on this message: http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201106.mbox/browser
> The JT schedules tasks on nodes based on metadata it gets from the NN. The namenode does not know on which disk a block resides. It might happen that on a node running 4 tasks, all read from the same disk. This can affect performance.
> An optimization might be to schedule horizontally over disks instead of nodes. Any ideas?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira