You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Haibo Chen (JIRA)" <ji...@apache.org> on 2016/07/21 04:15:20 UTC
[jira] [Assigned] (MAPREDUCE-6631) shuffle handler would benefit
from per-local-dir threads
[ https://issues.apache.org/jira/browse/MAPREDUCE-6631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Haibo Chen reassigned MAPREDUCE-6631:
-------------------------------------
Assignee: Haibo Chen
> shuffle handler would benefit from per-local-dir threads
> --------------------------------------------------------
>
> Key: MAPREDUCE-6631
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6631
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: nodemanager
> Affects Versions: 2.7.2, 3.0.0-alpha1
> Reporter: Nathan Roberts
> Assignee: Haibo Chen
>
> [~jlowe] and I discussed this while investigating I/O starvation we have been seeing on our clusters lately (possibly amplified by increased tez workloads).
> If a particular disk is being slow, it is very likely that all shuffle netty threads will be blocked on the read side of sendfile(). (sendfile() is asynchronous on the outbound socket side, but not on the read side.) This causes the entire shuffle subsystem to slow down.
> It seems like we could make the netty threads more asynchronous by introducing a small set of threads per local-dir that are responsible for the actual sendfile() invocations.
> This would not only improve shuffles that span drives, but also improve situations where there is a single large shuffle from a single local-dir. It would allow other drives to continue serving shuffle requests, AND avoid a large number of readers (2X number_of_cores by default) all fighting for the same drive, which becomes unfair to everything else on the system.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-help@hadoop.apache.org