You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org> on 2015/11/04 01:25:27 UTC
[jira] [Updated] (YARN-2681) Support bandwidth enforcement for
containers while reading from HDFS
[ https://issues.apache.org/jira/browse/YARN-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated YARN-2681:
------------------------------------------
Target Version/s: 2.8.0 (was: 2.7.2)
Moving new-features out of 2.7 maintenance lines into 2.8.0.
> Support bandwidth enforcement for containers while reading from HDFS
> --------------------------------------------------------------------
>
> Key: YARN-2681
> URL: https://issues.apache.org/jira/browse/YARN-2681
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: nodemanager
> Affects Versions: 2.5.1
> Environment: Linux
> Reporter: Nam H. Do
> Labels: BB2015-05-TBR
> Attachments: Traffic Control Design.png, YARN-2681.001.patch, YARN-2681.002.patch, YARN-2681.003.patch, YARN-2681.004.patch, YARN-2681.005.patch, YARN-2681.patch
>
>
> To read/write data from HDFS on data node, applications establise TCP/IP connections with the datanode. The HDFS read can be controled by setting Linux Traffic Control (TC) subsystem on the data node to make filters on appropriate connections.
> The current cgroups net_cls concept can not be applied on the node where the container is launched, netheir on data node since:
> - TC hanldes outgoing bandwidth only, so it can be set on container node (HDFS read = incoming data for the container)
> - Since HDFS data node is handled by only one process, it is not possible to use net_cls to separate connections from different containers to the datanode.
> Tasks:
> 1) Extend Resource model to define bandwidth enforcement rate
> 2) Monitor TCP/IP connection estabilised by container handling process and its child processes
> 3) Set Linux Traffic Control rules on data node base on address:port pairs in order to enforce bandwidth of outgoing data
> Concept: http://www.hit.bme.hu/~do/papers/EnforcementDesign.pdf
> Implementation:
> http://www.hit.bme.hu/~dohoai/documents/HdfsTrafficControl.pdf
> http://www.hit.bme.hu/~dohoai/documents/HdfsTrafficControl_UML_diagram.png
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)