[jira] [Resolved] (HDFS-324) Generate a network infrastructre map

["Allen Wittenauer (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/HDFS-324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Allen Wittenauer resolved HDFS-324.
-----------------------------------

    Resolution: Duplicate

This is part of 289.

> Generate a network infrastructre map
> ------------------------------------
>
>                 Key: HDFS-324
>                 URL: https://issues.apache.org/jira/browse/HDFS-324
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Kai Mosebach
>
> Assuming an inhomogeneous network it might be sensible to not only collect metrics about to and from the nodes but also
> - collect latency between nodes
> - network io between nodes
> - daytime depending network io between nodes (i.e. network is much slower in the morning time and in the early evening time)
> - network failures
> Assuming the we collect aging information over a period of time, it would allow us to create a "learning cloud" in senses of its infrastructure. From this a network map we can
> - see subclusters of high speed linked nodes*
> - see unreliable connections between nodes
> - see heavily used links over the time
> These information can be refed into the DFS (for data distribution as well as for the balancer) logic to increase its reliability and its performance a lot.
> *Note : This could already be managed with the rack awareness, but the aging approach would make this much more fine grained and in an automatic manner.



--
This message was sent by Atlassian JIRA
(v6.2#6252)