[jira] Created: (HADOOP-4082) Generate a network infrastructre map

["Kai Mosebach (JIRA)" <ji...@apache.org>]

Generate a network infrastructre map
------------------------------------

                 Key: HADOOP-4082
                 URL: https://issues.apache.org/jira/browse/HADOOP-4082
             Project: Hadoop Core
          Issue Type: Improvement
          Components: dfs, metrics
            Reporter: Kai Mosebach


Assuming an inhomogeneous network it might be sensible to not only collect metrics about to and from the nodes but also

- collect latency between nodes
- network io between nodes
- daytime depending network io between nodes (i.e. network is much slower in the morning time and in the early evening time)
- network failures

Assuming the we collect aging information over a period of time, it would allow us to create a "learning cloud" in senses of its infrastructure. From this a network map we can

- see subclusters of high speed linked nodes*
- see unreliable connections between nodes
- see heavily used links over the time

These information can be refed into the DFS (for data distribution as well as for the balancer) logic to increase its reliability and its performance a lot.

*Note : This could already be managed with the rack awareness, but the aging approach would make this much more fine grained and in an automatic manner.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.