You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Thomas Jungblut (JIRA)" <ji...@apache.org> on 2011/05/09 23:02:03 UTC

[jira] [Commented] (HAMA-363) Add network condition monitoring function to BSPMaster

    [ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030895#comment-13030895 ] 

Thomas Jungblut commented on HAMA-363:
--------------------------------------

As far as I know Hadoop only provides some JVM metrics and host metrices. I don't exactly find the correct source code position, but I think we should implement our own metrics package, which we can later add to ganglia. This is much more useful. 

We should define things we need to determine whether there are problems or not. 
Something like: "We ping every groom every 5 seconds and check the latency."
This can be easily implemented in BSPMaster.

To measure the IN and OUT rate or other fancy stuff we need something like heartbeat communication that will transfer the local groom data to the master. 
This should be in the newer versions of Hadoop >0.21 shouldn't it? Don't have the source codes haging around here.

> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
>                 Key: HAMA-363
>                 URL: https://issues.apache.org/jira/browse/HAMA-363
>             Project: Hama
>          Issue Type: New Feature
>          Components: bsp
>    Affects Versions: 0.3.0
>            Reporter: Edward J. Yoon
>             Fix For: 0.3.0
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
>  * to optimize the network usage
>  * to handle network connectivity problems
>  * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [jira] [Commented] (HAMA-363) Add network condition monitoring function to BSPMaster

Posted by Steve Loughran <st...@apache.org>.
On 09/05/11 22:02, Thomas Jungblut (JIRA) wrote:
>
>      [ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030895#comment-13030895 ]
>
> Thomas Jungblut commented on HAMA-363:
> --------------------------------------
>
> As far as I know Hadoop only provides some JVM metrics and host metrices. I don't exactly find the correct source code position, but I think we should implement our own metrics package, which we can later add to ganglia. This is much more useful.
>
> We should define things we need to determine whether there are problems or not.
> Something like: "We ping every groom every 5 seconds and check the latency."
> This can be easily implemented in BSPMaster.
>
> To measure the IN and OUT rate or other fancy stuff we need something like heartbeat communication that will transfer the local groom data to the master.
> This should be in the newer versions of Hadoop>0.21 shouldn't it? Don't have the source codes haging around here.
>

if you are doing perf stuff, I'd go for having some plugin monitoring 
that can go in before/after communications.

Why? I'm playing with sFlow monitoring of bits of Hadoop, and it's 
tricky to retrofit this stuff deep into the code. If the hooks where 
there it's easier.