You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hama.apache.org by "Edward J. Yoon (JIRA)" <ji...@apache.org> on 2011/03/10 06:31:59 UTC
[jira] Created: (HAMA-363) Assign a monitoring role to BSPMaster
Assign a monitoring role to BSPMaster
-------------------------------------
Key: HAMA-363
URL: https://issues.apache.org/jira/browse/HAMA-363
Project: Hama
Issue Type: New Feature
Components: bsp
Affects Versions: 0.3.0
Reporter: Edward J. Yoon
Fix For: 0.3.0
Basically Master server should know the status of cluster:
* to optimize the network usage
* to handle network connectivity problems
* to handle different network conditions
And, I would like to see some network usage statistics.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "ChiaHung Lin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158160#comment-13158160 ]
ChiaHung Lin commented on HAMA-363:
-----------------------------------
If just regarding to patch, it looks ok to commit it. But some other enhancement are required such as federation so that in the future the system would have better idea about e.g groom servers status.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "ChiaHung Lin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13053173#comment-13053173 ]
ChiaHung Lin commented on HAMA-363:
-----------------------------------
Hi Steve,
The way by plugin to receive internal state/ statistics can be done through implementing MetricsSink interface and register sink to the MetricsSystem. The metrics system will periodically harvests internal state from source, which implements MetricsSource, putting it to sink. For example, at the moment there is a JvmMetrics which records java vm related information; therefore, by registering a sink one can obtain jvm statistics periodically.
This seems to me correspond to the behaviour in use case explained in your comment. For monitoring network connection, an instance of network MetricsSource can be added to the metrics system beforehand, and a sink can be implemented for receiving related statistics such as source and dest value, etc. But there may have something I miss or I am not aware of due to my limited knowledge. Any chance you can help point out if something is missing in the current system?
Thank you for the feedback. It is very important to know actual user scenarios for improvement.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HAMA-363) Assign a network condition monitoring
role to BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-363:
--------------------------------
Labels: GSoC gsoc2011 mentor (was: )
Summary: Assign a network condition monitoring role to BSPMaster (was: Assign a monitoring role to BSPMaster)
> Assign a network condition monitoring role to BSPMaster
> -------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Labels: GSoC, gsoc2011, mentor
> Fix For: 0.3.0
>
>
> Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052371#comment-13052371 ]
Edward J. Yoon commented on HAMA-363:
-------------------------------------
Okay~
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-363:
--------------------------------
Summary: Add network condition monitoring function to BSPMaster (was: Add network condition monitoring role to BSPMaster)
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Labels: GSoC, gsoc2011, mentor
> Fix For: 0.3.0
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "ChiaHung Lin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115169#comment-13115169 ]
ChiaHung Lin commented on HAMA-363:
-----------------------------------
I will update a new patch to catch up the code in trunk.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HAMA-363) Add network condition monitoring role to
BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-363:
--------------------------------
Description:
There's few reason why this issue is important. Basically Master server should know the status of cluster:
* to optimize the network usage
* to handle network connectivity problems
* to handle different network conditions
And, I would like to see some network usage statistics.
In this issue, we implement only basic function which is collect network usage.
was:
Basically Master server should know the status of cluster:
* to optimize the network usage
* to handle network connectivity problems
* to handle different network conditions
And, I would like to see some network usage statistics.
Summary: Add network condition monitoring role to BSPMaster (was: Add network condition monitoring function to BSPMaster)
> Add network condition monitoring role to BSPMaster
> --------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Labels: GSoC, gsoc2011, mentor
> Fix For: 0.3.0
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "ChiaHung Lin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052365#comment-13052365 ]
ChiaHung Lin commented on HAMA-363:
-----------------------------------
Personally I am not rush to push this patch to 0.3 as this patch is a bit more related to fault tolerance; and it may need some improvement when usage cases increase.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052497#comment-13052497 ]
Steve Loughran commented on HAMA-363:
-------------------------------------
I've been doing some monitoring of HDFS; it's easy to plug in custom stuff in front of the filesystem, but not so good in the internals.
For HDFS, hama, etc, I'd like to be able to inject monitors into the code (i.e. a plugin point) once every network connection gets set up, to determine source and dest (hostname:port) values, plus #of bytes read/written. Just a thought. I wouldn't put Hama support for this on my priority need, but with a plugin point here the data can be collected and sampled or streamed to things for post-execution analysis.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "ChiaHung Lin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ChiaHung Lin updated HAMA-363:
------------------------------
Attachment: HAMA-363.patch
The attached files contains metrics system, its concept is basically borrowed from hadoop's metrics2. The intension is to let Hama have ideas of current system related status.
I agree what Thomas mentioned that Hama needs to have its own metrics package because IMO Hama and Hadoop's MapReduce are not completely the same. So the implementation does not just copy and paste metrics2 from Hadoop.
In addition, the patch at the moment just pass unit test. I will find spare time to test it on e.g vm; and it is highly appreciated if someone can help review as well.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-363:
--------------------------------
Summary: Add network condition monitoring function to BSPMaster (was: Assign a network condition monitoring role to BSPMaster)
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Labels: GSoC, gsoc2011, mentor
> Fix For: 0.3.0
>
>
> Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "ChiaHung Lin (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ChiaHung Lin resolved HAMA-363.
-------------------------------
Resolution: Fixed
Assignee: ChiaHung Lin
I've committed the patch accommodating the current package structure.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: ChiaHung Lin
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "ChiaHung Lin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010006#comment-13010006 ]
ChiaHung Lin commented on HAMA-363:
-----------------------------------
To my best of knowledge, Hadoop has metrics package which exposes runtime metrics for collecting data/ debugging. Applying Ganglia seems to be an alternative way for monitor network usage, connectivity, etc.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.3.0
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13115441#comment-13115441 ]
Edward J. Yoon commented on HAMA-363:
-------------------------------------
+1
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13052498#comment-13052498 ]
Steve Loughran commented on HAMA-363:
-------------------------------------
some more thoughts
# ganglia and collectl are the ways everyone monitors Hadoop, it would be simple and consistent for Hama to say "this is what we target"
# the monitoring v2 stuff in 0.23 is still unstable. At some point it might be nice to make re-usable, but it isn't there yet.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "ChiaHung Lin (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ChiaHung Lin updated HAMA-363:
------------------------------
Attachment: HAMA-363.patch
Change source tree for accommodating maven package structure. Also the diff is done under core package.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: ChiaHung Lin
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114355#comment-13114355 ]
Thomas Jungblut commented on HAMA-363:
--------------------------------------
I would love to see this in 0.4.0.
Would you please update this patch to the current trunk?
If you receive your commit rights, you can commit it for yourself.
I'm +1 after reviewing it.
And I'm wondering if we can implement this directly to our deamons.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Edward J. Yoon updated HAMA-363:
--------------------------------
Fix Version/s: (was: 0.3.0)
0.4.0
Let's schedule this to 0.4
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "ChiaHung Lin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13032245#comment-13032245 ]
ChiaHung Lin commented on HAMA-363:
-----------------------------------
If I understand correctly, a new metrics system is proposed (HADOOP-6728) targeting at 0.23 that has been in trunk. We probably will base on this to develop our own package, and integrate with other mechanics such as heartbeat, etc., which is underway.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.3.0
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063623#comment-13063623 ]
Edward J. Yoon commented on HAMA-363:
-------------------------------------
Is this patch ready to commit?
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "ChiaHung Lin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063815#comment-13063815 ]
ChiaHung Lin commented on HAMA-363:
-----------------------------------
My test on my current vms looks work.
But after reading parts of the paper[1], I discover the current implementation lacks of features such as federation, data transport, etc. The federation can provide aggregative information so that e.g. BSPMaster can proceed further by deciding if a worker fails, which is required by monitor service. In addition, plugin part may probably need to be improved as well, though at the moment from the developer's viewpoint it looks ok.
[1]. The ganglia distributed monitoring system: design, implementation, and experience. http://ganglia.info/papers/science.pdf
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "ChiaHung Lin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ChiaHung Lin updated HAMA-363:
------------------------------
Attachment: HAMA-363.patch
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "Edward J. Yoon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051942#comment-13051942 ]
Edward J. Yoon commented on HAMA-363:
-------------------------------------
Looks great. Should we add this 0.3?
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158905#comment-13158905 ]
Hudson commented on HAMA-363:
-----------------------------
Integrated in Hama-Nightly #370 (See [https://builds.apache.org/job/Hama-Nightly/370/])
[HAMA-363] Add network condition monitoring function to BSPMaster
chl501 :
Files :
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/AbstractPatternFilter.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/DefaultMetricsSystem.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/GlobalFilter.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/JvmMetrics.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/Metric.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/MetricsConfig.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/MetricsFactory.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/MetricsFilter.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/MetricsInfo.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/MetricsRecord.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/MetricsSink.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/MetricsSinkAdaptor.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/MetricsSource.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/MetricsSourceAdaptor.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/MetricsSystem.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/MetricsTag.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/Pair.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/RegexFilter.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/SystemMetrics.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/metrics/SystemMonitorSink.java
* /incubator/hama/trunk/core/src/main/java/org/apache/hama/util/ReflectionUtils.java
* /incubator/hama/trunk/core/src/test/java/org/apache/hama/metrics
* /incubator/hama/trunk/core/src/test/java/org/apache/hama/metrics/TestMetricsConfig.java
* /incubator/hama/trunk/core/src/test/java/org/apache/hama/metrics/TestMetricsSystem.java
* /incubator/hama/trunk/core/src/test/java/org/apache/hama/metrics/TestPatternFilter.java
* /incubator/hama/trunk/core/src/test/org
* /incubator/hama/trunk/core/src/test/org/apache
* /incubator/hama/trunk/core/src/test/resources/hama-metrics-msys.properties
* /incubator/hama/trunk/core/src/test/resources/hama-metrics-test-config.properties
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Assignee: ChiaHung Lin
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by Steve Loughran <st...@apache.org>.
On 09/05/11 22:02, Thomas Jungblut (JIRA) wrote:
>
> [ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030895#comment-13030895 ]
>
> Thomas Jungblut commented on HAMA-363:
> --------------------------------------
>
> As far as I know Hadoop only provides some JVM metrics and host metrices. I don't exactly find the correct source code position, but I think we should implement our own metrics package, which we can later add to ganglia. This is much more useful.
>
> We should define things we need to determine whether there are problems or not.
> Something like: "We ping every groom every 5 seconds and check the latency."
> This can be easily implemented in BSPMaster.
>
> To measure the IN and OUT rate or other fancy stuff we need something like heartbeat communication that will transfer the local groom data to the master.
> This should be in the newer versions of Hadoop>0.21 shouldn't it? Don't have the source codes haging around here.
>
if you are doing perf stuff, I'd go for having some plugin monitoring
that can go in before/after communications.
Why? I'm playing with sFlow monitoring of bits of Hadoop, and it's
tricky to retrofit this stuff deep into the code. If the hooks where
there it's easier.
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "Thomas Jungblut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030895#comment-13030895 ]
Thomas Jungblut commented on HAMA-363:
--------------------------------------
As far as I know Hadoop only provides some JVM metrics and host metrices. I don't exactly find the correct source code position, but I think we should implement our own metrics package, which we can later add to ganglia. This is much more useful.
We should define things we need to determine whether there are problems or not.
Something like: "We ping every groom every 5 seconds and check the latency."
This can be easily implemented in BSPMaster.
To measure the IN and OUT rate or other fancy stuff we need something like heartbeat communication that will transfer the local groom data to the master.
This should be in the newer versions of Hadoop >0.21 shouldn't it? Don't have the source codes haging around here.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.3.0
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "ChiaHung Lin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ChiaHung Lin updated HAMA-363:
------------------------------
Attachment: HAMA-363.patch
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "Thomas Jungblut (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157420#comment-13157420 ]
Thomas Jungblut commented on HAMA-363:
--------------------------------------
Hey ChiaHung, I know you are quite busy, but should I commit the patch?
There seems to be no dependency to our current trumk.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HAMA-363) Add network condition monitoring
function to BSPMaster
Posted by "Edward J. Yoon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HAMA-363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13158170#comment-13158170 ]
Edward J. Yoon commented on HAMA-363:
-------------------------------------
Please commit and close this issue.
Let's try to release 0.4 before the end of this month.
> Add network condition monitoring function to BSPMaster
> ------------------------------------------------------
>
> Key: HAMA-363
> URL: https://issues.apache.org/jira/browse/HAMA-363
> Project: Hama
> Issue Type: New Feature
> Components: bsp
> Affects Versions: 0.3.0
> Reporter: Edward J. Yoon
> Fix For: 0.4.0
>
> Attachments: HAMA-363.patch, HAMA-363.patch, HAMA-363.patch
>
>
> There's few reason why this issue is important. Basically Master server should know the status of cluster:
> * to optimize the network usage
> * to handle network connectivity problems
> * to handle different network conditions
> And, I would like to see some network usage statistics.
> In this issue, we implement only basic function which is collect network usage.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira