You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Eric Yang (JIRA)" <ji...@apache.org> on 2009/12/30 01:02:29 UTC
[jira] Created: (CHUKWA-441) Real time hadoop activity monitoring
capabilities via Client Trace Logs
Real time hadoop activity monitoring capabilities via Client Trace Logs
-----------------------------------------------------------------------
Key: CHUKWA-441
URL: https://issues.apache.org/jira/browse/CHUKWA-441
Project: Hadoop Chukwa
Issue Type: New Feature
Components: data collection, User Interface
Environment: Redhat EL 5.1, Java 6
Reporter: Eric Yang
Assignee: Eric Yang
Fix For: 0.4.0
Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.
The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:
Adaptor -> Agent -> Collector -> SocketTeeWriter <- Socket Data Loader <- HICC
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-441) Real time hadoop activity monitoring
capabilities via Client Trace Logs
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-441:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this, thanks Ari.
> Real time hadoop activity monitoring capabilities via Client Trace Logs
> -----------------------------------------------------------------------
>
> Key: CHUKWA-441
> URL: https://issues.apache.org/jira/browse/CHUKWA-441
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, User Interface
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Fix For: 0.4.0
>
> Attachments: CHUKWA-441.patch, Picture 1.png
>
>
> Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.
> The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:
> Adaptor -> Agent -> Collector -> SocketTeeWriter <- Socket Data Loader <- HICC
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-441) Real time hadoop activity monitoring
capabilities via Client Trace Logs
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-441:
-----------------------------
Attachment: CHUKWA-441.patch
- Implemented Socket Data Loader, ClientTrace REST API, and Hadoop Activity widget.
- Test cases included.
> Real time hadoop activity monitoring capabilities via Client Trace Logs
> -----------------------------------------------------------------------
>
> Key: CHUKWA-441
> URL: https://issues.apache.org/jira/browse/CHUKWA-441
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, User Interface
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Fix For: 0.4.0
>
> Attachments: CHUKWA-441.patch
>
>
> Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.
> The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:
> Adaptor -> Agent -> Collector -> SocketTeeWriter <- Socket Data Loader <- HICC
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (CHUKWA-441) Real time hadoop activity monitoring
capabilities via Client Trace Logs
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang reopened CHUKWA-441:
------------------------------
etting this error message trying to compile trunk (r894877):
[javac] /home/jiaqi/code/chukwa/latest3/trunk/src/java/org/apache/hadoop/chukwa/rest/resource/ClientTrace.java:39:
cannot find symbol
[javac] symbol : class EventBean
[javac] location: package org.apache.hadoop.chukwa.rest.bean
[javac] import org.apache.hadoop.chukwa.rest.bean.EventBean;
Unused import statement, should get clean up.
> Real time hadoop activity monitoring capabilities via Client Trace Logs
> -----------------------------------------------------------------------
>
> Key: CHUKWA-441
> URL: https://issues.apache.org/jira/browse/CHUKWA-441
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, User Interface
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Fix For: 0.4.0
>
> Attachments: CHUKWA-441.patch, Picture 1.png
>
>
> Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.
> The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:
> Adaptor -> Agent -> Collector -> SocketTeeWriter <- Socket Data Loader <- HICC
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-441) Real time hadoop activity monitoring
capabilities via Client Trace Logs
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795995#action_12795995 ]
Hudson commented on CHUKWA-441:
-------------------------------
Integrated in Chukwa-trunk #247 (See [http://hudson.zones.apache.org/hudson/job/Chukwa-trunk/247/])
. Fixed compile error.
> Real time hadoop activity monitoring capabilities via Client Trace Logs
> -----------------------------------------------------------------------
>
> Key: CHUKWA-441
> URL: https://issues.apache.org/jira/browse/CHUKWA-441
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, User Interface
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Fix For: 0.4.0
>
> Attachments: CHUKWA-441-1.patch, CHUKWA-441.patch, Picture 1.png
>
>
> Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.
> The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:
> Adaptor -> Agent -> Collector -> SocketTeeWriter <- Socket Data Loader <- HICC
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-441) Real time hadoop activity monitoring
capabilities via Client Trace Logs
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-441:
-----------------------------
Attachment: CHUKWA-441-1.patch
Remove unused EventBean.
> Real time hadoop activity monitoring capabilities via Client Trace Logs
> -----------------------------------------------------------------------
>
> Key: CHUKWA-441
> URL: https://issues.apache.org/jira/browse/CHUKWA-441
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, User Interface
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Fix For: 0.4.0
>
> Attachments: CHUKWA-441-1.patch, CHUKWA-441.patch, Picture 1.png
>
>
> Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.
> The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:
> Adaptor -> Agent -> Collector -> SocketTeeWriter <- Socket Data Loader <- HICC
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-441) Real time hadoop activity monitoring
capabilities via Client Trace Logs
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-441:
-----------------------------
Status: Patch Available (was: Open)
Real time hadoop activity monitor. This feature only works for single collector, to make this work on large scale cluster, the data needs to be channeled to a single collector.
> Real time hadoop activity monitoring capabilities via Client Trace Logs
> -----------------------------------------------------------------------
>
> Key: CHUKWA-441
> URL: https://issues.apache.org/jira/browse/CHUKWA-441
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, User Interface
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Fix For: 0.4.0
>
> Attachments: CHUKWA-441.patch, Picture 1.png
>
>
> Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.
> The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:
> Adaptor -> Agent -> Collector -> SocketTeeWriter <- Socket Data Loader <- HICC
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-441) Real time hadoop activity monitoring
capabilities via Client Trace Logs
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-441:
-----------------------------
Attachment: Picture 1.png
Screenshot of the activity monitor.
> Real time hadoop activity monitoring capabilities via Client Trace Logs
> -----------------------------------------------------------------------
>
> Key: CHUKWA-441
> URL: https://issues.apache.org/jira/browse/CHUKWA-441
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, User Interface
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Fix For: 0.4.0
>
> Attachments: CHUKWA-441.patch, Picture 1.png
>
>
> Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.
> The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:
> Adaptor -> Agent -> Collector -> SocketTeeWriter <- Socket Data Loader <- HICC
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-441) Real time hadoop activity monitoring
capabilities via Client Trace Logs
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-441:
-----------------------------
Status: Patch Available (was: Reopened)
Clean up compile error.
> Real time hadoop activity monitoring capabilities via Client Trace Logs
> -----------------------------------------------------------------------
>
> Key: CHUKWA-441
> URL: https://issues.apache.org/jira/browse/CHUKWA-441
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, User Interface
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Fix For: 0.4.0
>
> Attachments: CHUKWA-441-1.patch, CHUKWA-441.patch, Picture 1.png
>
>
> Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.
> The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:
> Adaptor -> Agent -> Collector -> SocketTeeWriter <- Socket Data Loader <- HICC
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-441) Real time hadoop activity monitoring
capabilities via Client Trace Logs
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12795218#action_12795218 ]
Ari Rabkin commented on CHUKWA-441:
-----------------------------------
+1
> Real time hadoop activity monitoring capabilities via Client Trace Logs
> -----------------------------------------------------------------------
>
> Key: CHUKWA-441
> URL: https://issues.apache.org/jira/browse/CHUKWA-441
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, User Interface
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Fix For: 0.4.0
>
> Attachments: CHUKWA-441.patch, Picture 1.png
>
>
> Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.
> The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:
> Adaptor -> Agent -> Collector -> SocketTeeWriter <- Socket Data Loader <- HICC
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-441) Real time hadoop activity monitoring
capabilities via Client Trace Logs
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Yang updated CHUKWA-441:
-----------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Resolved compile error.
> Real time hadoop activity monitoring capabilities via Client Trace Logs
> -----------------------------------------------------------------------
>
> Key: CHUKWA-441
> URL: https://issues.apache.org/jira/browse/CHUKWA-441
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection, User Interface
> Environment: Redhat EL 5.1, Java 6
> Reporter: Eric Yang
> Assignee: Eric Yang
> Fix For: 0.4.0
>
> Attachments: CHUKWA-441-1.patch, CHUKWA-441.patch, Picture 1.png
>
>
> Many moons ago, in one of the HUG talk for Chukwa, one user ask if it is possible to create a splitter interface for Chukwa Collector for monitoring data at near real time. The answer was possible, but there was no code to show this capabilities. Over the Xmas holidays, I decided to take on this task and write something simple to show how this is done.
> The idea is to reuse existing Chukwa Collector with PipelineStageWriter coupled with SocketTeeWriter for data stream split. By adding a SocketDataLoader inside HICC to act as data stream subscriber and extract client trace information to render on the browser. Hence, the additional pipeline looks like this:
> Adaptor -> Agent -> Collector -> SocketTeeWriter <- Socket Data Loader <- HICC
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.