You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@chukwa.apache.org by "Ari Rabkin (JIRA)" <ji...@apache.org> on 2009/07/23 08:37:15 UTC
[jira] Created: (CHUKWA-358) pluggable real-time monitoring at
collectors
pluggable real-time monitoring at collectors
--------------------------------------------
Key: CHUKWA-358
URL: https://issues.apache.org/jira/browse/CHUKWA-358
Project: Hadoop Chukwa
Issue Type: New Feature
Components: data collection
Reporter: Ari Rabkin
Assignee: Ari Rabkin
Fix For: 0.3.0
Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-358:
------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed.
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch, improvedRealtime.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734885#action_12734885 ]
Ari Rabkin commented on CHUKWA-358:
-----------------------------------
For now, this was about the most minimal possible protocol. RPC seemed like overkill for this sort of thing. And we really don't want to force clients to use Thrift. It should be possible to integrate Thrift or Avro into this sort of mechanism if the need arises.
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737200#action_12737200 ]
Ari Rabkin commented on CHUKWA-358:
-----------------------------------
Previous model was using Jetty threads to push out real-time notification. This is bad, because it means that broken sockets down in the realtime mechanism can gum up the rest of the collector. I'm now introducing a queue and a separate thread per realtime channel.
Note that if you don't use the SocketTeeWriter, none of this applies. :)
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch, improvedRealtime.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin reopened CHUKWA-358:
-------------------------------
Turns out the previous thread model was too simplistic.
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch, improvedRealtime.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734826#action_12734826 ]
Jerome Boulon commented on CHUKWA-358:
--------------------------------------
Just my 0.02 c
It's inline with one proposal I have at Yahoo! to integrate Chukwa with Yahoo! Real time monitoring stack and building that behind the ChukwaWriter interface it's a good think.
So it's definitively a +1 on the idea however instead of reimplementing yet another protocol I will prefer to leverage something like Thrift for now and may be Avro when the RPC stack will be there. The reason being that Thrift is well debugged and provide support a large number of language.
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-358:
------------------------------
Resolution: Fixed
Release Note: Support for real-time monitoring at collector.
Status: Resolved (was: Patch Available)
I just committed this, with some modifications to improve robustness and flexibility.
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-358:
------------------------------
Status: Patch Available (was: Open)
Developed in response to operational needs here at Berkeley -- but should be useful more broadly.
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Eric Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734860#action_12734860 ]
Eric Yang commented on CHUKWA-358:
----------------------------------
+1 Looks good.
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-358:
------------------------------
Status: Patch Available (was: Reopened)
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch, improvedRealtime.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Jerome Boulon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737195#action_12737195 ]
Jerome Boulon commented on CHUKWA-358:
--------------------------------------
Ari could you explain what was too simplistic?
Thanks.
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch, improvedRealtime.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737297#action_12737297 ]
Ari Rabkin commented on CHUKWA-358:
-----------------------------------
barring objections, I'm going to commit this tonight or tomorrow.
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch, improvedRealtime.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-358:
------------------------------
Attachment: CHUKWA-358.patch
design is as follows:
create a new Pipelineable writer, that accepts filters via a very simple socket API. One filter per socket. It then applies all active filters to chunks as they go past; if they match, chunks are written back across the socket.
Filters expire when the associated socket is closed.
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Jiaqi Tan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734486#action_12734486 ]
Jiaqi Tan commented on CHUKWA-358:
----------------------------------
Would this be effectively creating a fork at the collector, one allowing the current path for writing to HDFS, the other allowing a real-time query?
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ari Rabkin updated CHUKWA-358:
------------------------------
Attachment: improvedRealtime.patch
Patch includes modified tests, including small fix for TestCmd.
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch, improvedRealtime.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CHUKWA-358) pluggable real-time monitoring at
collectors
Posted by "Ari Rabkin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CHUKWA-358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12734490#action_12734490 ]
Ari Rabkin commented on CHUKWA-358:
-----------------------------------
Yes, that's a fair summary of the intent.
> pluggable real-time monitoring at collectors
> --------------------------------------------
>
> Key: CHUKWA-358
> URL: https://issues.apache.org/jira/browse/CHUKWA-358
> Project: Hadoop Chukwa
> Issue Type: New Feature
> Components: data collection
> Reporter: Ari Rabkin
> Assignee: Ari Rabkin
> Fix For: 0.3.0
>
> Attachments: CHUKWA-358.patch
>
>
> Collectors could allow real-time data monitoring
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.