You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "John George (Created) (JIRA)" <ji...@apache.org> on 2011/11/15 00:23:52 UTC
[jira] [Created] (HADOOP-7821) Hadoop Messaging Service
Hadoop Messaging Service
------------------------
Key: HADOOP-7821
URL: https://issues.apache.org/jira/browse/HADOOP-7821
Project: Hadoop Common
Issue Type: New Feature
Affects Versions: 0.24.0
Reporter: John George
It will be good if Hadoop supports some sort of a messaging service that allows users to subscribe.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7821) Hadoop Messaging Service
Posted by "Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151079#comment-13151079 ]
Aaron T. Myers commented on HADOOP-7821:
----------------------------------------
I had a similar reaction to Scott when I first encountered this JIRA. I think the title of this JIRA is a little misleading. Perhaps something like "Hadoop event notification system" ?
> Hadoop Messaging Service
> ------------------------
>
> Key: HADOOP-7821
> URL: https://issues.apache.org/jira/browse/HADOOP-7821
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.24.0
> Reporter: John George
>
> It will be good if Hadoop supports some sort of a messaging service that allows users to subscribe.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7821) Hadoop Messaging Service
Posted by "Scott Carey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151004#comment-13151004 ]
Scott Carey commented on HADOOP-7821:
-------------------------------------
IMO Hadoop should not re-invent the wheel. There are traditional pub/sub frameworks, as well as some new distributed ones.
HedWig:
http://wiki.apache.org/hadoop/HedWig
Kafka:
http://incubator.apache.org/kafka/
http://sna-projects.com/blog/2011/08/kafka/
Or maybe something else... the above two are just those I've noted in the last couple months.
> Hadoop Messaging Service
> ------------------------
>
> Key: HADOOP-7821
> URL: https://issues.apache.org/jira/browse/HADOOP-7821
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.24.0
> Reporter: John George
>
> It will be good if Hadoop supports some sort of a messaging service that allows users to subscribe.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7821) Hadoop Messaging Service
Posted by "Mahadev konar (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151006#comment-13151006 ]
Mahadev konar commented on HADOOP-7821:
---------------------------------------
@Scott,
The idea is not to build a messagging system but to be able to publish a message from the MR Job on job completion. The intention is to make the client pluggable to be able to use any of the messaging system (using jms?).
> Hadoop Messaging Service
> ------------------------
>
> Key: HADOOP-7821
> URL: https://issues.apache.org/jira/browse/HADOOP-7821
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.24.0
> Reporter: John George
>
> It will be good if Hadoop supports some sort of a messaging service that allows users to subscribe.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7821) Hadoop event notification system
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151319#comment-13151319 ]
Steve Loughran commented on HADOOP-7821:
----------------------------------------
"HENS" is good; makes for some good artwork. Tom White might appreciate these
http://www.flickr.com/photos/steve_l/5799326089/in/set-72157626966433556/
> Hadoop event notification system
> --------------------------------
>
> Key: HADOOP-7821
> URL: https://issues.apache.org/jira/browse/HADOOP-7821
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.24.0
> Reporter: John George
>
> It will be good if Hadoop supports some sort of a messaging service that allows users to subscribe.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7821) Hadoop Messaging Service
Posted by "John George (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150072#comment-13150072 ]
John George commented on HADOOP-7821:
-------------------------------------
There has been several JIRAs that talk about 'messaging type' service that could be utilized by Hadoop clients.
- HDFS-1742 talks about a system that could notify interested clients about major HDFS events (like file creation, deletion, etc).
- HADOOP-5640 talks about a system that could notify clients on certain Namenode events (like entering safemode, exiting safemode etc).
- OOZIE-599 talks about Hadoop supporting JMS based (eg: Apache ActiveMQ) messaging service as an alternative to obtain 'call-back' URLs.
- Some other examples that could utilize a messaging service are:
-- events/states/status specific to an application (eg: mapper percentage complete)
-- events/states/status common to any application using YARN
Some high level requirements for creating such a messaging service:
- Provide timely (and/or periodic) state/status/statistics updates to subscribed users.
- Provide the capability to let users specify 'interesting events'.
- Provide a 'customizable' and 'configurable' interface such that user-defined parameters can also be 'subscribed' by the user.
- Provide a pluggable security interface.
Such a messaging service will not only reduce the load on Hadoop
components like NN (reduced polling by clients), but also increase the
productivity of developers using hadoop.
Of course, there are challenges as well and this JIRA is filed to provide a platform for discussion about this feature and possibly act as a parent JIRA to implement a Hadoop Messaging Service.
> Hadoop Messaging Service
> ------------------------
>
> Key: HADOOP-7821
> URL: https://issues.apache.org/jira/browse/HADOOP-7821
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.24.0
> Reporter: John George
>
> It will be good if Hadoop supports some sort of a messaging service that allows users to subscribe.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HADOOP-7821) Hadoop Messaging Service
Posted by "Steve Loughran (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151125#comment-13151125 ]
Steve Loughran commented on HADOOP-7821:
----------------------------------------
I'd go for some plugin points and see how they get used. Distributed messaging services are hard to get right or scale -and problems that other projects have worked on.
while an in-VM event mechanism would be tractable, especially if the events were queued to decouple event from plugin notification from day one. Then I could do a bridge to push events into our partition-aware T-space; other people can push them into ActiveMQ, Kafka or anything else.
> Hadoop Messaging Service
> ------------------------
>
> Key: HADOOP-7821
> URL: https://issues.apache.org/jira/browse/HADOOP-7821
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.24.0
> Reporter: John George
>
> It will be good if Hadoop supports some sort of a messaging service that allows users to subscribe.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HADOOP-7821) Hadoop event notification system
Posted by "John George (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-7821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
John George updated HADOOP-7821:
--------------------------------
Summary: Hadoop event notification system (was: Hadoop Messaging Service)
Changing the title to what the JIRA really stands for
> Hadoop event notification system
> --------------------------------
>
> Key: HADOOP-7821
> URL: https://issues.apache.org/jira/browse/HADOOP-7821
> Project: Hadoop Common
> Issue Type: New Feature
> Affects Versions: 0.24.0
> Reporter: John George
>
> It will be good if Hadoop supports some sort of a messaging service that allows users to subscribe.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira