You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hcatalog-commits@incubator.apache.org by "Ashutosh Chauhan (JIRA)" <ji...@apache.org> on 2011/04/12 23:35:05 UTC
[jira] [Created] (HCATALOG-3) Send a message on a message bus when
events occur in Metastore
Send a message on a message bus when events occur in Metastore
---------------------------------------------------------------
Key: HCATALOG-3
URL: https://issues.apache.org/jira/browse/HCATALOG-3
Project: HCatalog
Issue Type: New Feature
Reporter: Ashutosh Chauhan
By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HCATALOG-3) Send a message on a message bus
when events occur in Metastore
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035113#comment-13035113 ]
Jakob Homan commented on HCATALOG-3:
------------------------------------
The activemq inclusion, particularly just for testing, concerns me as well. Eventually, it would be great to refactor the message sending away from JMS so that those with other messaging solutions (*cough* http://sna-projects.com/kafka/ *cough*) could drop those in. It appears the current WIP patch is quite WIP - HCatListener is overriding quite a few methods that aren't defined in the hierarchy. I'm also concerned about the use of the finalize method, which is generally agreed to be evil and unreliable. Also, it would be good to think about dependencies such that we don't need to fire up whole metastores for these tests and can instead use mocking, to avoid long pauses and slow tests.
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: hcat_3.patch, hcat_3_1.patch, hcat_3_1.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HCATALOG-3) Send a message on a message bus when
events occur in Metastore
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HCATALOG-3:
------------------------------------
Attachment: hcat_3_1.patch
WIP. Checkpointing the patch.
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: hcat_3.patch, hcat_3_1.patch, hcat_3_1.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HCATALOG-3) Send a message on a message bus when
events occur in Metastore
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HCATALOG-3:
------------------------------------
Attachment: (was: NotificationSender.java)
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HCATALOG-3) Send a message on a message bus
when events occur in Metastore
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035534#comment-13035534 ]
Alan Gates commented on HCATALOG-3:
-----------------------------------
The reason for JMS was exactly to allow different users to drop different messaging solutions in, since we assume that many users will already have a messaging solution. The issues with solutions like Kafka are their lack of compliance with any messaging standard (please correct me if I'm wrong here and they are conforming to a standard, but I couldn't find one). It seems easier to write JMS adapters for these systems (only for the 1% of JMS that HCat uses, not the whole API) than it does to have HCat define it's own API and make everyone adapt to it.
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: hcat_3.patch, hcat_3_1.patch, hcat_3_1.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HCATALOG-3) Send a message on a message bus when
events occur in Metastore
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HCATALOG-3:
------------------------------------
Attachment: NotificationSender.java
WIP. This is for a checkpoint. Will submit patch friendly diffs once I am able to build hcat trunk.
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: NotificationSender.java
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HCATALOG-3) Send a message on a message bus when
events occur in Metastore
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HCATALOG-3:
------------------------------------
Attachment: hcat_3_1.patch
bq. 1. In the jndi.properties file you have place holders for registering queues and topcis. Do we plan to register any such queues topics by default?
No, its not required to register queues or topics upfront. In fact specifying them there won't really register topic. Defining them there provides a way for jndi to discover message bus destinations in most portable ways. So, if destination names are static and known ahead of time, this is the recommended way. Otherwise, they can be created on demand as well. I would leave them in jndi.properties as example for completeness.
bq. 2. Is finializePartition message what I've been calling "mark set done"? If yes, I don't see the string specifying the filter that specifies which partitions are done. If no, what is it and where is the "mark set done" message?
Ya, finalizePartition is mark-set-done. subscriber will set message selector string as "HCAT_FINALIZE_PARTITON" on topic named "mydb.mytbl" to get MSD messages. Filter string itself is in msgBody
New patch with a testcase, which apart from testing also demonstrate how subscribers can receive messages. Added activemq jar as dependency in ivy, which is required for the testcase.
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: hcat_3.patch, hcat_3_1.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HCATALOG-3) Send a message on a message bus
when events occur in Metastore
Posted by "John Sichi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13030177#comment-13030177 ]
John Sichi commented on HCATALOG-3:
-----------------------------------
Maybe there is something lighterweight that can be used for testing instead of activemq as a dependency?
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: hcat_3.patch, hcat_3_1.patch, hcat_3_1.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HCATALOG-3) Send a message on a message bus when
events occur in Metastore
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan resolved HCATALOG-3.
-------------------------------------
Resolution: Fixed
Patch committed to trunk.
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Fix For: 0.2
>
> Attachments: hcat_3.patch, hcat_3.patch, hcat_3_1.patch, hcat_3_1.patch, hcat_3_2.patch, hcat_3_2.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HCATALOG-3) Send a message on a message bus when
events occur in Metastore
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HCATALOG-3:
------------------------------------
Attachment: hcat_3.patch
Patch is ready for review.
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: hcat_3.patch, hcat_3.patch, hcat_3_1.patch, hcat_3_1.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HCATALOG-3) Send a message on a message bus
when events occur in Metastore
Posted by "Mac Yang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13039206#comment-13039206 ]
Mac Yang commented on HCATALOG-3:
---------------------------------
A couple suggestions,
- It would be easier to apply authorization policy if the topic names share common top level name space
- Use package.jdo instead of annotation for object mapping to be consistent with Hive
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: hcat_3.patch, hcat_3.patch, hcat_3_1.patch, hcat_3_1.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HCATALOG-3) Send a message on a message bus
when events occur in Metastore
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023300#comment-13023300 ]
Alan Gates commented on HCATALOG-3:
-----------------------------------
# In the jndi.properties file you have place holders for registering queues and topcis. Do we plan to register any such queues topics by default?
# Is finializePartition message what I've been calling "mark set done"? If yes, I don't see the string specifying the filter that specifies which partitions are done. If no, what is it and where is the "mark set done" message?
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: hcat_3.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HCATALOG-3) Send a message on a message bus when
events occur in Metastore
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HCATALOG-3:
------------------------------------
Attachment: hcat_3_2.patch
Previous patch was incomplete. Attaching a new patch.
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: hcat_3.patch, hcat_3.patch, hcat_3_1.patch, hcat_3_1.patch, hcat_3_2.patch, hcat_3_2.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HCATALOG-3) Send a message on a message bus when
events occur in Metastore
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HCATALOG-3:
------------------------------------
Fix Version/s: 0.2
Assignee: Ashutosh Chauhan
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Assignee: Ashutosh Chauhan
> Fix For: 0.2
>
> Attachments: hcat_3.patch, hcat_3.patch, hcat_3_1.patch, hcat_3_1.patch, hcat_3_2.patch, hcat_3_2.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HCATALOG-3) Send a message on a message bus
when events occur in Metastore
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035561#comment-13035561 ]
Ashutosh Chauhan commented on HCATALOG-3:
-----------------------------------------
@John
We need to have a JMS provider for testing. ActiveMQ message broker can be run in an embedded mode (which is what I am using in my testcases) and is pretty light-weight. It is pretty straight forward to use and doesnt add any latencies. If there is some other JMS Provider which can run in embedded mode and has proven to be light weight then ActiveMQ we can explore that one.
@Jakob,
1. Are you suggesting we get away from JMS interface altogether? Then Alan I think has valid points.
2. For overriding methods there are jiras open in Hive which as I have noted above needs to be committed first.
3. There is no dependency on finalize(). We run in transacted mode, so closing of connection is not really required. So, if its not run, nothing will be lost. I still have it just as a good practice. If you think it can be done in better way, I can try that out.
4. MetaStoreUtils.startMetaStore() is a better way to start metastore. That will help to avoid long pauses. Mocking will also help.
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: hcat_3.patch, hcat_3_1.patch, hcat_3_1.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HCATALOG-3) Send a message on a message bus
when events occur in Metastore
Posted by "Jakob Homan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13035628#comment-13035628 ]
Jakob Homan commented on HCATALOG-3:
------------------------------------
I'm not requesting to drop JMS, just to keep it cleanly integrated so (1) it would be easy to test via mocking or such to avoid the activemq inclusion and (2) so that later on, we could contribute a patch to make other systems, like Kafka, work. Messaging is general enough that it should be relatively straightforward to have a clean interface around it. Since I don't have a patch ready for something like supporting Kafka, I'm certainly not trying to block JMS.
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: hcat_3.patch, hcat_3_1.patch, hcat_3_1.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (HCATALOG-3) Send a message on a
message bus when events occur in Metastore
Posted by "Alan Gates (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13023300#comment-13023300 ]
Alan Gates edited comment on HCATALOG-3 at 4/22/11 5:52 PM:
------------------------------------------------------------
1. In the jndi.properties file you have place holders for registering queues and topcis. Do we plan to register any such queues topics by default?
2. Is finializePartition message what I've been calling "mark set done"? If yes, I don't see the string specifying the filter that specifies which partitions are done. If no, what is it and where is the "mark set done" message?
was (Author: alangates):
# In the jndi.properties file you have place holders for registering queues and topcis. Do we plan to register any such queues topics by default?
# Is finializePartition message what I've been calling "mark set done"? If yes, I don't see the string specifying the filter that specifies which partitions are done. If no, what is it and where is the "mark set done" message?
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: hcat_3.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HCATALOG-3) Send a message on a message bus when
events occur in Metastore
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HCATALOG-3:
------------------------------------
Attachment: hcat_3_2.patch
New patch addressing Mac's comments.
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: hcat_3.patch, hcat_3.patch, hcat_3_1.patch, hcat_3_1.patch, hcat_3_2.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HCATALOG-3) Send a message on a message bus when
events occur in Metastore
Posted by "Ashutosh Chauhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HCATALOG-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HCATALOG-3:
------------------------------------
Attachment: hcat_3.patch
This patch is ready for review. But can't be committed until HIVE-2038 is committed.
> Send a message on a message bus when events occur in Metastore
> ---------------------------------------------------------------
>
> Key: HCATALOG-3
> URL: https://issues.apache.org/jira/browse/HCATALOG-3
> Project: HCatalog
> Issue Type: New Feature
> Reporter: Ashutosh Chauhan
> Attachments: hcat_3.patch
>
>
> By implementing a MetaStoreEventListener interface.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira