You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Santhosh Thomas <sa...@yahoo.com> on 2014/05/20 09:46:20 UTC
How to detect new table creation in hive
Hello
I am writing an application that detects new table creation in hive.
I came across attribute ' hive.metastore.event.listeners' in hive-default.xml, but could not find any usage samples.
The API docs also does not describe much. Can someone please help? Is there a better way to detect new table creation in hive?
thanks
Santhosh
Re: How to detect new table creation in hive
Posted by Santhosh Thomas <sa...@yahoo.com>.
For the benefit of others:
I was able to capture the CreateTable and DropTable events.
Got the sample NotificationListener from hcatalog/server-extensions/src/main/java/org/apache/hcatalog/listener/NotificationListener.java - hive - Git at Google , removed unwanted stuff, packaged into a jar and put the jar into /usr/lib/hive/lib.
Now restart hive-metastore and hive-server2, changes will take effect (ofcouse hive-site.xml needs an entry for the listener). Any error you can see in /var/log/hive/hive-metastore.log
This link has some code samples:http://programcreek.com/java-api-examples/index.php?api=org.apache.hadoop.hive.metastore.api.Table
thanks
Santhosh
hcatalog/server-extensions/src/main/java/org/apache/hcatalog/listener/NotificationListener.ja...
Git Sign in apache / hive / 01ca40833d06c035160d8188c116c11981b1d879 / . / hcatalog / server-extensions / src / main / java / org / apache / hcatalog / listener / NotificationListener.java
View on apache.googlesource.com Preview by Yahoo
From: Sunil N Kumar <su...@impetus.co.in>
To: Santhosh Thomas <sa...@yahoo.com>; "user@hive.apache.org" <us...@hive.apache.org>
Cc: Raptors <St...@impetus.co.in>
Sent: Tuesday, May 20, 2014 2:59 PM
Subject: RE: How to detect new table creation in hive
Hi Santosh,
This class already part of the Hive version 0.12 onwards in following location.
$HIVE_HOME/hcatalog/share/hcatalog/hcatalog-server-extensions-0.12.0.jar
Just make sure you run Hcatalog server instead of hive metastore.
Thanks and Regard.
Sunil Kumar
From:Santhosh Thomas [mailto:santhosh.thomas@yahoo.com]
Sent: Tuesday, May 20, 2014 2:43 PM
To: Sunil N Kumar; user@hive.apache.org
Cc: Raptors
Subject: Re: How to detect new table creation in hive
Thanks Sunil. I am already close to it. Where did you put the NotificationListener.class?
~santhosh
From:Sunil N Kumar <su...@impetus.co.in>
To: "user@hive.apache.org" <us...@hive.apache.org>; Santhosh Thomas <sa...@yahoo.com>
Cc: Raptors <St...@impetus.co.in>
Sent: Tuesday, May 20, 2014 2:22 PM
Subject: RE: How to detect new table creation in hive
Hi Santhosh,
I have used this in the Falcon for creating Hive process to manage Hive tables across multiple Hadoop cluster. Current Hcatalog publish an event to the Message Broker say Active MQ whenever new partition added or deleted in the Hive Metastore. Following setting needs to done at hive-site.xml. This is not documented in the Hive documnetion. But I have used this and it is working as expected. It create topic in the activeMQ as
Hcat.<Databasename>.<tablename>.
<property>
<name>hive.metastore.event.listeners</name>
<value>org.apache.hcatalog.listener.NotificationListener</value>
<description></description>
</property>
https://cwiki.apache.org/confluence/display/Hive/HCatalog+Notification
HCatalog provides notifications for certain events happening in the system. This way applications such as Oozie can wait for those events and schedule the work that depends on them. The current version of HCatalog supports two kinds of events:
· Notification when a new partition is added
· Notification when a set of partitions is added
From:Santhosh Thomas [mailto:santhosh.thomas@yahoo.com]
Sent: Tuesday, May 20, 2014 1:16 PM
To: user@hive.apache.org
Subject: How to detect new table creation in hive
Hello
I am writing an application that detects new table creation in hive.
I came across attribute ' hive.metastore.event.listeners' in hive-default.xml, but could not find any usage samples.
The API docs also does not describe much. Can someone please help? Is there a better way to detect new table creation in hive?
thanks
Santhosh
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email
is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email
is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: How to detect new table creation in hive
Posted by Sunil N Kumar <su...@impetus.co.in>.
Hi Santosh,
This class already part of the Hive version 0.12 onwards in following location.
$HIVE_HOME/hcatalog/share/hcatalog/hcatalog-server-extensions-0.12.0.jar
Just make sure you run Hcatalog server instead of hive metastore.
Thanks and Regard.
Sunil Kumar
From: Santhosh Thomas [mailto:santhosh.thomas@yahoo.com]
Sent: Tuesday, May 20, 2014 2:43 PM
To: Sunil N Kumar; user@hive.apache.org
Cc: Raptors
Subject: Re: How to detect new table creation in hive
Thanks Sunil. I am already close to it. Where did you put the NotificationListener.class?
~santhosh
From: Sunil N Kumar <su...@impetus.co.in>>
To: "user@hive.apache.org<ma...@hive.apache.org>" <us...@hive.apache.org>>; Santhosh Thomas <sa...@yahoo.com>>
Cc: Raptors <St...@impetus.co.in>>
Sent: Tuesday, May 20, 2014 2:22 PM
Subject: RE: How to detect new table creation in hive
Hi Santhosh,
I have used this in the Falcon for creating Hive process to manage Hive tables across multiple Hadoop cluster. Current Hcatalog publish an event to the Message Broker say Active MQ whenever new partition added or deleted in the Hive Metastore. Following setting needs to done at hive-site.xml. This is not documented in the Hive documnetion. But I have used this and it is working as expected. It create topic in the activeMQ as
Hcat.<Databasename>.<tablename>.
<property>
<name>hive.metastore.event.listeners</name>
<value>org.apache.hcatalog.listener.NotificationListener</value>
<description></description>
</property>
https://cwiki.apache.org/confluence/display/Hive/HCatalog+Notification
HCatalog provides notifications for certain events happening in the system. This way applications such as Oozie can wait for those events and schedule the work that depends on them. The current version of HCatalog supports two kinds of events:
* Notification when a new partition is added
* Notification when a set of partitions is added
From: Santhosh Thomas [mailto:santhosh.thomas@yahoo.com]
Sent: Tuesday, May 20, 2014 1:16 PM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: How to detect new table creation in hive
Hello
I am writing an application that detects new table creation in hive.
I came across attribute ' hive.metastore.event.listeners' in hive-default.xml, but could not find any usage samples.
The API docs also does not describe much. Can someone please help? Is there a better way to detect new table creation in hive?
thanks
Santhosh
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
Re: How to detect new table creation in hive
Posted by Santhosh Thomas <sa...@yahoo.com>.
Thanks Sunil. I am already close to it. Where did you put the NotificationListener.class?
~santhosh
From: Sunil N Kumar <su...@impetus.co.in>
To: "user@hive.apache.org" <us...@hive.apache.org>; Santhosh Thomas <sa...@yahoo.com>
Cc: Raptors <St...@impetus.co.in>
Sent: Tuesday, May 20, 2014 2:22 PM
Subject: RE: How to detect new table creation in hive
Hi Santhosh,
I have used this in the Falcon for creating Hive process to manage Hive tables across multiple Hadoop cluster. Current Hcatalog publish an event to the Message Broker say Active MQ whenever new partition added or deleted in the Hive Metastore. Following setting needs to done at hive-site.xml. This is not documented in the Hive documnetion. But I have used this and it is working as expected. It create topic in the activeMQ as
Hcat.<Databasename>.<tablename>.
<property>
<name>hive.metastore.event.listeners</name>
<value>org.apache.hcatalog.listener.NotificationListener</value>
<description></description>
</property>
https://cwiki.apache.org/confluence/display/Hive/HCatalog+Notification
HCatalog provides notifications for certain events happening in the system. This way applications such as Oozie can wait for those events and schedule the work that depends on them. The current version of HCatalog supports two kinds of events:
· Notification when a new partition is added
· Notification when a set of partitions is added
From:Santhosh Thomas [mailto:santhosh.thomas@yahoo.com]
Sent: Tuesday, May 20, 2014 1:16 PM
To: user@hive.apache.org
Subject: How to detect new table creation in hive
Hello
I am writing an application that detects new table creation in hive.
I came across attribute ' hive.metastore.event.listeners' in hive-default.xml, but could not find any usage samples.
The API docs also does not describe much. Can someone please help? Is there a better way to detect new table creation in hive?
thanks
Santhosh
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email
is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.
RE: How to detect new table creation in hive
Posted by Sunil N Kumar <su...@impetus.co.in>.
Hi Santhosh,
I have used this in the Falcon for creating Hive process to manage Hive tables across multiple Hadoop cluster. Current Hcatalog publish an event to the Message Broker say Active MQ whenever new partition added or deleted in the Hive Metastore. Following setting needs to done at hive-site.xml. This is not documented in the Hive documnetion. But I have used this and it is working as expected. It create topic in the activeMQ as
Hcat.<Databasename>.<tablename>.
<property>
<name>hive.metastore.event.listeners</name>
<value>org.apache.hcatalog.listener.NotificationListener</value>
<description></description>
</property>
https://cwiki.apache.org/confluence/display/Hive/HCatalog+Notification
HCatalog provides notifications for certain events happening in the system. This way applications such as Oozie can wait for those events and schedule the work that depends on them. The current version of HCatalog supports two kinds of events:
* Notification when a new partition is added
* Notification when a set of partitions is added
From: Santhosh Thomas [mailto:santhosh.thomas@yahoo.com]
Sent: Tuesday, May 20, 2014 1:16 PM
To: user@hive.apache.org
Subject: How to detect new table creation in hive
Hello
I am writing an application that detects new table creation in hive.
I came across attribute ' hive.metastore.event.listeners' in hive-default.xml, but could not find any usage samples.
The API docs also does not describe much. Can someone please help? Is there a better way to detect new table creation in hive?
thanks
Santhosh
________________________________
NOTE: This message may contain information that is confidential, proprietary, privileged or otherwise protected by law. The message is intended solely for the named addressee. If received in error, please destroy and notify the sender. Any use of this email is prohibited when received in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this communication has been maintained nor that the communication is free of errors, virus, interception or interference.