You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Chetan Kothari <ch...@oracle.com> on 2018/08/21 12:57:25 UTC
Hook is not getting called on running query from beeline
Hi
I have written custom PostExecutionHook which implements org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext to capture lineage of hive queries like create table as select etc.
This hook get's called when I run CTAS query o Hive Shell. But it is not getting called when I run same query from beeline or programmatically using JDBC Code.
Any inputs on why hook is not getting called when I run query from beeline or programmatically using JDBC Code.
Regards
Chetan
RE: Hook is not getting called on running query from beeline
Posted by Chetan Kothari <ch...@oracle.com>.
Hi All
Any input on following issue will be useful.
Regards
Chetan
From: Chetan Kothari
Sent: Monday, August 27, 2018 1:48 PM
To: user@hive.apache.org
Subject: RE: Hook is not getting called on running query from beeline
Hi Zoltan
I changed the configuration to connect to Hive.
[test@bdctest-1 hive]$ beeline
WARNING: Use "yarn jar" to launch YARN applications.
Beeline version 1.2.1000.2.4.2.0-258 by Apache Hive
Connected to: Apache Hive (version 1.2.1000.2.4.2.0-258)
Driver: Hive JDBC (version 1.2.1000.2.4.2.0-258)
Transaction isolation: TRANSACTION_REPEATABLE_READ
When I give wrong custom hook name or if hook class is not in classpath, it gives me following error.
Error: Error while processing statement: FAILED: Hive Internal Error: java.lang.ClassNotFoundException(test.dc.lineage.listeners.hive.TestPostExecutionHook) (state=08S01,code=12)
add jar /u01/bdctest/usr/hdp/2.4.2.0-258/hive/lib/hook-1.0.jar;
set hive.exec.post.hooks=test.dc.lineage.listeners.hive.TestPostExecutionHook;
I fixed this issue by explicitly setting hook and adding jar files.
After setting hook and adding jar file, I don’t set above error, query gets executed but custom hook is not getting called.
Regards
Chetan
-----Original Message-----
From: Zoltan Haindrich [mailto:kirk@rxd.hu]
Sent: Thursday, August 23, 2018 1:14 PM
To: HYPERLINK "mailto:user@hive.apache.org"user@hive.apache.org; Chetan Kothari
Subject: Re: Hook is not getting called on running query from beeline
Hello
There is an interesting message after connecting: "Connected to: Spark SQL (version 2.1.0.2.6.0.3-8)".
It seems you are not using Hive...please ask about this on a Spark related list.
cheers,
Zoltan
On 08/22/2018 08:43 PM, Chetan Kothari wrote:
> It seems --hiveconf is not getting picked from beeline.
>
> So even if I give wrong hook class name in following command, I am able to connect and execute queries.
>
> But hook is not getting called.
>
> beeline -u
> "jdbc:hive2://bdcsceforqa-bdcsce-1.dfml.ucfc2z3b.usdv1.abccloud.com:10
> 001/;transportMode=http;httpPath=cliservice" --hiveconf
> hive.exec.post.hooks=abcc.lineage.listeners.TestPostExecutionHook
>
> Connected to: Spark SQL (version 2.1.0.2.6.0.3-8)
>
> Driver: Hive JDBC (version 1.2.1000.2.4.2.0-258)
>
> Transaction isolation: TRANSACTION_REPEATABLE_READ
>
> Beeline version 1.2.1000.2.4.2.0-258 by Apache Hive
>
> Regards
>
> Chetan
>
> -----Original Message-----
> From: Zoltan Haindrich [mailto:kirk@rxd.hu]
> Sent: Wednesday, August 22, 2018 8:51 PM
> To: HYPERLINK "mailto:user@hive.apache.org"user@hive.apache.org; Chetan Kothari
> Subject: Re: Hook is not getting called on running query from beeline
>
> Hello
>
> I was wondering about that the property is not getting picked up from beeline; but it looks ok to me:
>
> $ beeline -u 'jdbc:hive2://...' --hiveconf hive.exec.post.hooks=asd -e 'select 1'
>
> Connected to: Apache Hive (version 1.2.1000.2.6.1.0-129)
>
> Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129) Transaction
> isolation: TRANSACTION_REPEATABLE_READ
>
> Error: Error while processing statement: FAILED: Hive Internal Error:
> java.lang.ClassNotFoundException(asd) (state=08S01,code=12)
>
> Which version of hive are you using?
>
> Try first with a non-existent classname. Executing the hooks usually leave some messages in the logs as well...
>
> cheers,
>
> Zoltan
>
> On 08/22/2018 11:22 AM, Chetan Kothari wrote:
>
> > Thanks Tanvi for your inputs.
>
> >
>
> > Hook is not getting called from beeline/JDBC for any queries.
>
> >
>
> > I ensured that hive-site.xml is in classpath and also tried following but still hook is not getting called.
>
> >
>
> > beeline -u "jdbc:hive2://example.com:2181
>
> > <http://example.com:2181>,example.com:2181
>
> >
> <http://example.com:2181>,example:2181/;serviceDiscoveryMode=zooKeeper
>
> > ;zooKeeperNamespace=hiveserver2" *--hiveconf
>
> > hive.exec.post.hooks=<your_custom_hook_here>*
>
> >
>
> > Regards
>
> >
>
> > Chetan
>
> >
>
> > *From:*Tanvi Thacker [mailto:tanvithacker5@gmail.com]
>
> > *Sent:* Wednesday, August 22, 2018 9:29 AM
>
> > *To:* HYPERLINK "mailto:user@hive.apache.org"user@hive.apache.org <ma...@hive.apache.org>
>
> > *Subject:* Re: Hook is not getting called on running query from
>
> > beeline
>
> >
>
> > Is the hook getting called from beeline/JDBC for queries other than CTAS?
>
> > can you make sure that your configuration is correctly loaded by
>
> > beeline or JDBC code? you may check hive-site.xml is in the
> classpath
>
> > or add a property when requesting a connection from DriverManager
> or
>
> > you may also try like this
>
> >
>
> > beeline -u "jdbc:hive2://example.com:2181
>
> > <http://example.com:2181>,example.com:2181
>
> >
> <http://example.com:2181>,example:2181/;serviceDiscoveryMode=zooKeeper
>
> > ;zooKeeperNamespace=hiveserver2" *--hiveconf
>
> > hive.exec.post.hooks=<your_custom_hook_here>*
>
> >
>
> > Regards,
>
> >
>
> > Tanvi Thacker
>
> >
>
> > On Tue, Aug 21, 2018 at 9:00 AM Chetan Kothari
> <HYPERLINK "mailto:chetan.kothari@oracle.com%20%3cmailto:chetan.kothari@oracle.com%20%3cmailto:chetan.kothari@oracle.com%20%3cmailto:chetan.kothari@oracle.com"chetan.kothari@oracle.com <mailto:chetan.kothari@oracle.com <ma...@oracle.com>>> wrote:
>
> >
>
> > Any inputs on following issue will be helpful
>
> >
>
> > Regards
>
> >
>
> > Chetan
>
> >
>
> > *From:* Chetan Kothari
>
> > *Sent:* Tuesday, August 21, 2018 6:27 PM
>
> > *To:* HYPERLINK "mailto:user@hive.apache.org"user@hive.apache.org <ma...@hive.apache.org>
> <ma...@hive.apache.org>
>
> > *Subject:* Hook is not getting called on running query from
>
> > beeline
>
> >
>
> > Hi
>
> >
>
> > I have written custom PostExecutionHook which implements
> org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext to capture
> lineage of hive queries like create table as
>
> > select etc.
>
> >
>
> > This hook get's called when I run CTAS query o Hive Shell. But it is not getting called when I run same query from beeline or programmatically using JDBC Code.
>
> >
>
> > Any inputs on why hook is not getting called when I run query from beeline or programmatically using JDBC Code.
>
> >
>
> > Regards
>
> >
>
> > Chetan
>
> >
>
RE: Hook is not getting called on running query from beeline
Posted by Chetan Kothari <ch...@oracle.com>.
Hi Zoltan
I changed the configuration to connect to Hive.
[test@bdctest-1 hive]$ beeline
WARNING: Use "yarn jar" to launch YARN applications.
Beeline version 1.2.1000.2.4.2.0-258 by Apache Hive
Connected to: Apache Hive (version 1.2.1000.2.4.2.0-258)
Driver: Hive JDBC (version 1.2.1000.2.4.2.0-258)
Transaction isolation: TRANSACTION_REPEATABLE_READ
When I give wrong custom hook name or if hook class is not in classpath, it gives me following error.
Error: Error while processing statement: FAILED: Hive Internal Error: java.lang.ClassNotFoundException(test.dc.lineage.listeners.hive.TestPostExecutionHook) (state=08S01,code=12)
add jar /u01/bdctest/usr/hdp/2.4.2.0-258/hive/lib/hook-1.0.jar;
set hive.exec.post.hooks=test.dc.lineage.listeners.hive.TestPostExecutionHook;
I fixed this issue by explicitly setting hook and adding jar files.
After setting hook and adding jar file, I don’t set above error, query gets executed but custom hook is not getting called.
Regards
Chetan
-----Original Message-----
From: Zoltan Haindrich [mailto:kirk@rxd.hu]
Sent: Thursday, August 23, 2018 1:14 PM
To: user@hive.apache.org; Chetan Kothari
Subject: Re: Hook is not getting called on running query from beeline
Hello
There is an interesting message after connecting: "Connected to: Spark SQL (version 2.1.0.2.6.0.3-8)".
It seems you are not using Hive...please ask about this on a Spark related list.
cheers,
Zoltan
On 08/22/2018 08:43 PM, Chetan Kothari wrote:
> It seems --hiveconf is not getting picked from beeline.
>
> So even if I give wrong hook class name in following command, I am able to connect and execute queries.
>
> But hook is not getting called.
>
> beeline -u
> "jdbc:hive2://bdcsceforqa-bdcsce-1.dfml.ucfc2z3b.usdv1.abccloud.com:10
> 001/;transportMode=http;httpPath=cliservice" --hiveconf
> hive.exec.post.hooks=abcc.lineage.listeners.TestPostExecutionHook
>
> Connected to: Spark SQL (version 2.1.0.2.6.0.3-8)
>
> Driver: Hive JDBC (version 1.2.1000.2.4.2.0-258)
>
> Transaction isolation: TRANSACTION_REPEATABLE_READ
>
> Beeline version 1.2.1000.2.4.2.0-258 by Apache Hive
>
> Regards
>
> Chetan
>
> -----Original Message-----
> From: Zoltan Haindrich [mailto:kirk@rxd.hu]
> Sent: Wednesday, August 22, 2018 8:51 PM
> To: HYPERLINK "mailto:user@hive.apache.org"user@hive.apache.org; Chetan Kothari
> Subject: Re: Hook is not getting called on running query from beeline
>
> Hello
>
> I was wondering about that the property is not getting picked up from beeline; but it looks ok to me:
>
> $ beeline -u 'jdbc:hive2://...' --hiveconf hive.exec.post.hooks=asd -e 'select 1'
>
> Connected to: Apache Hive (version 1.2.1000.2.6.1.0-129)
>
> Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129) Transaction
> isolation: TRANSACTION_REPEATABLE_READ
>
> Error: Error while processing statement: FAILED: Hive Internal Error:
> java.lang.ClassNotFoundException(asd) (state=08S01,code=12)
>
> Which version of hive are you using?
>
> Try first with a non-existent classname. Executing the hooks usually leave some messages in the logs as well...
>
> cheers,
>
> Zoltan
>
> On 08/22/2018 11:22 AM, Chetan Kothari wrote:
>
> > Thanks Tanvi for your inputs.
>
> >
>
> > Hook is not getting called from beeline/JDBC for any queries.
>
> >
>
> > I ensured that hive-site.xml is in classpath and also tried following but still hook is not getting called.
>
> >
>
> > beeline -u "jdbc:hive2://example.com:2181
>
> > <http://example.com:2181>,example.com:2181
>
> >
> <http://example.com:2181>,example:2181/;serviceDiscoveryMode=zooKeeper
>
> > ;zooKeeperNamespace=hiveserver2" *--hiveconf
>
> > hive.exec.post.hooks=<your_custom_hook_here>*
>
> >
>
> > Regards
>
> >
>
> > Chetan
>
> >
>
> > *From:*Tanvi Thacker [mailto:tanvithacker5@gmail.com]
>
> > *Sent:* Wednesday, August 22, 2018 9:29 AM
>
> > *To:* HYPERLINK "mailto:user@hive.apache.org"user@hive.apache.org <ma...@hive.apache.org>
>
> > *Subject:* Re: Hook is not getting called on running query from
>
> > beeline
>
> >
>
> > Is the hook getting called from beeline/JDBC for queries other than CTAS?
>
> > can you make sure that your configuration is correctly loaded by
>
> > beeline or JDBC code? you may check hive-site.xml is in the
> classpath
>
> > or add a property when requesting a connection from DriverManager
> or
>
> > you may also try like this
>
> >
>
> > beeline -u "jdbc:hive2://example.com:2181
>
> > <http://example.com:2181>,example.com:2181
>
> >
> <http://example.com:2181>,example:2181/;serviceDiscoveryMode=zooKeeper
>
> > ;zooKeeperNamespace=hiveserver2" *--hiveconf
>
> > hive.exec.post.hooks=<your_custom_hook_here>*
>
> >
>
> > Regards,
>
> >
>
> > Tanvi Thacker
>
> >
>
> > On Tue, Aug 21, 2018 at 9:00 AM Chetan Kothari
> <HYPERLINK "mailto:chetan.kothari@oracle.com%20%3cmailto:chetan.kothari@oracle.com%20%3cmailto:chetan.kothari@oracle.com%20%3cmailto:chetan.kothari@oracle.com"chetan.kothari@oracle.com <mailto:chetan.kothari@oracle.com <ma...@oracle.com>>> wrote:
>
> >
>
> > Any inputs on following issue will be helpful
>
> >
>
> > Regards
>
> >
>
> > Chetan
>
> >
>
> > *From:* Chetan Kothari
>
> > *Sent:* Tuesday, August 21, 2018 6:27 PM
>
> > *To:* HYPERLINK "mailto:user@hive.apache.org"user@hive.apache.org <ma...@hive.apache.org>
> <ma...@hive.apache.org>
>
> > *Subject:* Hook is not getting called on running query from
>
> > beeline
>
> >
>
> > Hi
>
> >
>
> > I have written custom PostExecutionHook which implements
> org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext to capture
> lineage of hive queries like create table as
>
> > select etc.
>
> >
>
> > This hook get's called when I run CTAS query o Hive Shell. But it is not getting called when I run same query from beeline or programmatically using JDBC Code.
>
> >
>
> > Any inputs on why hook is not getting called when I run query from beeline or programmatically using JDBC Code.
>
> >
>
> > Regards
>
> >
>
> > Chetan
>
> >
>
Re: Hook is not getting called on running query from beeline
Posted by Zoltan Haindrich <ki...@rxd.hu>.
Hello
There is an interesting message after connecting: "Connected to: Spark SQL (version 2.1.0.2.6.0.3-8)".
It seems you are not using Hive...please ask about this on a Spark related list.
cheers,
Zoltan
On 08/22/2018 08:43 PM, Chetan Kothari wrote:
> It seems --hiveconf is not getting picked from beeline.
>
> So even if I give wrong hook class name in following command, I am able to connect and execute queries.
>
> But hook is not getting called.
>
> beeline -u "jdbc:hive2://bdcsceforqa-bdcsce-1.dfml.ucfc2z3b.usdv1.abccloud.com:10001/;transportMode=http;httpPath=cliservice" --hiveconf
> hive.exec.post.hooks=abcc.lineage.listeners.TestPostExecutionHook
>
> Connected to: Spark SQL (version 2.1.0.2.6.0.3-8)
>
> Driver: Hive JDBC (version 1.2.1000.2.4.2.0-258)
>
> Transaction isolation: TRANSACTION_REPEATABLE_READ
>
> Beeline version 1.2.1000.2.4.2.0-258 by Apache Hive
>
> Regards
>
> Chetan
>
> -----Original Message-----
> From: Zoltan Haindrich [mailto:kirk@rxd.hu]
> Sent: Wednesday, August 22, 2018 8:51 PM
> To: user@hive.apache.org; Chetan Kothari
> Subject: Re: Hook is not getting called on running query from beeline
>
> Hello
>
> I was wondering about that the property is not getting picked up from beeline; but it looks ok to me:
>
> $ beeline -u 'jdbc:hive2://...' --hiveconf hive.exec.post.hooks=asd -e 'select 1'
>
> Connected to: Apache Hive (version 1.2.1000.2.6.1.0-129)
>
> Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129) Transaction isolation: TRANSACTION_REPEATABLE_READ
>
> Error: Error while processing statement: FAILED: Hive Internal Error: java.lang.ClassNotFoundException(asd) (state=08S01,code=12)
>
> Which version of hive are you using?
>
> Try first with a non-existent classname. Executing the hooks usually leave some messages in the logs as well...
>
> cheers,
>
> Zoltan
>
> On 08/22/2018 11:22 AM, Chetan Kothari wrote:
>
> > Thanks Tanvi for your inputs.
>
> >
>
> > Hook is not getting called from beeline/JDBC for any queries.
>
> >
>
> > I ensured that hive-site.xml is in classpath and also tried following but still hook is not getting called.
>
> >
>
> > beeline -u "jdbc:hive2://example.com:2181
>
> > <http://example.com:2181>,example.com:2181
>
> > <http://example.com:2181>,example:2181/;serviceDiscoveryMode=zooKeeper
>
> > ;zooKeeperNamespace=hiveserver2" *--hiveconf
>
> > hive.exec.post.hooks=<your_custom_hook_here>*
>
> >
>
> > Regards
>
> >
>
> > Chetan
>
> >
>
> > *From:*Tanvi Thacker [mailto:tanvithacker5@gmail.com]
>
> > *Sent:* Wednesday, August 22, 2018 9:29 AM
>
> > *To:* user@hive.apache.org <ma...@hive.apache.org>
>
> > *Subject:* Re: Hook is not getting called on running query from
>
> > beeline
>
> >
>
> > Is the hook getting called from beeline/JDBC for queries other than CTAS?
>
> > can you make sure that your configuration is correctly loaded by
>
> > beeline or JDBC code? you may check hive-site.xml is in the classpath
>
> > or add a property when requesting a connection from DriverManager or
>
> > you may also try like this
>
> >
>
> > beeline -u "jdbc:hive2://example.com:2181
>
> > <http://example.com:2181>,example.com:2181
>
> > <http://example.com:2181>,example:2181/;serviceDiscoveryMode=zooKeeper
>
> > ;zooKeeperNamespace=hiveserver2" *--hiveconf
>
> > hive.exec.post.hooks=<your_custom_hook_here>*
>
> >
>
> > Regards,
>
> >
>
> > Tanvi Thacker
>
> >
>
> > On Tue, Aug 21, 2018 at 9:00 AM Chetan Kothari <chetan.kothari@oracle.com <mailto:chetan.kothari@oracle.com
> <ma...@oracle.com>>> wrote:
>
> >
>
> > Any inputs on following issue will be helpful
>
> >
>
> > Regards
>
> >
>
> > Chetan
>
> >
>
> > *From:* Chetan Kothari
>
> > *Sent:* Tuesday, August 21, 2018 6:27 PM
>
> > *To:* user@hive.apache.org <ma...@hive.apache.org> <ma...@hive.apache.org>
>
> > *Subject:* Hook is not getting called on running query from
>
> > beeline
>
> >
>
> > Hi
>
> >
>
> > I have written custom PostExecutionHook which implements org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext to capture lineage of hive queries like create table as
>
> > select etc.
>
> >
>
> > This hook get's called when I run CTAS query o Hive Shell. But it is not getting called when I run same query from beeline or programmatically using JDBC Code.
>
> >
>
> > Any inputs on why hook is not getting called when I run query from beeline or programmatically using JDBC Code.
>
> >
>
> > Regards
>
> >
>
> > Chetan
>
> >
>
RE: Hook is not getting called on running query from beeline
Posted by Chetan Kothari <ch...@oracle.com>.
It seems --hiveconf is not getting picked from beeline.
So even if I give wrong hook class name in following command, I am able to connect and execute queries.
But hook is not getting called.
beeline -u "jdbc:hive2://bdcsceforqa-bdcsce-1.dfml.ucfc2z3b.usdv1.abccloud.com:10001/;transportMode=http;httpPath=cliservice" --hiveconf hive.exec.post.hooks=abcc.lineage.listeners.TestPostExecutionHook
Connected to: Spark SQL (version 2.1.0.2.6.0.3-8)
Driver: Hive JDBC (version 1.2.1000.2.4.2.0-258)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Beeline version 1.2.1000.2.4.2.0-258 by Apache Hive
Regards
Chetan
-----Original Message-----
From: Zoltan Haindrich [mailto:kirk@rxd.hu]
Sent: Wednesday, August 22, 2018 8:51 PM
To: user@hive.apache.org; Chetan Kothari
Subject: Re: Hook is not getting called on running query from beeline
Hello
I was wondering about that the property is not getting picked up from beeline; but it looks ok to me:
$ beeline -u 'jdbc:hive2://...' --hiveconf hive.exec.post.hooks=asd -e 'select 1'
Connected to: Apache Hive (version 1.2.1000.2.6.1.0-129)
Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129) Transaction isolation: TRANSACTION_REPEATABLE_READ
Error: Error while processing statement: FAILED: Hive Internal Error: java.lang.ClassNotFoundException(asd) (state=08S01,code=12)
Which version of hive are you using?
Try first with a non-existent classname. Executing the hooks usually leave some messages in the logs as well...
cheers,
Zoltan
On 08/22/2018 11:22 AM, Chetan Kothari wrote:
> Thanks Tanvi for your inputs.
>
> Hook is not getting called from beeline/JDBC for any queries.
>
> I ensured that hive-site.xml is in classpath and also tried following but still hook is not getting called.
>
> beeline -u "jdbc:hive2://example.com:2181
> <http://example.com:2181>,example.com:2181
> <http://example.com:2181>,example:2181/;serviceDiscoveryMode=zooKeeper
> ;zooKeeperNamespace=hiveserver2" *--hiveconf
> hive.exec.post.hooks=<your_custom_hook_here>*
>
> Regards
>
> Chetan
>
> *From:*Tanvi Thacker [mailto:tanvithacker5@gmail.com]
> *Sent:* Wednesday, August 22, 2018 9:29 AM
> *To:* HYPERLINK "mailto:user@hive.apache.org"user@hive.apache.org
> *Subject:* Re: Hook is not getting called on running query from
> beeline
>
> Is the hook getting called from beeline/JDBC for queries other than CTAS?
> can you make sure that your configuration is correctly loaded by
> beeline or JDBC code? you may check hive-site.xml is in the classpath
> or add a property when requesting a connection from DriverManager or
> you may also try like this
>
> beeline -u "jdbc:hive2://example.com:2181
> <http://example.com:2181>,example.com:2181
> <http://example.com:2181>,example:2181/;serviceDiscoveryMode=zooKeeper
> ;zooKeeperNamespace=hiveserver2" *--hiveconf
> hive.exec.post.hooks=<your_custom_hook_here>*
>
> Regards,
>
> Tanvi Thacker
>
> On Tue, Aug 21, 2018 at 9:00 AM Chetan Kothari <HYPERLINK "mailto:chetan.kothari@oracle.com%20%3cmailto:chetan.kothari@oracle.com"chetan.kothari@oracle.com <ma...@oracle.com>> wrote:
>
> Any inputs on following issue will be helpful
>
> Regards
>
> Chetan
>
> *From:* Chetan Kothari
> *Sent:* Tuesday, August 21, 2018 6:27 PM
> *To:* HYPERLINK "mailto:user@hive.apache.org"user@hive.apache.org <ma...@hive.apache.org>
> *Subject:* Hook is not getting called on running query from
> beeline
>
> Hi
>
> I have written custom PostExecutionHook which implements org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext to capture lineage of hive queries like create table as
> select etc.
>
> This hook get's called when I run CTAS query o Hive Shell. But it is not getting called when I run same query from beeline or programmatically using JDBC Code.
>
> Any inputs on why hook is not getting called when I run query from beeline or programmatically using JDBC Code.
>
> Regards
>
> Chetan
>
Re: Hook is not getting called on running query from beeline
Posted by Zoltan Haindrich <ki...@rxd.hu>.
Hello
I was wondering about that the property is not getting picked up from beeline; but it looks ok to me:
$ beeline -u 'jdbc:hive2://...' --hiveconf hive.exec.post.hooks=asd -e 'select 1'
Connected to: Apache Hive (version 1.2.1000.2.6.1.0-129)
Driver: Hive JDBC (version 1.2.1000.2.6.1.0-129)
Transaction isolation: TRANSACTION_REPEATABLE_READ
Error: Error while processing statement: FAILED: Hive Internal Error: java.lang.ClassNotFoundException(asd) (state=08S01,code=12)
Which version of hive are you using?
Try first with a non-existent classname. Executing the hooks usually leave some messages in the logs as well...
cheers,
Zoltan
On 08/22/2018 11:22 AM, Chetan Kothari wrote:
> Thanks Tanvi for your inputs.
>
> Hook is not getting called from beeline/JDBC for any queries.
>
> I ensured that hive-site.xml is in classpath and also tried following but still hook is not getting called.
>
> beeline -u "jdbc:hive2://example.com:2181 <http://example.com:2181>,example.com:2181
> <http://example.com:2181>,example:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2" *--hiveconf hive.exec.post.hooks=<your_custom_hook_here>*
>
> Regards
>
> Chetan
>
> *From:*Tanvi Thacker [mailto:tanvithacker5@gmail.com]
> *Sent:* Wednesday, August 22, 2018 9:29 AM
> *To:* user@hive.apache.org
> *Subject:* Re: Hook is not getting called on running query from beeline
>
> Is the hook getting called from beeline/JDBC for queries other than CTAS?
> can you make sure that your configuration is correctly loaded by beeline or JDBC code? you may check hive-site.xml is in the classpath or add a property when requesting a
> connection from DriverManager or you may also try like this
>
> beeline -u "jdbc:hive2://example.com:2181 <http://example.com:2181>,example.com:2181
> <http://example.com:2181>,example:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2" *--hiveconf hive.exec.post.hooks=<your_custom_hook_here>*
>
> Regards,
>
> Tanvi Thacker
>
> On Tue, Aug 21, 2018 at 9:00 AM Chetan Kothari <chetan.kothari@oracle.com <ma...@oracle.com>> wrote:
>
> Any inputs on following issue will be helpful
>
> Regards
>
> Chetan
>
> *From:* Chetan Kothari
> *Sent:* Tuesday, August 21, 2018 6:27 PM
> *To:* user@hive.apache.org <ma...@hive.apache.org>
> *Subject:* Hook is not getting called on running query from beeline
>
> Hi
>
> I have written custom PostExecutionHook which implements org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext to capture lineage of hive queries like create table as
> select etc.
>
> This hook get's called when I run CTAS query o Hive Shell. But it is not getting called when I run same query from beeline or programmatically using JDBC Code.
>
> Any inputs on why hook is not getting called when I run query from beeline or programmatically using JDBC Code.
>
> Regards
>
> Chetan
>
RE: Hook is not getting called on running query from beeline
Posted by Chetan Kothari <ch...@oracle.com>.
Thanks Tanvi for your inputs.
Hook is not getting called from beeline/JDBC for any queries.
I ensured that hive-site.xml is in classpath and also tried following but still hook is not getting called.
beeline -u "jdbc:hive2://HYPERLINK "http://example.com:2181"example.com:2181,HYPERLINK "http://example.com:2181"example.com:2181,example:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2" --hiveconf hive.exec.post.hooks=<your_custom_hook_here>
Regards
Chetan
From: Tanvi Thacker [mailto:tanvithacker5@gmail.com]
Sent: Wednesday, August 22, 2018 9:29 AM
To: user@hive.apache.org
Subject: Re: Hook is not getting called on running query from beeline
Is the hook getting called from beeline/JDBC for queries other than CTAS?
can you make sure that your configuration is correctly loaded by beeline or JDBC code? you may check hive-site.xml is in the classpath or add a property when requesting a connection from DriverManager or you may also try like this
beeline -u "jdbc:hive2://HYPERLINK "http://example.com:2181"example.com:2181,HYPERLINK "http://example.com:2181"example.com:2181,example:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2" --hiveconf hive.exec.post.hooks=<your_custom_hook_here>
Regards,
Tanvi Thacker
On Tue, Aug 21, 2018 at 9:00 AM Chetan Kothari <HYPERLINK "mailto:chetan.kothari@oracle.com"chetan.kothari@oracle.com> wrote:
Any inputs on following issue will be helpful
Regards
Chetan
From: Chetan Kothari
Sent: Tuesday, August 21, 2018 6:27 PM
To: HYPERLINK "mailto:user@hive.apache.org"user@hive.apache.org
Subject: Hook is not getting called on running query from beeline
Hi
I have written custom PostExecutionHook which implements org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext to capture lineage of hive queries like create table as select etc.
This hook get's called when I run CTAS query o Hive Shell. But it is not getting called when I run same query from beeline or programmatically using JDBC Code.
Any inputs on why hook is not getting called when I run query from beeline or programmatically using JDBC Code.
Regards
Chetan
Re: Hook is not getting called on running query from beeline
Posted by Tanvi Thacker <ta...@gmail.com>.
Is the hook getting called from beeline/JDBC for queries other than CTAS?
can you make sure that your configuration is correctly loaded by beeline or
JDBC code? you may check hive-site.xml is in the classpath or add a
property when requesting a connection from DriverManager or you may also
try like this
beeline -u "jdbc:hive2://example.com:2181,example.com:2181,example:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2"
*--hiveconf hive.exec.post.hooks=<your_custom_hook_here>*
Regards,
Tanvi Thacker
On Tue, Aug 21, 2018 at 9:00 AM Chetan Kothari <ch...@oracle.com>
wrote:
> Any inputs on following issue will be helpful
>
>
>
> Regards
>
> Chetan
>
>
>
> *From:* Chetan Kothari
> *Sent:* Tuesday, August 21, 2018 6:27 PM
> *To:* user@hive.apache.org
> *Subject:* Hook is not getting called on running query from beeline
>
>
>
> Hi
>
>
>
> I have written custom PostExecutionHook which implements
> org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext to capture lineage
> of hive queries like create table as select etc.
>
> This hook get's called when I run CTAS query o Hive Shell. But it is not
> getting called when I run same query from beeline or programmatically using
> JDBC Code.
>
>
>
> Any inputs on why hook is not getting called when I run query from beeline
> or programmatically using JDBC Code.
>
>
>
> Regards
>
> Chetan
>
RE: Hook is not getting called on running query from beeline
Posted by Chetan Kothari <ch...@oracle.com>.
Any inputs on following issue will be helpful
Regards
Chetan
From: Chetan Kothari
Sent: Tuesday, August 21, 2018 6:27 PM
To: user@hive.apache.org
Subject: Hook is not getting called on running query from beeline
Hi
I have written custom PostExecutionHook which implements org.apache.hadoop.hive.ql.hooks.ExecuteWithHookContext to capture lineage of hive queries like create table as select etc.
This hook get's called when I run CTAS query o Hive Shell. But it is not getting called when I run same query from beeline or programmatically using JDBC Code.
Any inputs on why hook is not getting called when I run query from beeline or programmatically using JDBC Code.
Regards
Chetan