You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by Amal Gupta <Am...@aexp.com> on 2015/03/21 10:04:32 UTC
Executing HQL files from JAVA application.
Hi Everyone,
I am trying to execute a hive *.hql file from a java application. I had tried a couple of ways of doing it through JDBC driver for hive and through spring jdbc template but yet, the only way which was successful for me was to create a runtime process and then execute it.
The java code to do that is
Process p = Runtime.getRuntime().exec((new String[]{"hive","-f","/hive/scripts/demoHiveTable.hql"}));
Although this works but, I wanted to know if there is better way to do it using some driver functionality or any other api for hive. The method I used will do the execution for me, but yet, in case of failures while executing the hql script, the same will not be visible to the Java application.
Any pointers or suggestions will be greatly helpful. Thanks in advance.
Please let me know in case you need any other details from me.
Regards,
Amal
American Express made the following annotations
******************************************************************************
"This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you."
American Express a ajout� le commentaire suivant le Ce courrier et toute pi�ce jointe qu'il contient sont r�serv�s au seul destinataire indiqu� et peuvent renfermer des
renseignements confidentiels et privil�gi�s. Si vous n'�tes pas le destinataire pr�vu, toute divulgation, duplication, utilisation ou distribution du courrier ou de toute pi�ce jointe est interdite. Si vous avez re�u cette communication par erreur, veuillez nous en aviser par courrier et d�truire imm�diatement le courrier et les pi�ces jointes. Merci.
******************************************************************************
Re: Executing HQL files from JAVA application.
Posted by Steve Howard <st...@gmail.com>.
I would argue that executing arbitrary code from a random remote server has just increased your security scope footprint in terms of the need to control another access point.
Purely out of curiosity, is there a compelling architectural reason or environment limitation that results in your need to do it this way?
Sent from my iPad
> On Mar 25, 2015, at 9:57 AM, Amal Gupta <Am...@aexp.com> wrote:
>
> Hi Gopal,
>
> Thanks a lot.
> Connectivity to the HiveServer2 was not an issue. We were able to connect using the example that you shared and using Beeline. The issue is a script execution from java app. May be I missed something, but I was not able to find an efficient and elegant way to execute hive scripts placed at a specific location from the java app.
>
> The scenario is
> App Placed at a location A should be able to connect to hiveServer2 at B and execute script TestHive.hql placed at a location on B(say root/testProject/hive/scripts/TestHive.hql).
>
> Regards,
> Amal
>
> -----Original Message-----
> From: Gopal Vijayaraghavan [mailto:gopal@hortonworks.com] On Behalf Of Gopal Vijayaraghavan
> Sent: Wednesday, March 25, 2015 8:49 AM
> To: user@hive.apache.org
> Cc: Amal Gupta
> Subject: Re: Executing HQL files from JAVA application.
>
> Hi,
>
> Any mechanism which bypasses schema layers for SQL is a bad idea.
>
> See this example for how you can connect to HiveServer2 directly from Java
> -
> https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveSe
> rver2Clients-JDBCClientSampleCode
>
> Use the JDBC driver to access HiveServer2 through a first-class Java API.
>
> If you find any performance issues with this method, let me know and I can fix it.
>
> Cheers,
> Gopal
>
> From: Amal Gupta <Am...@aexp.com>
> Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
> Date: Sunday, March 22, 2015 at 10:53 PM
> To: "user@hive.apache.org" <us...@hive.apache.org>
> Subject: RE: Executing HQL files from JAVA application.
>
>
> Hey Mich,
>
>
> Got any clues regarding the failure of the code that I sent?
>
> I was going through the project and the code again and I suspect the mis-matching dependencies to be the culprits. I am currently trying to re-align the dependencies as per the pom given on the mvnrepository.com while trying to see if a particular configuration succeeds.
>
> Will keep you posted on my progress. Thanks again for all the help that you are providing.
> J
>
> Regards,
> Amal
>
>
> From: Amal Gupta
>
> Sent: Sunday, March 22, 2015 7:52 AM
> To: user@hive.apache.org
> Subject: RE: Executing HQL files from JAVA application.
>
>
>
> Hi Mich,
>
> J A coincidence. Even I am new to hive. My test script which I am trying to execute contains a drop and a create statement.
>
> Script :-
>
> use test_db;
> DROP TABLE IF EXISTS demoHiveTable;
> CREATE EXTERNAL TABLE demoHiveTable (
> demoId string,
> demoName string
> ) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'
>
> STORED AS TEXTFILE LOCATION '/hive/';
>
>
> Java Code: -
> Not sure whether this will have an impact but the the code is a part of Spring batch Tasklet being triggered from the Batch-Context. This tasklet runs in parallel with other tasklets.
>
>
> public RepeatStatus execute(StepContribution arg0, ChunkContext arg1)
>
> throws Exception {
> String[] args =
> {"-d",BeeLine.BEELINE_DEFAULT_JDBC_DRIVER,"-u","jdbc:hive2://<server-name>:
> 10000/test_db",
>
> "-n","******","-p","******",
> "-f","C://Work//test_hive.hql"};
>
> BeeLine
> beeline = new BeeLine();
> ByteArrayOutputStream os = new ByteArrayOutputStream();
> PrintStream beelineOutputStream = new PrintStream(os);
> beeline.setOutputStream(beelineOutputStream);
> beeline.setErrorStream(beelineOutputStream);
> beeline.begin(args,null);
> String output = os.toString("UTF8");
> System.out.println(output);
>
> return RepeatStatus.FINISHED;
> }
>
> It will be great if you can share the piece of code that worked for you.
> May be it will give me some pointers on how to go ahead.
>
>
> Best Regards,
> Amal
>
>
> From: Mich Talebzadeh [mailto:mich@peridale.co.uk]
>
> Sent: Sunday, March 22, 2015 2:58 AM
> To: user@hive.apache.org
> Subject: RE: Executing HQL files from JAVA application.
>
>
>
> Hi Amal;
>
> Me coming from relational database (Oracle, Sybase) background J always expect that a DDL statement like DROP TABLE has to run in its own transaction and cannot be combined with a DML statement.
>
> Now I suspect that when you run the command DROP TABLE IF EXIASTS <TABLE_NAME>; like below in beehive it works
>
> 0: jdbc:hive2://rhes564:10010/default> drop table if exists mytest; No rows affected (0.216 seconds)
>
> That runs in its own transaction so it works. However, I suspect in JAVA that is not the case. Can you possibly provide your JAVA code to see what exactly it is doing.
>
> Thanks,
>
> Mich
>
> http://talebzadehmich.wordpress.com
>
> Publications due shortly:
> Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache
>
> NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
>
>
> From: Amal Gupta [mailto:Amal.Gupta1@aexp.com]
>
> Sent: 21 March 2015 18:16
> To: user@hive.apache.org
> Subject: RE: Executing HQL files from JAVA application.
>
>
>
> Hi Mich,
>
>
> Thank you for your response. I was not aware of beeline. I have now included this in my app and this looks a much better solution going forward. In the last couple of hours I have tried to work with beeline but have been facing some issues.
>
>
> 1.
> I was able to run on the remote server command line a beeline command given below . This was successful.
>
> beeline -u jdbc:hive2://<server-name>:10000/test_db
> org.apache.hive.jdbc.HiveDriver -n ***** -p ****** -f /hive/scripts/demoHiveTable.hql
>
> 2.
> Running the same from the java app results in the issues. My script contains a drop table for the demoTable but the table is not dropped when running
> from java. (DROP TABLE IF EXISTS demoHiveTable;) . I see the following
> logs.
>
> SLF4J: This version of SLF4J requires log4j version 1.2.12 or later. See also http://www.slf4j.org/codes.html#log4j_version
> Exception in thread "Thread-1" java.lang.NoSuchMethodError:
> org.apache.hive.jdbc.HiveStatement.hasMoreLogs()Z
> at org.apache.hive.beeline.Commands$1.run(Commands.java:839)
> at java.lang.Thread.run(Thread.java:662)
> Connecting to jdbc:hive2: ://<server-name>:10000/test_db
>
> Connected to: Hive (version 0.12-mapr-1401-140130)
> Driver: Hive (version 0.12-mapr-1401-140130) Transaction isolation: TRANSACTION_REPEATABLE_READ
> 0: jdbc:hive2://<server-name>:10000/test_db > DROP TABLE IF EXISTS demoHiveTable; No rows affected (0.797 seconds) org.apache.hive.jdbc.HiveStatement.getQueryLog()Ljava/util/List;
>
> Closing: 0: jdbc:hive2://<server-name>:10000/test_db
>
> I tried running the following commands
>
> String[] args =
> {"-d",BeeLine.BEELINE_DEFAULT_JDBC_DRIVER,"-u","jdbc:hive2://<server-name:1
> 0000/test_db",
>
> "-n","******","-p","******",
> "-f","C://Work//test_hive.hql"};
>
> And
>
> String[] args = {"beeline","-u","jdbc:hive2:// ://<server-name:10000/test_db²,
>
> "org.apache.hive.jdbc.HiveDriver","-n","******","-p","******","-f","C://Wor
> k//test_hive.hql"};
>
> 3.
> The link I referred was from googlesource (https://apache.googlesource.com/hive/+/a681b57609ae306c7623491549cc0176b6c
> df38d/beeline/src/test/org/apache/hive/beeline/src/test/TestBeeLineWithArgs
> .java)
>
> 4.
> I also see that there are errors being propagated back to the application.
> Is there some configuration to get detailed logs too?
>
>
> Not sure if I am missing something here.
>
> Any pointers or assistance will be of great help.
>
> Regards,
> Amal
>
>
> From: Mich Talebzadeh [mailto:mich@peridale.co.uk]
>
> Sent: Saturday, March 21, 2015 6:59 PM
> To: user@hive.apache.org
> Subject: RE: Executing HQL files from JAVA application.
>
>
>
> Hi Amal,
>
> Do you have hiveserver2 running?
>
> You can use beeline to execute the query outside of JAVA
>
> beeline -u jdbc:hive2://rhes564:10010/default
> org.apache.hive.jdbc.HiveDriver -n hduser -p xxxx' -f ./create_index_on_t.sql
>
> And the output shows there as well.
>
> scan complete in 10ms
> Connecting to jdbc:hive2://rhes564:10010/default
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/home/hduser/hadoop/hadoop-2.6.0/share/hadoop/common/lib/slf4j-lo
> g4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/usr/lib/hive/lib/hive-jdbc-0.14.0-standalone.jar!/org/slf4j/impl
> /StaticLoggerBinder.class]
> SLF4J: See
> http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> Connected to: Apache Hive (version 0.14.0)
> Driver: Hive JDBC (version 0.14.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> 0: jdbc:hive2://rhes564:10010/default> use asehadoop; No rows affected (0.084 seconds)
> 0: jdbc:hive2://rhes564:10010/default> --drop index t_ui on t;
> 0: jdbc:hive2://rhes564:10010/default> create index t_ui on table t
> (object_id) as 'COMPACT' WITH DEFERRED REBUILD;
>
> HTH
>
> Mich Talebzadeh
>
> http://talebzadehmich.wordpress.com
>
> Publications due shortly:
> Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache
>
> NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
>
>
> From: Amal Gupta [mailto:Amal.Gupta1@aexp.com]
>
> Sent: 21 March 2015 09:05
> To: user@hive.apache.org
> Subject: Executing HQL files from JAVA application.
>
>
>
> Hi Everyone,
>
> I am trying to execute a hive *.hql file from a java application. I had tried a couple of ways of doing it through JDBC driver for hive and through spring jdbc template but yet, the only way which was successful for me was to create a runtime process and then execute it.
>
>
> The java code to do that is
>
> Process p = Runtime.getRuntime().exec((new String[]{"hive","-f","/hive/scripts/demoHiveTable.hql"}));
>
>
> Although this works but, I wanted to know if there is better way to do it using some driver functionality or any other api for hive. The method I used will do the execution for me, but yet, in case of failures while executing the hql script, the same will not be visible to the Java application.
>
>
> Any pointers or suggestions will be greatly helpful. Thanks in advance.
>
>
> Please let me know in case you need any other details from me.
>
>
> Regards,
> Amal
>
>
> ________________________________________
> American Express made the following annotations ________________________________________
>
> "This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you."
>
>
> American Express a ajouté le commentaire suivant le Ce courrier et toute pièce jointe qu'il contient sont réservés au seul destinataire indiqué et peuvent renfermer des renseignements confidentiels et privilégiés. Si vous n'êtes pas le destinataire prévu, toute divulgation, duplication, utilisation ou distribution du courrier ou de toute pièce jointe est interdite. Si vous avez reçu cette communication par erreur, veuillez nous en aviser par courrier et détruire immédiatement le courrier et les pièces jointes. Merci.
>
> ________________________________________
>
>
>
>
> American Express made the following annotations
> ******************************************************************************
> "This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you."
>
> American Express a ajouté le commentaire suivant le Ce courrier et toute pièce jointe qu'il contient sont réservés au seul destinataire indiqué et peuvent renfermer des
> renseignements confidentiels et privilégiés. Si vous n'êtes pas le destinataire prévu, toute divulgation, duplication, utilisation ou distribution du courrier ou de toute pièce jointe est interdite. Si vous avez reçu cette communication par erreur, veuillez nous en aviser par courrier et détruire immédiatement le courrier et les pièces jointes. Merci.
>
> ******************************************************************************
RE: Executing HQL files from JAVA application.
Posted by Amal Gupta <Am...@aexp.com>.
Hi Gopal,
Thanks a lot.
Connectivity to the HiveServer2 was not an issue. We were able to connect using the example that you shared and using Beeline. The issue is a script execution from java app. May be I missed something, but I was not able to find an efficient and elegant way to execute hive scripts placed at a specific location from the java app.
The scenario is
App Placed at a location A should be able to connect to hiveServer2 at B and execute script TestHive.hql placed at a location on B(say root/testProject/hive/scripts/TestHive.hql).
Regards,
Amal
-----Original Message-----
From: Gopal Vijayaraghavan [mailto:gopal@hortonworks.com] On Behalf Of Gopal Vijayaraghavan
Sent: Wednesday, March 25, 2015 8:49 AM
To: user@hive.apache.org
Cc: Amal Gupta
Subject: Re: Executing HQL files from JAVA application.
Hi,
Any mechanism which bypasses schema layers for SQL is a bad idea.
See this example for how you can connect to HiveServer2 directly from Java
-
https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveSe
rver2Clients-JDBCClientSampleCode
Use the JDBC driver to access HiveServer2 through a first-class Java API.
If you find any performance issues with this method, let me know and I can fix it.
Cheers,
Gopal
From: Amal Gupta <Am...@aexp.com>
Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
Date: Sunday, March 22, 2015 at 10:53 PM
To: "user@hive.apache.org" <us...@hive.apache.org>
Subject: RE: Executing HQL files from JAVA application.
Hey Mich,
Got any clues regarding the failure of the code that I sent?
I was going through the project and the code again and I suspect the mis-matching dependencies to be the culprits. I am currently trying to re-align the dependencies as per the pom given on the mvnrepository.com while trying to see if a particular configuration succeeds.
Will keep you posted on my progress. Thanks again for all the help that you are providing.
J
Regards,
Amal
From: Amal Gupta
Sent: Sunday, March 22, 2015 7:52 AM
To: user@hive.apache.org
Subject: RE: Executing HQL files from JAVA application.
Hi Mich,
J A coincidence. Even I am new to hive. My test script which I am trying to execute contains a drop and a create statement.
Script :-
use test_db;
DROP TABLE IF EXISTS demoHiveTable;
CREATE EXTERNAL TABLE demoHiveTable (
demoId string,
demoName string
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'
STORED AS TEXTFILE LOCATION '/hive/';
Java Code: -
Not sure whether this will have an impact but the the code is a part of Spring batch Tasklet being triggered from the Batch-Context. This tasklet runs in parallel with other tasklets.
public RepeatStatus execute(StepContribution arg0, ChunkContext arg1)
throws Exception {
String[] args =
{"-d",BeeLine.BEELINE_DEFAULT_JDBC_DRIVER,"-u","jdbc:hive2://<server-name>:
10000/test_db",
"-n","******","-p","******",
"-f","C://Work//test_hive.hql"};
BeeLine
beeline = new BeeLine();
ByteArrayOutputStream os = new ByteArrayOutputStream();
PrintStream beelineOutputStream = new PrintStream(os);
beeline.setOutputStream(beelineOutputStream);
beeline.setErrorStream(beelineOutputStream);
beeline.begin(args,null);
String output = os.toString("UTF8");
System.out.println(output);
return RepeatStatus.FINISHED;
}
It will be great if you can share the piece of code that worked for you.
May be it will give me some pointers on how to go ahead.
Best Regards,
Amal
From: Mich Talebzadeh [mailto:mich@peridale.co.uk]
Sent: Sunday, March 22, 2015 2:58 AM
To: user@hive.apache.org
Subject: RE: Executing HQL files from JAVA application.
Hi Amal;
Me coming from relational database (Oracle, Sybase) background J always expect that a DDL statement like DROP TABLE has to run in its own transaction and cannot be combined with a DML statement.
Now I suspect that when you run the command DROP TABLE IF EXIASTS <TABLE_NAME>; like below in beehive it works
0: jdbc:hive2://rhes564:10010/default> drop table if exists mytest; No rows affected (0.216 seconds)
That runs in its own transaction so it works. However, I suspect in JAVA that is not the case. Can you possibly provide your JAVA code to see what exactly it is doing.
Thanks,
Mich
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
From: Amal Gupta [mailto:Amal.Gupta1@aexp.com]
Sent: 21 March 2015 18:16
To: user@hive.apache.org
Subject: RE: Executing HQL files from JAVA application.
Hi Mich,
Thank you for your response. I was not aware of beeline. I have now included this in my app and this looks a much better solution going forward. In the last couple of hours I have tried to work with beeline but have been facing some issues.
1.
I was able to run on the remote server command line a beeline command given below . This was successful.
beeline -u jdbc:hive2://<server-name>:10000/test_db
org.apache.hive.jdbc.HiveDriver -n ***** -p ****** -f /hive/scripts/demoHiveTable.hql
2.
Running the same from the java app results in the issues. My script contains a drop table for the demoTable but the table is not dropped when running
from java. (DROP TABLE IF EXISTS demoHiveTable;) . I see the following
logs.
SLF4J: This version of SLF4J requires log4j version 1.2.12 or later. See also http://www.slf4j.org/codes.html#log4j_version
Exception in thread "Thread-1" java.lang.NoSuchMethodError:
org.apache.hive.jdbc.HiveStatement.hasMoreLogs()Z
at org.apache.hive.beeline.Commands$1.run(Commands.java:839)
at java.lang.Thread.run(Thread.java:662)
Connecting to jdbc:hive2: ://<server-name>:10000/test_db
Connected to: Hive (version 0.12-mapr-1401-140130)
Driver: Hive (version 0.12-mapr-1401-140130) Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://<server-name>:10000/test_db > DROP TABLE IF EXISTS demoHiveTable; No rows affected (0.797 seconds) org.apache.hive.jdbc.HiveStatement.getQueryLog()Ljava/util/List;
Closing: 0: jdbc:hive2://<server-name>:10000/test_db
I tried running the following commands
String[] args =
{"-d",BeeLine.BEELINE_DEFAULT_JDBC_DRIVER,"-u","jdbc:hive2://<server-name:1
0000/test_db",
"-n","******","-p","******",
"-f","C://Work//test_hive.hql"};
And
String[] args = {"beeline","-u","jdbc:hive2:// ://<server-name:10000/test_db²,
"org.apache.hive.jdbc.HiveDriver","-n","******","-p","******","-f","C://Wor
k//test_hive.hql"};
3.
The link I referred was from googlesource (https://apache.googlesource.com/hive/+/a681b57609ae306c7623491549cc0176b6c
df38d/beeline/src/test/org/apache/hive/beeline/src/test/TestBeeLineWithArgs
.java)
4.
I also see that there are errors being propagated back to the application.
Is there some configuration to get detailed logs too?
Not sure if I am missing something here.
Any pointers or assistance will be of great help.
Regards,
Amal
From: Mich Talebzadeh [mailto:mich@peridale.co.uk]
Sent: Saturday, March 21, 2015 6:59 PM
To: user@hive.apache.org
Subject: RE: Executing HQL files from JAVA application.
Hi Amal,
Do you have hiveserver2 running?
You can use beeline to execute the query outside of JAVA
beeline -u jdbc:hive2://rhes564:10010/default
org.apache.hive.jdbc.HiveDriver -n hduser -p xxxx' -f ./create_index_on_t.sql
And the output shows there as well.
scan complete in 10ms
Connecting to jdbc:hive2://rhes564:10010/default
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/hduser/hadoop/hadoop-2.6.0/share/hadoop/common/lib/slf4j-lo
g4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hive/lib/hive-jdbc-0.14.0-standalone.jar!/org/slf4j/impl
/StaticLoggerBinder.class]
SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Connected to: Apache Hive (version 0.14.0)
Driver: Hive JDBC (version 0.14.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://rhes564:10010/default> use asehadoop; No rows affected (0.084 seconds)
0: jdbc:hive2://rhes564:10010/default> --drop index t_ui on t;
0: jdbc:hive2://rhes564:10010/default> create index t_ui on table t
(object_id) as 'COMPACT' WITH DEFERRED REBUILD;
HTH
Mich Talebzadeh
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
From: Amal Gupta [mailto:Amal.Gupta1@aexp.com]
Sent: 21 March 2015 09:05
To: user@hive.apache.org
Subject: Executing HQL files from JAVA application.
Hi Everyone,
I am trying to execute a hive *.hql file from a java application. I had tried a couple of ways of doing it through JDBC driver for hive and through spring jdbc template but yet, the only way which was successful for me was to create a runtime process and then execute it.
The java code to do that is
Process p = Runtime.getRuntime().exec((new String[]{"hive","-f","/hive/scripts/demoHiveTable.hql"}));
Although this works but, I wanted to know if there is better way to do it using some driver functionality or any other api for hive. The method I used will do the execution for me, but yet, in case of failures while executing the hql script, the same will not be visible to the Java application.
Any pointers or suggestions will be greatly helpful. Thanks in advance.
Please let me know in case you need any other details from me.
Regards,
Amal
________________________________________
American Express made the following annotations ________________________________________
"This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you."
American Express a ajouté le commentaire suivant le Ce courrier et toute pièce jointe qu'il contient sont réservés au seul destinataire indiqué et peuvent renfermer des renseignements confidentiels et privilégiés. Si vous n'êtes pas le destinataire prévu, toute divulgation, duplication, utilisation ou distribution du courrier ou de toute pièce jointe est interdite. Si vous avez reçu cette communication par erreur, veuillez nous en aviser par courrier et détruire immédiatement le courrier et les pièces jointes. Merci.
________________________________________
American Express made the following annotations
******************************************************************************
"This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you."
American Express a ajouté le commentaire suivant le Ce courrier et toute pièce jointe qu'il contient sont réservés au seul destinataire indiqué et peuvent renfermer des
renseignements confidentiels et privilégiés. Si vous n'êtes pas le destinataire prévu, toute divulgation, duplication, utilisation ou distribution du courrier ou de toute pièce jointe est interdite. Si vous avez reçu cette communication par erreur, veuillez nous en aviser par courrier et détruire immédiatement le courrier et les pièces jointes. Merci.
******************************************************************************
Re: Executing HQL files from JAVA application.
Posted by Gopal Vijayaraghavan <go...@apache.org>.
Hi,
Any mechanism which bypasses schema layers for SQL is a bad idea.
See this example for how you can connect to HiveServer2 directly from Java
-
https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveSe
rver2Clients-JDBCClientSampleCode
Use the JDBC driver to access HiveServer2 through a first-class Java API.
If you find any performance issues with this method, let me know and I can
fix it.
Cheers,
Gopal
From: Amal Gupta <Am...@aexp.com>
Reply-To: "user@hive.apache.org" <us...@hive.apache.org>
Date: Sunday, March 22, 2015 at 10:53 PM
To: "user@hive.apache.org" <us...@hive.apache.org>
Subject: RE: Executing HQL files from JAVA application.
Hey Mich,
Got any clues regarding the failure of the code that I sent?
I was going through the project and the code again and I suspect the
mis-matching dependencies to be the culprits. I am currently trying to
re-align the dependencies
as per the pom given on the mvnrepository.com while trying to see if a
particular configuration succeeds.
Will keep you posted on my progress. Thanks again for all the help that
you are providing.
J
Regards,
Amal
From: Amal Gupta
Sent: Sunday, March 22, 2015 7:52 AM
To: user@hive.apache.org
Subject: RE: Executing HQL files from JAVA application.
Hi Mich,
J A coincidence. Even I am new to hive. My test script which I am trying
to execute
contains a drop and a create statement.
Script :-
use test_db;
DROP TABLE IF EXISTS demoHiveTable;
CREATE EXTERNAL TABLE demoHiveTable (
demoId string,
demoName string
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'
STORED AS TEXTFILE LOCATION '/hive/';
Java Code: -
Not sure whether this will have an impact but the the code is a part of
Spring batch Tasklet being triggered from the Batch-Context. This tasklet
runs in parallel with other tasklets.
public RepeatStatus execute(StepContribution arg0, ChunkContext arg1)
throws Exception {
String[] args =
{"-d",BeeLine.BEELINE_DEFAULT_JDBC_DRIVER,"-u","jdbc:hive2://<server-name>:
10000/test_db",
"-n","******","-p","******",
"-f","C://Work//test_hive.hql"};
BeeLine
beeline = new BeeLine();
ByteArrayOutputStream os =
new ByteArrayOutputStream();
PrintStream beelineOutputStream =
new PrintStream(os);
beeline.setOutputStream(beelineOutputStream);
beeline.setErrorStream(beelineOutputStream);
beeline.begin(args,null);
String output = os.toString("UTF8");
System.out.println(output);
return RepeatStatus.FINISHED;
}
It will be great if you can share the piece of code that worked for you.
May be it will give me some pointers on how to go ahead.
Best Regards,
Amal
From: Mich Talebzadeh [mailto:mich@peridale.co.uk]
Sent: Sunday, March 22, 2015 2:58 AM
To: user@hive.apache.org
Subject: RE: Executing HQL files from JAVA application.
Hi Amal;
Me coming from relational database (Oracle, Sybase) background
J always expect that a DDL statement like DROP TABLE has to run in its own
transaction and cannot be
combined with a DML statement.
Now I suspect that when you run the command DROP TABLE IF EXIASTS
<TABLE_NAME>; like below in beehive it works
0: jdbc:hive2://rhes564:10010/default> drop table if exists mytest;
No rows affected (0.216 seconds)
That runs in its own transaction so it works. However, I suspect in JAVA
that is not the case. Can you possibly provide your JAVA code to see what
exactly it is
doing.
Thanks,
Mich
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and
Coherence Cache
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Ltd, its
subsidiaries or their employees, unless expressly so stated. It is the
responsibility of the recipient to ensure
that this email is virus free, therefore neither Peridale Ltd, its
subsidiaries nor their employees accept any responsibility.
From: Amal Gupta [mailto:Amal.Gupta1@aexp.com]
Sent: 21 March 2015 18:16
To: user@hive.apache.org
Subject: RE: Executing HQL files from JAVA application.
Hi Mich,
Thank you for your response. I was not aware of beeline. I have now
included this in my app and this looks a much better solution going
forward. In the last
couple of hours I have tried to work with beeline but have been facing
some issues.
1.
I was able to run on the remote server command line a beeline command
given below . This was successful.
beeline -u jdbc:hive2://<server-name>:10000/test_db
org.apache.hive.jdbc.HiveDriver -n ***** -p ****** -f
/hive/scripts/demoHiveTable.hql
2.
Running the same from the java app results in the issues. My script
contains a drop table for the demoTable but the table is not dropped when
running
from java. (DROP TABLE IF EXISTS demoHiveTable;) . I see the following
logs.
SLF4J: This version of SLF4J requires log4j version 1.2.12 or later. See
also
http://www.slf4j.org/codes.html#log4j_version
Exception in thread "Thread-1" java.lang.NoSuchMethodError:
org.apache.hive.jdbc.HiveStatement.hasMoreLogs()Z
at org.apache.hive.beeline.Commands$1.run(Commands.java:839)
at java.lang.Thread.run(Thread.java:662)
Connecting to jdbc:hive2: ://<server-name>:10000/test_db
Connected to: Hive (version 0.12-mapr-1401-140130)
Driver: Hive (version 0.12-mapr-1401-140130)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://<server-name>:10000/test_db > DROP TABLE IF EXISTS
demoHiveTable;
No rows affected (0.797 seconds)
org.apache.hive.jdbc.HiveStatement.getQueryLog()Ljava/util/List;
Closing: 0: jdbc:hive2://<server-name>:10000/test_db
I tried running the following commands
String[] args =
{"-d",BeeLine.BEELINE_DEFAULT_JDBC_DRIVER,"-u","jdbc:hive2://<server-name:1
0000/test_db",
"-n","******","-p","******",
"-f","C://Work//test_hive.hql"};
And
String[] args = {"beeline","-u","jdbc:hive2://
://<server-name:10000/test_db²,
"org.apache.hive.jdbc.HiveDriver","-n","******","-p","******","-f","C://Wor
k//test_hive.hql"};
3.
The link I referred was from googlesource
(https://apache.googlesource.com/hive/+/a681b57609ae306c7623491549cc0176b6c
df38d/beeline/src/test/org/apache/hive/beeline/src/test/TestBeeLineWithArgs
.java)
4.
I also see that there are errors being propagated back to the application.
Is there some configuration to get detailed logs too?
Not sure if I am missing something here.
Any pointers or assistance will be of great help.
Regards,
Amal
From: Mich Talebzadeh [mailto:mich@peridale.co.uk]
Sent: Saturday, March 21, 2015 6:59 PM
To: user@hive.apache.org
Subject: RE: Executing HQL files from JAVA application.
Hi Amal,
Do you have hiveserver2 running?
You can use beeline to execute the query outside of JAVA
beeline -u jdbc:hive2://rhes564:10010/default
org.apache.hive.jdbc.HiveDriver -n hduser -p xxxx' -f
./create_index_on_t.sql
And the output shows there as well.
scan complete in 10ms
Connecting to jdbc:hive2://rhes564:10010/default
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/hduser/hadoop/hadoop-2.6.0/share/hadoop/common/lib/slf4j-lo
g4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hive/lib/hive-jdbc-0.14.0-standalone.jar!/org/slf4j/impl
/StaticLoggerBinder.class]
SLF4J: See
http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Connected to: Apache Hive (version 0.14.0)
Driver: Hive JDBC (version 0.14.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://rhes564:10010/default> use asehadoop;
No rows affected (0.084 seconds)
0: jdbc:hive2://rhes564:10010/default> --drop index t_ui on t;
0: jdbc:hive2://rhes564:10010/default> create index t_ui on table t
(object_id) as 'COMPACT' WITH DEFERRED REBUILD;
HTH
Mich Talebzadeh
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and
Coherence Cache
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Ltd, its
subsidiaries or their employees, unless expressly so stated. It is the
responsibility of the recipient to ensure
that this email is virus free, therefore neither Peridale Ltd, its
subsidiaries nor their employees accept any responsibility.
From: Amal Gupta [mailto:Amal.Gupta1@aexp.com]
Sent: 21 March 2015 09:05
To: user@hive.apache.org
Subject: Executing HQL files from JAVA application.
Hi Everyone,
I am trying to execute a hive *.hql file from a java application. I had
tried a couple of ways of doing it through JDBC driver for hive and
through spring jdbc template
but yet, the only way which was successful for me was to create a runtime
process and then execute it.
The java code to do that is
Process p = Runtime.getRuntime().exec((new
String[]{"hive","-f","/hive/scripts/demoHiveTable.hql"}));
Although this works but, I wanted to know
if there is better way to do it using some driver functionality or any
other api for hive. The method I used will do the execution for me, but
yet, in case of failures while executing the hql script, the same will not
be visible to the Java application.
Any pointers or suggestions will be greatly helpful. Thanks in advance.
Please let me know in case you need any other details from me.
Regards,
Amal
________________________________________
American Express made the following annotations
________________________________________
"This message and any attachments are solely for the intended recipient
and may contain confidential or privileged information. If you are not the
intended recipient, any disclosure, copying, use, or distribution of the
information included in this message
and any attachments is prohibited. If you have received this
communication in error, please notify us by reply e-mail and immediately
and permanently delete this message and any attachments. Thank you."
American Express a ajouté le commentaire suivant le
Ce courrier et toute pièce jointe qu'il contient sont réservés au seul
destinataire indiqué et peuvent renfermer des renseignements confidentiels
et privilégiés. Si vous n'êtes pas le destinataire prévu, toute
divulgation, duplication, utilisation ou distribution
du courrier ou de toute pièce jointe est interdite. Si vous avez reçu
cette communication par erreur, veuillez nous en aviser par courrier et
détruire immédiatement le courrier et les pièces jointes. Merci.
________________________________________
RE: Executing HQL files from JAVA application.
Posted by Amal Gupta <Am...@aexp.com>.
Hey Mich,
Got any clues regarding the failure of the code that I sent?
I was going through the project and the code again and I suspect the mis-matching dependencies to be the culprits. I am currently trying to re-align the dependencies as per the pom given on the mvnrepository.com while trying to see if a particular configuration succeeds.
Will keep you posted on my progress. Thanks again for all the help that you are providing. :)
Regards,
Amal
From: Amal Gupta
Sent: Sunday, March 22, 2015 7:52 AM
To: user@hive.apache.org
Subject: RE: Executing HQL files from JAVA application.
Hi Mich,
:) A coincidence. Even I am new to hive. My test script which I am trying to execute contains a drop and a create statement.
Script :-
use test_db;
DROP TABLE IF EXISTS demoHiveTable;
CREATE EXTERNAL TABLE demoHiveTable (
demoId string,
demoName string
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'
STORED AS TEXTFILE LOCATION '/hive/';
Java Code: -
Not sure whether this will have an impact but the the code is a part of Spring batch Tasklet being triggered from the Batch-Context. This tasklet runs in parallel with other tasklets.
public RepeatStatus execute(StepContribution arg0, ChunkContext arg1)
throws Exception {
String[] args = {"-d",BeeLine.BEELINE_DEFAULT_JDBC_DRIVER,"-u","jdbc:hive2://<server-name>:10000/test_db",
"-n","******","-p","******", "-f","C://Work//test_hive.hql"};
BeeLine beeline = new BeeLine();
ByteArrayOutputStream os = new ByteArrayOutputStream();
PrintStream beelineOutputStream = new PrintStream(os);
beeline.setOutputStream(beelineOutputStream);
beeline.setErrorStream(beelineOutputStream);
beeline.begin(args,null);
String output = os.toString("UTF8");
System.out.println(output);
return RepeatStatus.FINISHED;
}
It will be great if you can share the piece of code that worked for you. May be it will give me some pointers on how to go ahead.
Best Regards,
Amal
From: Mich Talebzadeh [mailto:mich@peridale.co.uk]
Sent: Sunday, March 22, 2015 2:58 AM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: RE: Executing HQL files from JAVA application.
Hi Amal;
Me coming from relational database (Oracle, Sybase) background :) always expect that a DDL statement like DROP TABLE has to run in its own transaction and cannot be combined with a DML statement.
Now I suspect that when you run the command DROP TABLE IF EXIASTS <TABLE_NAME>; like below in beehive it works
0: jdbc:hive2://rhes564:10010/default> drop table if exists mytest;
No rows affected (0.216 seconds)
That runs in its own transaction so it works. However, I suspect in JAVA that is not the case. Can you possibly provide your JAVA code to see what exactly it is doing.
Thanks,
Mich
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
From: Amal Gupta [mailto:Amal.Gupta1@aexp.com]
Sent: 21 March 2015 18:16
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: RE: Executing HQL files from JAVA application.
Hi Mich,
Thank you for your response. I was not aware of beeline. I have now included this in my app and this looks a much better solution going forward. In the last couple of hours I have tried to work with beeline but have been facing some issues.
1. I was able to run on the remote server command line a beeline command given below . This was successful.
beeline -u jdbc:hive2://<server-name>:10000/test_db org.apache.hive.jdbc.HiveDriver -n ***** -p ****** -f /hive/scripts/demoHiveTable.hql
2. Running the same from the java app results in the issues. My script contains a drop table for the demoTable but the table is not dropped when running from java. (DROP TABLE IF EXISTS demoHiveTable;) . I see the following logs.
SLF4J: This version of SLF4J requires log4j version 1.2.12 or later. See also http://www.slf4j.org/codes.html#log4j_version
Exception in thread "Thread-1" java.lang.NoSuchMethodError: org.apache.hive.jdbc.HiveStatement.hasMoreLogs()Z
at org.apache.hive.beeline.Commands$1.run(Commands.java:839)
at java.lang.Thread.run(Thread.java:662)
Connecting to jdbc:hive2: ://<server-name>:10000/test_db
Connected to: Hive (version 0.12-mapr-1401-140130)
Driver: Hive (version 0.12-mapr-1401-140130)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://<server-name>:10000/test_db > DROP TABLE IF EXISTS demoHiveTable;
No rows affected (0.797 seconds)
org.apache.hive.jdbc.HiveStatement.getQueryLog()Ljava/util/List;
Closing: 0: jdbc:hive2://<server-name>:10000/test_db
I tried running the following commands
String[] args = {"-d",BeeLine.BEELINE_DEFAULT_JDBC_DRIVER,"-u","jdbc:hive2://<server-name:10000/test_db",
"-n","******","-p","******", "-f","C://Work//test_hive.hql"};
And
String[] args = {"beeline","-u","jdbc:hive2:// ://<server-name:10000/test_db",
"org.apache.hive.jdbc.HiveDriver","-n","******","-p","******","-f","C://Work//test_hive.hql"};
3. The link I referred was from googlesource (https://apache.googlesource.com/hive/+/a681b57609ae306c7623491549cc0176b6cdf38d/beeline/src/test/org/apache/hive/beeline/src/test/TestBeeLineWithArgs.java)
4. I also see that there are errors being propagated back to the application. Is there some configuration to get detailed logs too?
Not sure if I am missing something here.
Any pointers or assistance will be of great help.
Regards,
Amal
From: Mich Talebzadeh [mailto:mich@peridale.co.uk]
Sent: Saturday, March 21, 2015 6:59 PM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: RE: Executing HQL files from JAVA application.
Hi Amal,
Do you have hiveserver2 running?
You can use beeline to execute the query outside of JAVA
beeline -u jdbc:hive2://rhes564:10010/default org.apache.hive.jdbc.HiveDriver -n hduser -p xxxx' -f ./create_index_on_t.sql
And the output shows there as well.
scan complete in 10ms
Connecting to jdbc:hive2://rhes564:10010/default
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hduser/hadoop/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/hive-jdbc-0.14.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Connected to: Apache Hive (version 0.14.0)
Driver: Hive JDBC (version 0.14.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://rhes564:10010/default> use asehadoop;
No rows affected (0.084 seconds)
0: jdbc:hive2://rhes564:10010/default> --drop index t_ui on t;
0: jdbc:hive2://rhes564:10010/default> create index t_ui on table t (object_id) as 'COMPACT' WITH DEFERRED REBUILD;
HTH
Mich Talebzadeh
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
From: Amal Gupta [mailto:Amal.Gupta1@aexp.com]
Sent: 21 March 2015 09:05
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Executing HQL files from JAVA application.
Hi Everyone,
I am trying to execute a hive *.hql file from a java application. I had tried a couple of ways of doing it through JDBC driver for hive and through spring jdbc template but yet, the only way which was successful for me was to create a runtime process and then execute it.
The java code to do that is
Process p = Runtime.getRuntime().exec((new String[]{"hive","-f","/hive/scripts/demoHiveTable.hql"}));
Although this works but, I wanted to know if there is better way to do it using some driver functionality or any other api for hive. The method I used will do the execution for me, but yet, in case of failures while executing the hql script, the same will not be visible to the Java application.
Any pointers or suggestions will be greatly helpful. Thanks in advance.
Please let me know in case you need any other details from me.
Regards,
Amal
American Express made the following annotations
******************************************************************************
"This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you."
American Express a ajout� le commentaire suivant le Ce courrier et toute pi�ce jointe qu'il contient sont r�serv�s au seul destinataire indiqu� et peuvent renfermer des
renseignements confidentiels et privil�gi�s. Si vous n'�tes pas le destinataire pr�vu, toute divulgation, duplication, utilisation ou distribution du courrier ou de toute pi�ce jointe est interdite. Si vous avez re�u cette communication par erreur, veuillez nous en aviser par courrier et d�truire imm�diatement le courrier et les pi�ces jointes. Merci.
******************************************************************************
RE: Executing HQL files from JAVA application.
Posted by Amal Gupta <Am...@aexp.com>.
Hi Mich,
:) A coincidence. Even I am new to hive. My test script which I am trying to execute contains a drop and a create statement.
Script :-
use test_db;
DROP TABLE IF EXISTS demoHiveTable;
CREATE EXTERNAL TABLE demoHiveTable (
demoId string,
demoName string
) ROW FORMAT DELIMITED FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n'
STORED AS TEXTFILE LOCATION '/hive/';
Java Code: -
Not sure whether this will have an impact but the the code is a part of Spring batch Tasklet being triggered from the Batch-Context. This tasklet runs in parallel with other tasklets.
public RepeatStatus execute(StepContribution arg0, ChunkContext arg1)
throws Exception {
String[] args = {"-d",BeeLine.BEELINE_DEFAULT_JDBC_DRIVER,"-u","jdbc:hive2://<server-name>:10000/test_db",
"-n","******","-p","******", "-f","C://Work//test_hive.hql"};
BeeLine beeline = new BeeLine();
ByteArrayOutputStream os = new ByteArrayOutputStream();
PrintStream beelineOutputStream = new PrintStream(os);
beeline.setOutputStream(beelineOutputStream);
beeline.setErrorStream(beelineOutputStream);
beeline.begin(args,null);
String output = os.toString("UTF8");
System.out.println(output);
return RepeatStatus.FINISHED;
}
It will be great if you can share the piece of code that worked for you. May be it will give me some pointers on how to go ahead.
Best Regards,
Amal
From: Mich Talebzadeh [mailto:mich@peridale.co.uk]
Sent: Sunday, March 22, 2015 2:58 AM
To: user@hive.apache.org
Subject: RE: Executing HQL files from JAVA application.
Hi Amal;
Me coming from relational database (Oracle, Sybase) background :) always expect that a DDL statement like DROP TABLE has to run in its own transaction and cannot be combined with a DML statement.
Now I suspect that when you run the command DROP TABLE IF EXIASTS <TABLE_NAME>; like below in beehive it works
0: jdbc:hive2://rhes564:10010/default> drop table if exists mytest;
No rows affected (0.216 seconds)
That runs in its own transaction so it works. However, I suspect in JAVA that is not the case. Can you possibly provide your JAVA code to see what exactly it is doing.
Thanks,
Mich
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
From: Amal Gupta [mailto:Amal.Gupta1@aexp.com]
Sent: 21 March 2015 18:16
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: RE: Executing HQL files from JAVA application.
Hi Mich,
Thank you for your response. I was not aware of beeline. I have now included this in my app and this looks a much better solution going forward. In the last couple of hours I have tried to work with beeline but have been facing some issues.
1. I was able to run on the remote server command line a beeline command given below . This was successful.
beeline -u jdbc:hive2://<server-name>:10000/test_db org.apache.hive.jdbc.HiveDriver -n ***** -p ****** -f /hive/scripts/demoHiveTable.hql
2. Running the same from the java app results in the issues. My script contains a drop table for the demoTable but the table is not dropped when running from java. (DROP TABLE IF EXISTS demoHiveTable;) . I see the following logs.
SLF4J: This version of SLF4J requires log4j version 1.2.12 or later. See also http://www.slf4j.org/codes.html#log4j_version
Exception in thread "Thread-1" java.lang.NoSuchMethodError: org.apache.hive.jdbc.HiveStatement.hasMoreLogs()Z
at org.apache.hive.beeline.Commands$1.run(Commands.java:839)
at java.lang.Thread.run(Thread.java:662)
Connecting to jdbc:hive2: ://<server-name>:10000/test_db
Connected to: Hive (version 0.12-mapr-1401-140130)
Driver: Hive (version 0.12-mapr-1401-140130)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://<server-name>:10000/test_db > DROP TABLE IF EXISTS demoHiveTable;
No rows affected (0.797 seconds)
org.apache.hive.jdbc.HiveStatement.getQueryLog()Ljava/util/List;
Closing: 0: jdbc:hive2://<server-name>:10000/test_db
I tried running the following commands
String[] args = {"-d",BeeLine.BEELINE_DEFAULT_JDBC_DRIVER,"-u","jdbc:hive2://<server-name:10000/test_db",
"-n","******","-p","******", "-f","C://Work//test_hive.hql"};
And
String[] args = {"beeline","-u","jdbc:hive2:// ://<server-name:10000/test_db",
"org.apache.hive.jdbc.HiveDriver","-n","******","-p","******","-f","C://Work//test_hive.hql"};
3. The link I referred was from googlesource (https://apache.googlesource.com/hive/+/a681b57609ae306c7623491549cc0176b6cdf38d/beeline/src/test/org/apache/hive/beeline/src/test/TestBeeLineWithArgs.java)
4. I also see that there are errors being propagated back to the application. Is there some configuration to get detailed logs too?
Not sure if I am missing something here.
Any pointers or assistance will be of great help.
Regards,
Amal
From: Mich Talebzadeh [mailto:mich@peridale.co.uk]
Sent: Saturday, March 21, 2015 6:59 PM
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: RE: Executing HQL files from JAVA application.
Hi Amal,
Do you have hiveserver2 running?
You can use beeline to execute the query outside of JAVA
beeline -u jdbc:hive2://rhes564:10010/default org.apache.hive.jdbc.HiveDriver -n hduser -p xxxx' -f ./create_index_on_t.sql
And the output shows there as well.
scan complete in 10ms
Connecting to jdbc:hive2://rhes564:10010/default
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hduser/hadoop/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/hive-jdbc-0.14.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Connected to: Apache Hive (version 0.14.0)
Driver: Hive JDBC (version 0.14.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://rhes564:10010/default> use asehadoop;
No rows affected (0.084 seconds)
0: jdbc:hive2://rhes564:10010/default> --drop index t_ui on t;
0: jdbc:hive2://rhes564:10010/default> create index t_ui on table t (object_id) as 'COMPACT' WITH DEFERRED REBUILD;
HTH
Mich Talebzadeh
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
From: Amal Gupta [mailto:Amal.Gupta1@aexp.com]
Sent: 21 March 2015 09:05
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Executing HQL files from JAVA application.
Hi Everyone,
I am trying to execute a hive *.hql file from a java application. I had tried a couple of ways of doing it through JDBC driver for hive and through spring jdbc template but yet, the only way which was successful for me was to create a runtime process and then execute it.
The java code to do that is
Process p = Runtime.getRuntime().exec((new String[]{"hive","-f","/hive/scripts/demoHiveTable.hql"}));
Although this works but, I wanted to know if there is better way to do it using some driver functionality or any other api for hive. The method I used will do the execution for me, but yet, in case of failures while executing the hql script, the same will not be visible to the Java application.
Any pointers or suggestions will be greatly helpful. Thanks in advance.
Please let me know in case you need any other details from me.
Regards,
Amal
________________________________
American Express made the following annotations
________________________________
"This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you."
American Express a ajouté le commentaire suivant le
Ce courrier et toute pièce jointe qu'il contient sont réservés au seul destinataire indiqué et peuvent renfermer des renseignements confidentiels et privilégiés. Si vous n'êtes pas le destinataire prévu, toute divulgation, duplication, utilisation ou distribution du courrier ou de toute pièce jointe est interdite. Si vous avez reçu cette communication par erreur, veuillez nous en aviser par courrier et détruire immédiatement le courrier et les pièces jointes. Merci.
________________________________
American Express made the following annotations
******************************************************************************
"This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you."
American Express a ajouté le commentaire suivant le Ce courrier et toute pièce jointe qu'il contient sont réservés au seul destinataire indiqué et peuvent renfermer des
renseignements confidentiels et privilégiés. Si vous n'êtes pas le destinataire prévu, toute divulgation, duplication, utilisation ou distribution du courrier ou de toute pièce jointe est interdite. Si vous avez reçu cette communication par erreur, veuillez nous en aviser par courrier et détruire immédiatement le courrier et les pièces jointes. Merci.
******************************************************************************
RE: Executing HQL files from JAVA application.
Posted by Mich Talebzadeh <mi...@peridale.co.uk>.
Hi Amal;
Me coming from relational database (Oracle, Sybase) background J always
expect that a DDL statement like DROP TABLE has to run in its own
transaction and cannot be combined with a DML statement.
Now I suspect that when you run the command DROP TABLE IF EXIASTS
<TABLE_NAME>; like below in beehive it works
0: jdbc:hive2://rhes564:10010/default> drop table if exists mytest;
No rows affected (0.216 seconds)
That runs in its own transaction so it works. However, I suspect in JAVA
that is not the case. Can you possibly provide your JAVA code to see what
exactly it is doing.
Thanks,
Mich
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and
Coherence Cache
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Ltd, its
subsidiaries or their employees, unless expressly so stated. It is the
responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept
any responsibility.
From: Amal Gupta [mailto:Amal.Gupta1@aexp.com]
Sent: 21 March 2015 18:16
To: user@hive.apache.org
Subject: RE: Executing HQL files from JAVA application.
Hi Mich,
Thank you for your response. I was not aware of beeline. I have now
included this in my app and this looks a much better solution going forward.
In the last couple of hours I have tried to work with beeline but have been
facing some issues.
1. I was able to run on the remote server command line a beeline
command given below . This was successful.
beeline -u jdbc:hive2://<server-name>:10000/test_db
org.apache.hive.jdbc.HiveDriver -n ***** -p ****** -f
/hive/scripts/demoHiveTable.hql
2. Running the same from the java app results in the issues. My
script contains a drop table for the demoTable but the table is not dropped
when running from java. (DROP TABLE IF EXISTS demoHiveTable;) . I see the
following logs.
SLF4J: This version of SLF4J requires log4j version 1.2.12 or later. See
also http://www.slf4j.org/codes.html#log4j_version
Exception in thread "Thread-1" java.lang.NoSuchMethodError:
org.apache.hive.jdbc.HiveStatement.hasMoreLogs()Z
at org.apache.hive.beeline.Commands$1.run(Commands.java:839)
at java.lang.Thread.run(Thread.java:662)
Connecting to jdbc:hive2: ://<server-name>:10000/test_db
Connected to: Hive (version 0.12-mapr-1401-140130)
Driver: Hive (version 0.12-mapr-1401-140130)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://<server-name>:10000/test_db > DROP TABLE IF EXISTS
demoHiveTable;
No rows affected (0.797 seconds)
org.apache.hive.jdbc.HiveStatement.getQueryLog()Ljava/util/List;
Closing: 0: jdbc:hive2://<server-name>:10000/test_db
I tried running the following commands
String[] args =
{"-d",BeeLine.BEELINE_DEFAULT_JDBC_DRIVER,"-u","jdbc:hive2://<server-name:10
000/test_db",
"-n","******","-p","******",
"-f","C://Work//test_hive.hql"};
And
String[] args = {"beeline","-u","jdbc:hive2://
://<server-name:10000/test_db,
"org.apache.hive.jdbc.HiveDriver","-n","******","-p","******","-f","C://Work
//test_hive.hql"};
3. The link I referred was from googlesource
(https://apache.googlesource.com/hive/+/a681b57609ae306c7623491549cc0176b6cd
f38d/beeline/src/test/org/apache/hive/beeline/src/test/TestBeeLineWithArgs.j
ava)
4. I also see that there are errors being propagated back to the
application. Is there some configuration to get detailed logs too?
Not sure if I am missing something here.
Any pointers or assistance will be of great help.
Regards,
Amal
From: Mich Talebzadeh [mailto:mich@peridale.co.uk]
Sent: Saturday, March 21, 2015 6:59 PM
To: user@hive.apache.org
Subject: RE: Executing HQL files from JAVA application.
Hi Amal,
Do you have hiveserver2 running?
You can use beeline to execute the query outside of JAVA
beeline -u jdbc:hive2://rhes564:10010/default
org.apache.hive.jdbc.HiveDriver -n hduser -p xxxx' -f
./create_index_on_t.sql
And the output shows there as well.
scan complete in 10ms
Connecting to jdbc:hive2://rhes564:10010/default
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/hduser/hadoop/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log
4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hive/lib/hive-jdbc-0.14.0-standalone.jar!/org/slf4j/impl/
StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Connected to: Apache Hive (version 0.14.0)
Driver: Hive JDBC (version 0.14.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://rhes564:10010/default> use asehadoop;
No rows affected (0.084 seconds)
0: jdbc:hive2://rhes564:10010/default> --drop index t_ui on t;
0: jdbc:hive2://rhes564:10010/default> create index t_ui on table t
(object_id) as 'COMPACT' WITH DEFERRED REBUILD;
HTH
Mich Talebzadeh
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and
Coherence Cache
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Ltd, its
subsidiaries or their employees, unless expressly so stated. It is the
responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept
any responsibility.
From: Amal Gupta [mailto:Amal.Gupta1@aexp.com]
Sent: 21 March 2015 09:05
To: user@hive.apache.org
Subject: Executing HQL files from JAVA application.
Hi Everyone,
I am trying to execute a hive *.hql file from a java application. I had
tried a couple of ways of doing it through JDBC driver for hive and through
spring jdbc template but yet, the only way which was successful for me was
to create a runtime process and then execute it.
The java code to do that is
Process p = Runtime.getRuntime().exec((new
String[]{"hive","-f","/hive/scripts/demoHiveTable.hql"}));
Although this works but, I wanted to know if there is better way to do it
using some driver functionality or any other api for hive. The method I used
will do the execution for me, but yet, in case of failures while executing
the hql script, the same will not be visible to the Java application.
Any pointers or suggestions will be greatly helpful. Thanks in advance.
Please let me know in case you need any other details from me.
Regards,
Amal
_____
American Express made the following annotations
_____
"This message and any attachments are solely for the intended recipient and
may contain confidential or privileged information. If you are not the
intended recipient, any disclosure, copying, use, or distribution of the
information included in this message and any attachments is prohibited. If
you have received this communication in error, please notify us by reply
e-mail and immediately and permanently delete this message and any
attachments. Thank you."
American Express a ajouté le commentaire suivant le
Ce courrier et toute pièce jointe qu'il contient sont réservés au seul
destinataire indiqué et peuvent renfermer des renseignements confidentiels
et privilégiés. Si vous n'êtes pas le destinataire prévu, toute divulgation,
duplication, utilisation ou distribution du courrier ou de toute pièce
jointe est interdite. Si vous avez reçu cette communication par erreur,
veuillez nous en aviser par courrier et détruire immédiatement le courrier
et les pièces jointes. Merci.
_____
RE: Executing HQL files from JAVA application.
Posted by Amal Gupta <Am...@aexp.com>.
Hi Mich,
Thank you for your response. I was not aware of beeline. I have now included this in my app and this looks a much better solution going forward. In the last couple of hours I have tried to work with beeline but have been facing some issues.
1. I was able to run on the remote server command line a beeline command given below . This was successful.
beeline -u jdbc:hive2://<server-name>:10000/test_db org.apache.hive.jdbc.HiveDriver -n ***** -p ****** -f /hive/scripts/demoHiveTable.hql
2. Running the same from the java app results in the issues. My script contains a drop table for the demoTable but the table is not dropped when running from java. (DROP TABLE IF EXISTS demoHiveTable;) . I see the following logs.
SLF4J: This version of SLF4J requires log4j version 1.2.12 or later. See also http://www.slf4j.org/codes.html#log4j_version
Exception in thread "Thread-1" java.lang.NoSuchMethodError: org.apache.hive.jdbc.HiveStatement.hasMoreLogs()Z
at org.apache.hive.beeline.Commands$1.run(Commands.java:839)
at java.lang.Thread.run(Thread.java:662)
Connecting to jdbc:hive2: ://<server-name>:10000/test_db
Connected to: Hive (version 0.12-mapr-1401-140130)
Driver: Hive (version 0.12-mapr-1401-140130)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://<server-name>:10000/test_db > DROP TABLE IF EXISTS demoHiveTable;
No rows affected (0.797 seconds)
org.apache.hive.jdbc.HiveStatement.getQueryLog()Ljava/util/List;
Closing: 0: jdbc:hive2://<server-name>:10000/test_db
I tried running the following commands
String[] args = {"-d",BeeLine.BEELINE_DEFAULT_JDBC_DRIVER,"-u","jdbc:hive2://<server-name:10000/test_db",
"-n","******","-p","******", "-f","C://Work//test_hive.hql"};
And
String[] args = {"beeline","-u","jdbc:hive2:// ://<server-name:10000/test_db",
"org.apache.hive.jdbc.HiveDriver","-n","******","-p","******","-f","C://Work//test_hive.hql"};
3. The link I referred was from googlesource (https://apache.googlesource.com/hive/+/a681b57609ae306c7623491549cc0176b6cdf38d/beeline/src/test/org/apache/hive/beeline/src/test/TestBeeLineWithArgs.java)
4. I also see that there are errors being propagated back to the application. Is there some configuration to get detailed logs too?
Not sure if I am missing something here.
Any pointers or assistance will be of great help.
Regards,
Amal
From: Mich Talebzadeh [mailto:mich@peridale.co.uk]
Sent: Saturday, March 21, 2015 6:59 PM
To: user@hive.apache.org
Subject: RE: Executing HQL files from JAVA application.
Hi Amal,
Do you have hiveserver2 running?
You can use beeline to execute the query outside of JAVA
beeline -u jdbc:hive2://rhes564:10010/default org.apache.hive.jdbc.HiveDriver -n hduser -p xxxx' -f ./create_index_on_t.sql
And the output shows there as well.
scan complete in 10ms
Connecting to jdbc:hive2://rhes564:10010/default
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/home/hduser/hadoop/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/usr/lib/hive/lib/hive-jdbc-0.14.0-standalone.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Connected to: Apache Hive (version 0.14.0)
Driver: Hive JDBC (version 0.14.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://rhes564:10010/default> use asehadoop;
No rows affected (0.084 seconds)
0: jdbc:hive2://rhes564:10010/default> --drop index t_ui on t;
0: jdbc:hive2://rhes564:10010/default> create index t_ui on table t (object_id) as 'COMPACT' WITH DEFERRED REBUILD;
HTH
Mich Talebzadeh
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and Coherence Cache
NOTE: The information in this email is proprietary and confidential. This message is for the designated recipient only, if you are not the intended recipient, you should destroy it immediately. Any information in this message shall not be understood as given or endorsed by Peridale Ltd, its subsidiaries or their employees, unless expressly so stated. It is the responsibility of the recipient to ensure that this email is virus free, therefore neither Peridale Ltd, its subsidiaries nor their employees accept any responsibility.
From: Amal Gupta [mailto:Amal.Gupta1@aexp.com]
Sent: 21 March 2015 09:05
To: user@hive.apache.org<ma...@hive.apache.org>
Subject: Executing HQL files from JAVA application.
Hi Everyone,
I am trying to execute a hive *.hql file from a java application. I had tried a couple of ways of doing it through JDBC driver for hive and through spring jdbc template but yet, the only way which was successful for me was to create a runtime process and then execute it.
The java code to do that is
Process p = Runtime.getRuntime().exec((new String[]{"hive","-f","/hive/scripts/demoHiveTable.hql"}));
Although this works but, I wanted to know if there is better way to do it using some driver functionality or any other api for hive. The method I used will do the execution for me, but yet, in case of failures while executing the hql script, the same will not be visible to the Java application.
Any pointers or suggestions will be greatly helpful. Thanks in advance.
Please let me know in case you need any other details from me.
Regards,
Amal
American Express made the following annotations
******************************************************************************
"This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you."
American Express a ajout� le commentaire suivant le Ce courrier et toute pi�ce jointe qu'il contient sont r�serv�s au seul destinataire indiqu� et peuvent renfermer des
renseignements confidentiels et privil�gi�s. Si vous n'�tes pas le destinataire pr�vu, toute divulgation, duplication, utilisation ou distribution du courrier ou de toute pi�ce jointe est interdite. Si vous avez re�u cette communication par erreur, veuillez nous en aviser par courrier et d�truire imm�diatement le courrier et les pi�ces jointes. Merci.
******************************************************************************
RE: Executing HQL files from JAVA application.
Posted by Mich Talebzadeh <mi...@peridale.co.uk>.
Hi Amal,
Do you have hiveserver2 running?
You can use beeline to execute the query outside of JAVA
beeline -u jdbc:hive2://rhes564:10010/default
org.apache.hive.jdbc.HiveDriver -n hduser -p xxxx' -f
./create_index_on_t.sql
And the output shows there as well.
scan complete in 10ms
Connecting to jdbc:hive2://rhes564:10010/default
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/home/hduser/hadoop/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log
4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/lib/hive/lib/hive-jdbc-0.14.0-standalone.jar!/org/slf4j/impl/
StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Connected to: Apache Hive (version 0.14.0)
Driver: Hive JDBC (version 0.14.0)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://rhes564:10010/default> use asehadoop;
No rows affected (0.084 seconds)
0: jdbc:hive2://rhes564:10010/default> --drop index t_ui on t;
0: jdbc:hive2://rhes564:10010/default> create index t_ui on table t
(object_id) as 'COMPACT' WITH DEFERRED REBUILD;
HTH
Mich Talebzadeh
http://talebzadehmich.wordpress.com
Publications due shortly:
Creating in-memory Data Grid for Trading Systems with Oracle TimesTen and
Coherence Cache
NOTE: The information in this email is proprietary and confidential. This
message is for the designated recipient only, if you are not the intended
recipient, you should destroy it immediately. Any information in this
message shall not be understood as given or endorsed by Peridale Ltd, its
subsidiaries or their employees, unless expressly so stated. It is the
responsibility of the recipient to ensure that this email is virus free,
therefore neither Peridale Ltd, its subsidiaries nor their employees accept
any responsibility.
From: Amal Gupta [mailto:Amal.Gupta1@aexp.com]
Sent: 21 March 2015 09:05
To: user@hive.apache.org
Subject: Executing HQL files from JAVA application.
Hi Everyone,
I am trying to execute a hive *.hql file from a java application. I had
tried a couple of ways of doing it through JDBC driver for hive and through
spring jdbc template but yet, the only way which was successful for me was
to create a runtime process and then execute it.
The java code to do that is
Process p = Runtime.getRuntime().exec((new
String[]{"hive","-f","/hive/scripts/demoHiveTable.hql"}));
Although this works but, I wanted to know if there is better way to do it
using some driver functionality or any other api for hive. The method I used
will do the execution for me, but yet, in case of failures while executing
the hql script, the same will not be visible to the Java application.
Any pointers or suggestions will be greatly helpful. Thanks in advance.
Please let me know in case you need any other details from me.
Regards,
Amal
_____
American Express made the following annotations
_____
"This message and any attachments are solely for the intended recipient and
may contain confidential or privileged information. If you are not the
intended recipient, any disclosure, copying, use, or distribution of the
information included in this message and any attachments is prohibited. If
you have received this communication in error, please notify us by reply
e-mail and immediately and permanently delete this message and any
attachments. Thank you."
American Express a ajouté le commentaire suivant le
Ce courrier et toute pièce jointe qu'il contient sont réservés au seul
destinataire indiqué et peuvent renfermer des renseignements confidentiels
et privilégiés. Si vous n'êtes pas le destinataire prévu, toute divulgation,
duplication, utilisation ou distribution du courrier ou de toute pièce
jointe est interdite. Si vous avez reçu cette communication par erreur,
veuillez nous en aviser par courrier et détruire immédiatement le courrier
et les pièces jointes. Merci.
_____
RE: Executing HQL files from JAVA application.
Posted by Amal Gupta <Am...@aexp.com>.
Thanks lot Steve for your time and response.
I did try that out and was able to execute the basic select statements. But the concern I had is that the queries contained in the hql files are complicated queries with joins, grouping etc and they also do some insert operations. Essentially, I was a bit skeptical about doing their execution in the java app as I would just be bringing the computation logic to my application. Furthermore there would be multiple such scripts running in parallel from the same Java app.
Regards,
Amal
From: Steve Howard [mailto:stevedhoward@gmail.com]
Sent: Saturday, March 21, 2015 3:03 PM
To: user@hive.apache.org
Subject: Re: Executing HQL files from JAVA application.
There are more elegant ways I am sure, but you could also use a java.io.BufferedReader and read the file content into a string and execute it much as you would a hard coded SQL statement in your class.
Sent from my iPad
On Mar 21, 2015, at 5:04 AM, Amal Gupta <Am...@aexp.com>> wrote:
Hi Everyone,
I am trying to execute a hive *.hql file from a java application. I had tried a couple of ways of doing it through JDBC driver for hive and through spring jdbc template but yet, the only way which was successful for me was to create a runtime process and then execute it.
The java code to do that is
Process p = Runtime.getRuntime().exec((new String[]{"hive","-f","/hive/scripts/demoHiveTable.hql"}));
Although this works but, I wanted to know if there is better way to do it using some driver functionality or any other api for hive. The method I used will do the execution for me, but yet, in case of failures while executing the hql script, the same will not be visible to the Java application.
Any pointers or suggestions will be greatly helpful. Thanks in advance.
Please let me know in case you need any other details from me.
Regards,
Amal
________________________________
American Express made the following annotations
________________________________
"This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you."
American Express a ajouté le commentaire suivant le
Ce courrier et toute pièce jointe qu'il contient sont réservés au seul destinataire indiqué et peuvent renfermer des renseignements confidentiels et privilégiés. Si vous n'êtes pas le destinataire prévu, toute divulgation, duplication, utilisation ou distribution du courrier ou de toute pièce jointe est interdite. Si vous avez reçu cette communication par erreur, veuillez nous en aviser par courrier et détruire immédiatement le courrier et les pièces jointes. Merci.
________________________________
American Express made the following annotations
******************************************************************************
"This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you."
American Express a ajouté le commentaire suivant le Ce courrier et toute pièce jointe qu'il contient sont réservés au seul destinataire indiqué et peuvent renfermer des
renseignements confidentiels et privilégiés. Si vous n'êtes pas le destinataire prévu, toute divulgation, duplication, utilisation ou distribution du courrier ou de toute pièce jointe est interdite. Si vous avez reçu cette communication par erreur, veuillez nous en aviser par courrier et détruire immédiatement le courrier et les pièces jointes. Merci.
******************************************************************************
Re: Executing HQL files from JAVA application.
Posted by Steve Howard <st...@gmail.com>.
There are more elegant ways I am sure, but you could also use a java.io.BufferedReader and read the file content into a string and execute it much as you would a hard coded SQL statement in your class.
Sent from my iPad
> On Mar 21, 2015, at 5:04 AM, Amal Gupta <Am...@aexp.com> wrote:
>
> Hi Everyone,
>
> I am trying to execute a hive *.hql file from a java application. I had tried a couple of ways of doing it through JDBC driver for hive and through spring jdbc template but yet, the only way which was successful for me was to create a runtime process and then execute it.
>
> The java code to do that is
>
> Process p = Runtime.getRuntime().exec((new String[]{"hive","-f","/hive/scripts/demoHiveTable.hql"}));
>
> Although this works but, I wanted to know if there is better way to do it using some driver functionality or any other api for hive. The method I used will do the execution for me, but yet, in case of failures while executing the hql script, the same will not be visible to the Java application.
>
> Any pointers or suggestions will be greatly helpful. Thanks in advance.
>
> Please let me know in case you need any other details from me.
>
> Regards,
> Amal
> American Express made the following annotations
>
> "This message and any attachments are solely for the intended recipient and may contain confidential or privileged information. If you are not the intended recipient, any disclosure, copying, use, or distribution of the information included in this message and any attachments is prohibited. If you have received this communication in error, please notify us by reply e-mail and immediately and permanently delete this message and any attachments. Thank you."
>
> American Express a ajouté le commentaire suivant le
> Ce courrier et toute pièce jointe qu'il contient sont réservés au seul destinataire indiqué et peuvent renfermer des renseignements confidentiels et privilégiés. Si vous n'êtes pas le destinataire prévu, toute divulgation, duplication, utilisation ou distribution du courrier ou de toute pièce jointe est interdite. Si vous avez reçu cette communication par erreur, veuillez nous en aviser par courrier et détruire immédiatement le courrier et les pièces jointes. Merci.