You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Zsolt Miskolczi <zs...@gmail.com> on 2023/08/15 15:07:48 UTC
MiniHS2 and postgresql jars
Hey there!Do you know how it is possible to use minihs2 with embedded
metastore service but postgresql as the metastore database?
Since I'm pretty sure it broke with that change
https://github.com/apache/hive/pull/4317, handling drivers is changed:
We don't bundle postgresql drivers with the hive build.Firstly, when I copy
the postgresql driver jar into the
packaging/target/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/lib
directory, I can run the schematool as it sees the driver.
The real issue is with miniHS2Cluster: it is basically a maven test and I
have really no idea how I can pass the driver to the test. If it would be a
simple java command, I would try to pass the -cp argument but I see no
other way to add extra jar file to the test execution.I want to run minihs2
with this command: mvn test -Dtest=StartMiniHS2Cluster -DminiHS2.run=true
-DminiHS2.usePortsFromConf=true -DminiHS2.clusterType=LLAP
-DminiHS2.conf="../../data/conf/llap/hive-site.xml"
-Dpackaging.minimizeJar=false -DskipShade -Dremoteresources.skip=true
-Dmaven.javadoc.skip=true -Denforcer.skip=trueAnd get that error (it seems
the metastore client cannot find the proper driver):
java.lang.RuntimeException: Error applying authorization policy on hive
configuration: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hive.service.cli.CLIService.init(CLIService.java:122)
at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:243)
at org.apache.hive.jdbc.miniHS2.MiniHS2.start(MiniHS2.java:394)
at
org.apache.hive.jdbc.miniHS2.StartMiniHS2Cluster.testRunCluster(StartMiniHS2Cluster.java:78)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
at
org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
Caused by: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at
org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:1001)
at
org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:2003)
at
org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:135)
at org.apache.hive.service.cli.CLIService.init(CLIService.java:119)
... 32 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at
org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1033)
at
org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:993)
... 35 more
Caused by: java.lang.RuntimeException: Unable to instantiate
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at
org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:88)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:96)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:149)
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:120)
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:5769)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5847)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5827)
at
org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1029)
... 36 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86)
... 43 more
Caused by: MetaException(message:Got exception:
org.apache.hadoop.hive.metastore.api.MetaException java.sql.SQLException:
No suitable driver)
at
org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.throwMetaException(MetaStoreUtils.java:150)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.callEmbeddedMetastore(HiveMetaStoreClient.java:309)
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:220)
at
org.apache.hadoop.hive.ql.metadata.HiveMetaStoreClientWithLocalCache.<init>(HiveMetaStoreClientWithLocalCache.java:120)
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:154)
... 48 more
Any help is appreciated,
Thank you
Re: MiniHS2 and postgresql jars
Posted by Zoltán Rátkai <zr...@cloudera.com.INVALID>.
Hi,
if I understand correctly the descriptions this would be no problem for the
followings ?
- Postgres
- MSSQL
- Derby
"MariaDB, MySQL, and Oracle licenses are not compatible with Apache License
2 "
I created a ticket to solve this issue:
https://issues.apache.org/jira/browse/HIVE-27642
If we can agree on to restore for Postgres then I will create a PR.
Regards,
Zoltán Rátkai
On Wed, Aug 23, 2023 at 11:23 AM Stamatis Zampetakis <za...@gmail.com>
wrote:
> Hello,
>
> For those interested about the full background behind the changes that
> partially broke StartMiniHS2Cluster please have a look at [1, 2, 3].
> If something is not clear or we need to revisit those decisions I am
> happy to discuss further.
>
> Summarizing in one sentence, the bin.tar.gz and src.tag.gz release
> artifacts should never contain jars or code which have licenses in
> Category-X [4]. If we want to use or rely on such code things are a
> bit more flexible as described in [4].
>
> Best,
> Stamatis
>
> [1] https://lists.apache.org/thread/xd25nhox103t2zj52lnzbjkm6d41ls94
> [2] https://issues.apache.org/jira/browse/HIVE-25701
> [3] https://issues.apache.org/jira/browse/HIVE-27338
> [4] https://www.apache.org/legal/resolved.html#category-x
>
> On Wed, Aug 23, 2023 at 11:44 AM Zsolt Miskolczi
> <zs...@gmail.com> wrote:
> >
> > As I understand, they have been removed due to licensing issues. Before
> we
> > re-add the jar or add a possibility to include jars into the classpath, I
> > want to understand the reason. What was the exact licensing issue?
> >
> > Do we have to make sure that we don't provide those jars in production?
> Or
> > is it necessary for tests as well?
> >
> > For me, the solution would be something that I, as a developer, have to
> set
> > up one time only. Based on my current knowledge, it seems the ability to
> > add external jars to the classpath could be more comfortable if we cannot
> > provide the jars into the production.
> > The reason is that we need the jdbc driver at two places:
> > - for MiniHS2
> > - and in the conf directory, next to bin (to be able to easily use it for
> > schema tool, beeline and standalone metastore service).
> >
> >
> > On the other hand, I'm not completely sure if MiniHS2 as a test is the
> most
> > comfortable idea. In the long term, I would prefer to replace it with a
> run
> > script/configuration that runs Hive Server in a 'Mini' mode, for local
> > development purposes.
> >
> > Thanks,
> > Zsolt
> >
> >
> > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023. aug.
> 22.,
> > K, 18:06):
> >
> > > I am not against restoring itest.jdbc.jars property but for this case
> > > I prefer the explicit declaration of the dependency.
> > >
> > > Adding an optional or test scope dependency is much simpler and works
> > > out of the box. We don't need to download jars manually and we don't
> > > need to remember how the system property is called in order to run the
> > > test.
> > >
> > > Anyways, we all agree that we want StartMiniHS2Cluster to run on
> > > different metastore backends so we can create a JIRA/PR and move this
> > > forward. How we are going to do it is implementation details so we can
> > > continue the discussion under the respective ticket.
> > >
> > > Best,
> > > Stamatis
> > >
> > > On Tue, Aug 22, 2023 at 6:46 PM László Bodor <
> bodorlaszlo0202@gmail.com>
> > > wrote:
> > > >
> > > > Yeah, I think we should simply re-add the possibility to add jars to
> > > > classpath, call it "itest.jdbc.jars" to preserve the old behavior and
> > > > parameter name.
> > > >
> > > > Denys Kuzmenko <dk...@apache.org> ezt írta (időpont: 2023. aug.
> > > 22., K,
> > > > 12:43):
> > > >
> > > > > Instead of adding the dependencies, can't we add the possibility to
> > > > > include jdbc jars in the classpath?
> > > > > something like this:
> > > > > <plugin>
> > > > > <groupId>org.apache.maven.plugins</groupId>
> > > > > <artifactId>maven-failsafe-plugin</artifactId>
> > > > > <executions>
> > > > > <execution>
> > > > > <goals>
> > > > > <goal>integration-test</goal>
> > > > > <goal>verify</goal>
> > > > > </goals>
> > > > > </execution>
> > > > > </executions>
> > > > > <configuration>
> > > > > ....
> > > > > <additionalClasspathElements>
> > > > >
> > > > >
> > >
> <additionalClasspathElement>${itest.jdbc.jars}</additionalClasspathElement>
> > > > > </additionalClasspathElements>
> > > > > ...
> > > > > </configuration>
> > > > > </plugin>
> > > > >
> > >
>
Re: MiniHS2 and postgresql jars
Posted by Stamatis Zampetakis <za...@gmail.com>.
Hello,
For those interested about the full background behind the changes that
partially broke StartMiniHS2Cluster please have a look at [1, 2, 3].
If something is not clear or we need to revisit those decisions I am
happy to discuss further.
Summarizing in one sentence, the bin.tar.gz and src.tag.gz release
artifacts should never contain jars or code which have licenses in
Category-X [4]. If we want to use or rely on such code things are a
bit more flexible as described in [4].
Best,
Stamatis
[1] https://lists.apache.org/thread/xd25nhox103t2zj52lnzbjkm6d41ls94
[2] https://issues.apache.org/jira/browse/HIVE-25701
[3] https://issues.apache.org/jira/browse/HIVE-27338
[4] https://www.apache.org/legal/resolved.html#category-x
On Wed, Aug 23, 2023 at 11:44 AM Zsolt Miskolczi
<zs...@gmail.com> wrote:
>
> As I understand, they have been removed due to licensing issues. Before we
> re-add the jar or add a possibility to include jars into the classpath, I
> want to understand the reason. What was the exact licensing issue?
>
> Do we have to make sure that we don't provide those jars in production? Or
> is it necessary for tests as well?
>
> For me, the solution would be something that I, as a developer, have to set
> up one time only. Based on my current knowledge, it seems the ability to
> add external jars to the classpath could be more comfortable if we cannot
> provide the jars into the production.
> The reason is that we need the jdbc driver at two places:
> - for MiniHS2
> - and in the conf directory, next to bin (to be able to easily use it for
> schema tool, beeline and standalone metastore service).
>
>
> On the other hand, I'm not completely sure if MiniHS2 as a test is the most
> comfortable idea. In the long term, I would prefer to replace it with a run
> script/configuration that runs Hive Server in a 'Mini' mode, for local
> development purposes.
>
> Thanks,
> Zsolt
>
>
> Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023. aug. 22.,
> K, 18:06):
>
> > I am not against restoring itest.jdbc.jars property but for this case
> > I prefer the explicit declaration of the dependency.
> >
> > Adding an optional or test scope dependency is much simpler and works
> > out of the box. We don't need to download jars manually and we don't
> > need to remember how the system property is called in order to run the
> > test.
> >
> > Anyways, we all agree that we want StartMiniHS2Cluster to run on
> > different metastore backends so we can create a JIRA/PR and move this
> > forward. How we are going to do it is implementation details so we can
> > continue the discussion under the respective ticket.
> >
> > Best,
> > Stamatis
> >
> > On Tue, Aug 22, 2023 at 6:46 PM László Bodor <bo...@gmail.com>
> > wrote:
> > >
> > > Yeah, I think we should simply re-add the possibility to add jars to
> > > classpath, call it "itest.jdbc.jars" to preserve the old behavior and
> > > parameter name.
> > >
> > > Denys Kuzmenko <dk...@apache.org> ezt írta (időpont: 2023. aug.
> > 22., K,
> > > 12:43):
> > >
> > > > Instead of adding the dependencies, can't we add the possibility to
> > > > include jdbc jars in the classpath?
> > > > something like this:
> > > > <plugin>
> > > > <groupId>org.apache.maven.plugins</groupId>
> > > > <artifactId>maven-failsafe-plugin</artifactId>
> > > > <executions>
> > > > <execution>
> > > > <goals>
> > > > <goal>integration-test</goal>
> > > > <goal>verify</goal>
> > > > </goals>
> > > > </execution>
> > > > </executions>
> > > > <configuration>
> > > > ....
> > > > <additionalClasspathElements>
> > > >
> > > >
> > <additionalClasspathElement>${itest.jdbc.jars}</additionalClasspathElement>
> > > > </additionalClasspathElements>
> > > > ...
> > > > </configuration>
> > > > </plugin>
> > > >
> >
Re: MiniHS2 and postgresql jars
Posted by Zsolt Miskolczi <zs...@gmail.com>.
As I understand, they have been removed due to licensing issues. Before we
re-add the jar or add a possibility to include jars into the classpath, I
want to understand the reason. What was the exact licensing issue?
Do we have to make sure that we don't provide those jars in production? Or
is it necessary for tests as well?
For me, the solution would be something that I, as a developer, have to set
up one time only. Based on my current knowledge, it seems the ability to
add external jars to the classpath could be more comfortable if we cannot
provide the jars into the production.
The reason is that we need the jdbc driver at two places:
- for MiniHS2
- and in the conf directory, next to bin (to be able to easily use it for
schema tool, beeline and standalone metastore service).
On the other hand, I'm not completely sure if MiniHS2 as a test is the most
comfortable idea. In the long term, I would prefer to replace it with a run
script/configuration that runs Hive Server in a 'Mini' mode, for local
development purposes.
Thanks,
Zsolt
Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023. aug. 22.,
K, 18:06):
> I am not against restoring itest.jdbc.jars property but for this case
> I prefer the explicit declaration of the dependency.
>
> Adding an optional or test scope dependency is much simpler and works
> out of the box. We don't need to download jars manually and we don't
> need to remember how the system property is called in order to run the
> test.
>
> Anyways, we all agree that we want StartMiniHS2Cluster to run on
> different metastore backends so we can create a JIRA/PR and move this
> forward. How we are going to do it is implementation details so we can
> continue the discussion under the respective ticket.
>
> Best,
> Stamatis
>
> On Tue, Aug 22, 2023 at 6:46 PM László Bodor <bo...@gmail.com>
> wrote:
> >
> > Yeah, I think we should simply re-add the possibility to add jars to
> > classpath, call it "itest.jdbc.jars" to preserve the old behavior and
> > parameter name.
> >
> > Denys Kuzmenko <dk...@apache.org> ezt írta (időpont: 2023. aug.
> 22., K,
> > 12:43):
> >
> > > Instead of adding the dependencies, can't we add the possibility to
> > > include jdbc jars in the classpath?
> > > something like this:
> > > <plugin>
> > > <groupId>org.apache.maven.plugins</groupId>
> > > <artifactId>maven-failsafe-plugin</artifactId>
> > > <executions>
> > > <execution>
> > > <goals>
> > > <goal>integration-test</goal>
> > > <goal>verify</goal>
> > > </goals>
> > > </execution>
> > > </executions>
> > > <configuration>
> > > ....
> > > <additionalClasspathElements>
> > >
> > >
> <additionalClasspathElement>${itest.jdbc.jars}</additionalClasspathElement>
> > > </additionalClasspathElements>
> > > ...
> > > </configuration>
> > > </plugin>
> > >
>
Re: MiniHS2 and postgresql jars
Posted by Stamatis Zampetakis <za...@gmail.com>.
I am not against restoring itest.jdbc.jars property but for this case
I prefer the explicit declaration of the dependency.
Adding an optional or test scope dependency is much simpler and works
out of the box. We don't need to download jars manually and we don't
need to remember how the system property is called in order to run the
test.
Anyways, we all agree that we want StartMiniHS2Cluster to run on
different metastore backends so we can create a JIRA/PR and move this
forward. How we are going to do it is implementation details so we can
continue the discussion under the respective ticket.
Best,
Stamatis
On Tue, Aug 22, 2023 at 6:46 PM László Bodor <bo...@gmail.com> wrote:
>
> Yeah, I think we should simply re-add the possibility to add jars to
> classpath, call it "itest.jdbc.jars" to preserve the old behavior and
> parameter name.
>
> Denys Kuzmenko <dk...@apache.org> ezt írta (időpont: 2023. aug. 22., K,
> 12:43):
>
> > Instead of adding the dependencies, can't we add the possibility to
> > include jdbc jars in the classpath?
> > something like this:
> > <plugin>
> > <groupId>org.apache.maven.plugins</groupId>
> > <artifactId>maven-failsafe-plugin</artifactId>
> > <executions>
> > <execution>
> > <goals>
> > <goal>integration-test</goal>
> > <goal>verify</goal>
> > </goals>
> > </execution>
> > </executions>
> > <configuration>
> > ....
> > <additionalClasspathElements>
> >
> > <additionalClasspathElement>${itest.jdbc.jars}</additionalClasspathElement>
> > </additionalClasspathElements>
> > ...
> > </configuration>
> > </plugin>
> >
Re: MiniHS2 and postgresql jars
Posted by László Bodor <bo...@gmail.com>.
Yeah, I think we should simply re-add the possibility to add jars to
classpath, call it "itest.jdbc.jars" to preserve the old behavior and
parameter name.
Denys Kuzmenko <dk...@apache.org> ezt írta (időpont: 2023. aug. 22., K,
12:43):
> Instead of adding the dependencies, can't we add the possibility to
> include jdbc jars in the classpath?
> something like this:
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-failsafe-plugin</artifactId>
> <executions>
> <execution>
> <goals>
> <goal>integration-test</goal>
> <goal>verify</goal>
> </goals>
> </execution>
> </executions>
> <configuration>
> ....
> <additionalClasspathElements>
>
> <additionalClasspathElement>${itest.jdbc.jars}</additionalClasspathElement>
> </additionalClasspathElements>
> ...
> </configuration>
> </plugin>
>
Re: MiniHS2 and postgresql jars
Posted by Denys Kuzmenko <dk...@apache.org>.
Instead of adding the dependencies, can't we add the possibility to include jdbc jars in the classpath?
something like this:
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-failsafe-plugin</artifactId>
<executions>
<execution>
<goals>
<goal>integration-test</goal>
<goal>verify</goal>
</goals>
</execution>
</executions>
<configuration>
....
<additionalClasspathElements>
<additionalClasspathElement>${itest.jdbc.jars}</additionalClasspathElement>
</additionalClasspathElements>
...
</configuration>
</plugin>
Re: MiniHS2 and postgresql jars
Posted by Stamatis Zampetakis <za...@gmail.com>.
If people are using it then it makes sense to keep it functional.
Feel free to open a PR and add the missing dependency(ies) in the
appropriate place; just ensure that they are not widely propagated if
not necessary.
Best,
Stamatis
On Mon, Aug 21, 2023 at 12:04 PM Zoltán Rátkai
<zr...@cloudera.com.invalid> wrote:
>
> Hi Team,
>
> I agree with Laszlo Bodor, it is a good tool to develop Hive. It worked
> properly before this PR.
> My question is how do you provide the missing database jar (e.g. Postgres
> driver) for StartMiniHS2Cluster, which is run by maven? If it requires to
> modify the pom file after this change, then there is no difference then
> just alway reverting this PR mentioned earlier, but it has the risk this
> will be committed together with the actual work a developer does. I think
> this is a wrong approach to modify a pom or java file to be able to run it.
> We must be able to run it without alway modifying.
>
> Regards,
>
> Zoltan Ratkai
>
> On Mon, Aug 21, 2023 at 10:44 AM László Bodor <bo...@gmail.com>
> wrote:
>
> > Hey!
> >
> > I tend to return to StartMiniHS2Cluster from time to time. It's very good
> > for the "change code - compile - run - debug - repeat" way of doing things.
> > From this point of view, docker image is not an alternative to that. Also,
> > StartMiniHS2Cluster just works, always, moreover, it uses the same
> > minicluster architecture as our qtests (I mean the way it achieves a mini
> > mr/tez/llap/whatever cluster from the hadoop shim). Qtests are also not an
> > alternative: I often need the ability to run queries in random order or in
> > adhoc way without editing a qfile.
> >
> > I feel that HIVE-27338 <https://issues.apache.org/jira/browse/HIVE-27338>
> > was
> > about to solve licensing problems, and it achieved that, but maybe went too
> > far: I believe we should be able to provide jars to the test classpath,
> > e.g. jars that were downloaded beforehand, even manually.
> >
> > Regards,
> > Laszlo Bodor
> >
> >
> > Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023. aug. 18.,
> > P, 13:08):
> >
> > > Hey Zsolt,
> > >
> > > I would divide this discussion into three topics:
> > >
> > > 1. What are the benefits of using the StartMiniHS2Cluster?
> > > 2. What other alternatives are there for testing HS2 with different
> > > metastore DBMS?
> > > 3. How can we make StartMiniHS2Cluster work as before?
> > >
> > > Regarding the point 1, I don't have an answer because I never used
> > > StartMiniHS2Cluster myself. Obviously, other people here are using it so
> > I
> > > would be curious to know in which cases this is useful.
> > >
> > > There are various alternatives for testing HS2 with different metastore
> > > DBMS.
> > >
> > > The first and in my opinion the easiest to use is the classic qtest
> > > framework with the various CLI drivers. Basically, with the work that
> > > Laszlo started in HIVE-21954 it is pretty easy to run any kind of test
> > over
> > > any metastore just by setting the respective system property.
> > >
> > > mvn test -Dtest=TestMiniLlapLocalCliDriver
> > > -Dqfile=partition_params_postgres.q -Dtest.metastore.db=postgres
> > >
> > > The second is to use our brand new and shiny docker images contributed by
> > > Zhihua in HIVE-26400 and get the real feel of HS2 and HMS in a prod-like
> > > setup. I haven't played much with this in apache/master but I did use
> > some
> > > similar images in our internal forks and it's pretty easy to get it up
> > and
> > > running.
> > >
> > > The third is the old and classic hive-dev-box [1] started by Zoltan. With
> > > two/three commands you have a Hive cluster like environment and of course
> > > you can choose which DBMS you want for the metastore.
> > >
> > > Regarding point 3, I assume that it is pretty easy to fix by adding the
> > > postgresql (or other JDBC driver) dependency inside the hive-it-unit
> > > module.
> > >
> > > <dependency>
> > > <groupId>org.postgresql</groupId>
> > > <artifactId>postgresql</artifactId>
> > > <optional>true</optional>
> > > </dependency>
> > >
> > > Given that we have other alternatives do we really need to go into this
> > > direction? In fact, do we really need the StartMiniHS2Cluster class?
> > >
> > > Best,
> > > Stamatis
> > >
> > > [1] https://github.com/kgyrtkirk/hive-dev-box
> > >
> > > On Tue, Aug 15, 2023, 6:08 PM Zsolt Miskolczi <zsolt.miskolczi@gmail.com
> > >
> > > wrote:
> > >
> > > > Hey there!Do you know how it is possible to use minihs2 with embedded
> > > > metastore service but postgresql as the metastore database?
> > > >
> > > > Since I'm pretty sure it broke with that change
> > > > https://github.com/apache/hive/pull/4317, handling drivers is changed:
> > > >
> > > > We don't bundle postgresql drivers with the hive build.Firstly, when I
> > > copy
> > > > the postgresql driver jar into the
> > > >
> > > >
> > >
> > packaging/target/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/lib
> > > > directory, I can run the schematool as it sees the driver.
> > > > The real issue is with miniHS2Cluster: it is basically a maven test
> > and I
> > > > have really no idea how I can pass the driver to the test. If it would
> > > be a
> > > > simple java command, I would try to pass the -cp argument but I see no
> > > > other way to add extra jar file to the test execution.I want to run
> > > minihs2
> > > > with this command: mvn test -Dtest=StartMiniHS2Cluster
> > -DminiHS2.run=true
> > > > -DminiHS2.usePortsFromConf=true -DminiHS2.clusterType=LLAP
> > > > -DminiHS2.conf="../../data/conf/llap/hive-site.xml"
> > > > -Dpackaging.minimizeJar=false -DskipShade -Dremoteresources.skip=true
> > > > -Dmaven.javadoc.skip=true -Denforcer.skip=trueAnd get that error (it
> > > seems
> > > > the metastore client cannot find the proper driver):
> > > >
> > > > java.lang.RuntimeException: Error applying authorization policy on hive
> > > > configuration: org.apache.hadoop.hive.ql.metadata.HiveException: Unable
> > > to
> > > > instantiate
> > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> > > > at org.apache.hive.service.cli.CLIService.init(CLIService.java:122)
> > > > at
> > > org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> > > > at
> > org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:243)
> > > > at org.apache.hive.jdbc.miniHS2.MiniHS2.start(MiniHS2.java:394)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hive.jdbc.miniHS2.StartMiniHS2Cluster.testRunCluster(StartMiniHS2Cluster.java:78)
> > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > at
> > > >
> > > >
> > >
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > at
> > > >
> > > >
> > >
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > at
> > > >
> > > >
> > >
> > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> > > > at
> > > >
> > > >
> > >
> > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> > > > at
> > > >
> > > >
> > >
> > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> > > > at
> > > >
> > > >
> > >
> > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> > > > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> > > > at
> > > >
> > > >
> > >
> > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> > > > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> > > > at
> > > >
> > > >
> > >
> > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> > > > at
> > > >
> > > >
> > >
> > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> > > > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> > > > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> > > > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> > > > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> > > > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> > > > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> > > > at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> > > > at
> > > >
> > > >
> > >
> > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> > > > at
> > > >
> > > >
> > >
> > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> > > > at
> > > >
> > > >
> > >
> > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> > > > at
> > > >
> > > >
> > >
> > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> > > > at
> > > >
> > > >
> > >
> > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
> > > > at
> > > >
> > > >
> > >
> > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
> > > > at
> > > org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
> > > > at
> > > >
> > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
> > > > Caused by: java.lang.RuntimeException:
> > > > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to instantiate
> > > > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:1001)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:2003)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:135)
> > > > at org.apache.hive.service.cli.CLIService.init(CLIService.java:119)
> > > > ... 32 more
> > > > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
> > > > instantiate
> > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1033)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:993)
> > > > ... 35 more
> > > > Caused by: java.lang.RuntimeException: Unable to instantiate
> > > > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:88)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:96)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:149)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:120)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:5769)
> > > > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5847)
> > > > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5827)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1029)
> > > > ... 36 more
> > > > Caused by: java.lang.reflect.InvocationTargetException
> > > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> > Method)
> > > > at
> > > >
> > > >
> > >
> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> > > > at
> > > >
> > > >
> > >
> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > > > at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86)
> > > > ... 43 more
> > > > Caused by: MetaException(message:Got exception:
> > > > org.apache.hadoop.hive.metastore.api.MetaException
> > java.sql.SQLException:
> > > > No suitable driver)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.throwMetaException(MetaStoreUtils.java:150)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.callEmbeddedMetastore(HiveMetaStoreClient.java:309)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:220)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.ql.metadata.HiveMetaStoreClientWithLocalCache.<init>(HiveMetaStoreClientWithLocalCache.java:120)
> > > > at
> > > >
> > > >
> > >
> > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:154)
> > > > ... 48 more
> > > >
> > > >
> > > > Any help is appreciated,
> > > > Thank you
> > > >
> > >
> >
Re: MiniHS2 and postgresql jars
Posted by Zoltán Rátkai <zr...@cloudera.com.INVALID>.
Hi Team,
I agree with Laszlo Bodor, it is a good tool to develop Hive. It worked
properly before this PR.
My question is how do you provide the missing database jar (e.g. Postgres
driver) for StartMiniHS2Cluster, which is run by maven? If it requires to
modify the pom file after this change, then there is no difference then
just alway reverting this PR mentioned earlier, but it has the risk this
will be committed together with the actual work a developer does. I think
this is a wrong approach to modify a pom or java file to be able to run it.
We must be able to run it without alway modifying.
Regards,
Zoltan Ratkai
On Mon, Aug 21, 2023 at 10:44 AM László Bodor <bo...@gmail.com>
wrote:
> Hey!
>
> I tend to return to StartMiniHS2Cluster from time to time. It's very good
> for the "change code - compile - run - debug - repeat" way of doing things.
> From this point of view, docker image is not an alternative to that. Also,
> StartMiniHS2Cluster just works, always, moreover, it uses the same
> minicluster architecture as our qtests (I mean the way it achieves a mini
> mr/tez/llap/whatever cluster from the hadoop shim). Qtests are also not an
> alternative: I often need the ability to run queries in random order or in
> adhoc way without editing a qfile.
>
> I feel that HIVE-27338 <https://issues.apache.org/jira/browse/HIVE-27338>
> was
> about to solve licensing problems, and it achieved that, but maybe went too
> far: I believe we should be able to provide jars to the test classpath,
> e.g. jars that were downloaded beforehand, even manually.
>
> Regards,
> Laszlo Bodor
>
>
> Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023. aug. 18.,
> P, 13:08):
>
> > Hey Zsolt,
> >
> > I would divide this discussion into three topics:
> >
> > 1. What are the benefits of using the StartMiniHS2Cluster?
> > 2. What other alternatives are there for testing HS2 with different
> > metastore DBMS?
> > 3. How can we make StartMiniHS2Cluster work as before?
> >
> > Regarding the point 1, I don't have an answer because I never used
> > StartMiniHS2Cluster myself. Obviously, other people here are using it so
> I
> > would be curious to know in which cases this is useful.
> >
> > There are various alternatives for testing HS2 with different metastore
> > DBMS.
> >
> > The first and in my opinion the easiest to use is the classic qtest
> > framework with the various CLI drivers. Basically, with the work that
> > Laszlo started in HIVE-21954 it is pretty easy to run any kind of test
> over
> > any metastore just by setting the respective system property.
> >
> > mvn test -Dtest=TestMiniLlapLocalCliDriver
> > -Dqfile=partition_params_postgres.q -Dtest.metastore.db=postgres
> >
> > The second is to use our brand new and shiny docker images contributed by
> > Zhihua in HIVE-26400 and get the real feel of HS2 and HMS in a prod-like
> > setup. I haven't played much with this in apache/master but I did use
> some
> > similar images in our internal forks and it's pretty easy to get it up
> and
> > running.
> >
> > The third is the old and classic hive-dev-box [1] started by Zoltan. With
> > two/three commands you have a Hive cluster like environment and of course
> > you can choose which DBMS you want for the metastore.
> >
> > Regarding point 3, I assume that it is pretty easy to fix by adding the
> > postgresql (or other JDBC driver) dependency inside the hive-it-unit
> > module.
> >
> > <dependency>
> > <groupId>org.postgresql</groupId>
> > <artifactId>postgresql</artifactId>
> > <optional>true</optional>
> > </dependency>
> >
> > Given that we have other alternatives do we really need to go into this
> > direction? In fact, do we really need the StartMiniHS2Cluster class?
> >
> > Best,
> > Stamatis
> >
> > [1] https://github.com/kgyrtkirk/hive-dev-box
> >
> > On Tue, Aug 15, 2023, 6:08 PM Zsolt Miskolczi <zsolt.miskolczi@gmail.com
> >
> > wrote:
> >
> > > Hey there!Do you know how it is possible to use minihs2 with embedded
> > > metastore service but postgresql as the metastore database?
> > >
> > > Since I'm pretty sure it broke with that change
> > > https://github.com/apache/hive/pull/4317, handling drivers is changed:
> > >
> > > We don't bundle postgresql drivers with the hive build.Firstly, when I
> > copy
> > > the postgresql driver jar into the
> > >
> > >
> >
> packaging/target/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/lib
> > > directory, I can run the schematool as it sees the driver.
> > > The real issue is with miniHS2Cluster: it is basically a maven test
> and I
> > > have really no idea how I can pass the driver to the test. If it would
> > be a
> > > simple java command, I would try to pass the -cp argument but I see no
> > > other way to add extra jar file to the test execution.I want to run
> > minihs2
> > > with this command: mvn test -Dtest=StartMiniHS2Cluster
> -DminiHS2.run=true
> > > -DminiHS2.usePortsFromConf=true -DminiHS2.clusterType=LLAP
> > > -DminiHS2.conf="../../data/conf/llap/hive-site.xml"
> > > -Dpackaging.minimizeJar=false -DskipShade -Dremoteresources.skip=true
> > > -Dmaven.javadoc.skip=true -Denforcer.skip=trueAnd get that error (it
> > seems
> > > the metastore client cannot find the proper driver):
> > >
> > > java.lang.RuntimeException: Error applying authorization policy on hive
> > > configuration: org.apache.hadoop.hive.ql.metadata.HiveException: Unable
> > to
> > > instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> > > at org.apache.hive.service.cli.CLIService.init(CLIService.java:122)
> > > at
> > org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> > > at
> org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:243)
> > > at org.apache.hive.jdbc.miniHS2.MiniHS2.start(MiniHS2.java:394)
> > > at
> > >
> > >
> >
> org.apache.hive.jdbc.miniHS2.StartMiniHS2Cluster.testRunCluster(StartMiniHS2Cluster.java:78)
> > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > at
> > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > at
> > >
> > >
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> > > at
> > >
> > >
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> > > at
> > >
> > >
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> > > at
> > >
> > >
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> > > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> > > at
> > >
> > >
> >
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> > > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> > > at
> > >
> > >
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> > > at
> > >
> > >
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> > > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> > > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> > > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> > > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> > > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> > > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> > > at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> > > at
> > >
> > >
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> > > at
> > >
> > >
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> > > at
> > >
> > >
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> > > at
> > >
> > >
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> > > at
> > >
> > >
> >
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
> > > at
> > >
> > >
> >
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
> > > at
> > org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
> > > at
> > >
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
> > > Caused by: java.lang.RuntimeException:
> > > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to instantiate
> > > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:1001)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:2003)
> > > at
> > >
> > >
> >
> org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:135)
> > > at org.apache.hive.service.cli.CLIService.init(CLIService.java:119)
> > > ... 32 more
> > > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
> > > instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1033)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:993)
> > > ... 35 more
> > > Caused by: java.lang.RuntimeException: Unable to instantiate
> > > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:88)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:96)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:149)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:120)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:5769)
> > > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5847)
> > > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5827)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1029)
> > > ... 36 more
> > > Caused by: java.lang.reflect.InvocationTargetException
> > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> > > at
> > >
> > >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> > > at
> > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > > at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86)
> > > ... 43 more
> > > Caused by: MetaException(message:Got exception:
> > > org.apache.hadoop.hive.metastore.api.MetaException
> java.sql.SQLException:
> > > No suitable driver)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.throwMetaException(MetaStoreUtils.java:150)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.callEmbeddedMetastore(HiveMetaStoreClient.java:309)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:220)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.ql.metadata.HiveMetaStoreClientWithLocalCache.<init>(HiveMetaStoreClientWithLocalCache.java:120)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:154)
> > > ... 48 more
> > >
> > >
> > > Any help is appreciated,
> > > Thank you
> > >
> >
>
Re: MiniHS2 and postgresql jars
Posted by László Bodor <bo...@gmail.com>.
Hey!
I tend to return to StartMiniHS2Cluster from time to time. It's very good
for the "change code - compile - run - debug - repeat" way of doing things.
From this point of view, docker image is not an alternative to that. Also,
StartMiniHS2Cluster just works, always, moreover, it uses the same
minicluster architecture as our qtests (I mean the way it achieves a mini
mr/tez/llap/whatever cluster from the hadoop shim). Qtests are also not an
alternative: I often need the ability to run queries in random order or in
adhoc way without editing a qfile.
I feel that HIVE-27338 <https://issues.apache.org/jira/browse/HIVE-27338> was
about to solve licensing problems, and it achieved that, but maybe went too
far: I believe we should be able to provide jars to the test classpath,
e.g. jars that were downloaded beforehand, even manually.
Regards,
Laszlo Bodor
Stamatis Zampetakis <za...@gmail.com> ezt írta (időpont: 2023. aug. 18.,
P, 13:08):
> Hey Zsolt,
>
> I would divide this discussion into three topics:
>
> 1. What are the benefits of using the StartMiniHS2Cluster?
> 2. What other alternatives are there for testing HS2 with different
> metastore DBMS?
> 3. How can we make StartMiniHS2Cluster work as before?
>
> Regarding the point 1, I don't have an answer because I never used
> StartMiniHS2Cluster myself. Obviously, other people here are using it so I
> would be curious to know in which cases this is useful.
>
> There are various alternatives for testing HS2 with different metastore
> DBMS.
>
> The first and in my opinion the easiest to use is the classic qtest
> framework with the various CLI drivers. Basically, with the work that
> Laszlo started in HIVE-21954 it is pretty easy to run any kind of test over
> any metastore just by setting the respective system property.
>
> mvn test -Dtest=TestMiniLlapLocalCliDriver
> -Dqfile=partition_params_postgres.q -Dtest.metastore.db=postgres
>
> The second is to use our brand new and shiny docker images contributed by
> Zhihua in HIVE-26400 and get the real feel of HS2 and HMS in a prod-like
> setup. I haven't played much with this in apache/master but I did use some
> similar images in our internal forks and it's pretty easy to get it up and
> running.
>
> The third is the old and classic hive-dev-box [1] started by Zoltan. With
> two/three commands you have a Hive cluster like environment and of course
> you can choose which DBMS you want for the metastore.
>
> Regarding point 3, I assume that it is pretty easy to fix by adding the
> postgresql (or other JDBC driver) dependency inside the hive-it-unit
> module.
>
> <dependency>
> <groupId>org.postgresql</groupId>
> <artifactId>postgresql</artifactId>
> <optional>true</optional>
> </dependency>
>
> Given that we have other alternatives do we really need to go into this
> direction? In fact, do we really need the StartMiniHS2Cluster class?
>
> Best,
> Stamatis
>
> [1] https://github.com/kgyrtkirk/hive-dev-box
>
> On Tue, Aug 15, 2023, 6:08 PM Zsolt Miskolczi <zs...@gmail.com>
> wrote:
>
> > Hey there!Do you know how it is possible to use minihs2 with embedded
> > metastore service but postgresql as the metastore database?
> >
> > Since I'm pretty sure it broke with that change
> > https://github.com/apache/hive/pull/4317, handling drivers is changed:
> >
> > We don't bundle postgresql drivers with the hive build.Firstly, when I
> copy
> > the postgresql driver jar into the
> >
> >
> packaging/target/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/lib
> > directory, I can run the schematool as it sees the driver.
> > The real issue is with miniHS2Cluster: it is basically a maven test and I
> > have really no idea how I can pass the driver to the test. If it would
> be a
> > simple java command, I would try to pass the -cp argument but I see no
> > other way to add extra jar file to the test execution.I want to run
> minihs2
> > with this command: mvn test -Dtest=StartMiniHS2Cluster -DminiHS2.run=true
> > -DminiHS2.usePortsFromConf=true -DminiHS2.clusterType=LLAP
> > -DminiHS2.conf="../../data/conf/llap/hive-site.xml"
> > -Dpackaging.minimizeJar=false -DskipShade -Dremoteresources.skip=true
> > -Dmaven.javadoc.skip=true -Denforcer.skip=trueAnd get that error (it
> seems
> > the metastore client cannot find the proper driver):
> >
> > java.lang.RuntimeException: Error applying authorization policy on hive
> > configuration: org.apache.hadoop.hive.ql.metadata.HiveException: Unable
> to
> > instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> > at org.apache.hive.service.cli.CLIService.init(CLIService.java:122)
> > at
> org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> > at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:243)
> > at org.apache.hive.jdbc.miniHS2.MiniHS2.start(MiniHS2.java:394)
> > at
> >
> >
> org.apache.hive.jdbc.miniHS2.StartMiniHS2Cluster.testRunCluster(StartMiniHS2Cluster.java:78)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at
> >
> >
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> > at
> >
> >
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> > at
> >
> >
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> > at
> >
> >
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> > at
> >
> >
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> > at
> >
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> > at
> >
> >
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> > at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> > at
> >
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> > at
> >
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> > at
> >
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> > at
> >
> >
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> > at
> >
> >
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
> > at
> >
> >
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
> > at
> org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
> > at
> > org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
> > Caused by: java.lang.RuntimeException:
> > org.apache.hadoop.hive.ql.metadata.HiveException: Unable to instantiate
> > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> > at
> >
> >
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:1001)
> > at
> >
> >
> org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:2003)
> > at
> >
> >
> org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:135)
> > at org.apache.hive.service.cli.CLIService.init(CLIService.java:119)
> > ... 32 more
> > Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
> > instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> > at
> >
> >
> org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1033)
> > at
> >
> >
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:993)
> > ... 35 more
> > Caused by: java.lang.RuntimeException: Unable to instantiate
> > org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> > at
> >
> >
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:88)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:96)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:149)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:120)
> > at
> >
> >
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:5769)
> > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5847)
> > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5827)
> > at
> >
> >
> org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1029)
> > ... 36 more
> > Caused by: java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> > at
> >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> > at
> >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86)
> > ... 43 more
> > Caused by: MetaException(message:Got exception:
> > org.apache.hadoop.hive.metastore.api.MetaException java.sql.SQLException:
> > No suitable driver)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.throwMetaException(MetaStoreUtils.java:150)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.callEmbeddedMetastore(HiveMetaStoreClient.java:309)
> > at
> >
> >
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:220)
> > at
> >
> >
> org.apache.hadoop.hive.ql.metadata.HiveMetaStoreClientWithLocalCache.<init>(HiveMetaStoreClientWithLocalCache.java:120)
> > at
> >
> >
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:154)
> > ... 48 more
> >
> >
> > Any help is appreciated,
> > Thank you
> >
>
Re: MiniHS2 and postgresql jars
Posted by Stamatis Zampetakis <za...@gmail.com>.
Hey Zsolt,
I would divide this discussion into three topics:
1. What are the benefits of using the StartMiniHS2Cluster?
2. What other alternatives are there for testing HS2 with different
metastore DBMS?
3. How can we make StartMiniHS2Cluster work as before?
Regarding the point 1, I don't have an answer because I never used
StartMiniHS2Cluster myself. Obviously, other people here are using it so I
would be curious to know in which cases this is useful.
There are various alternatives for testing HS2 with different metastore
DBMS.
The first and in my opinion the easiest to use is the classic qtest
framework with the various CLI drivers. Basically, with the work that
Laszlo started in HIVE-21954 it is pretty easy to run any kind of test over
any metastore just by setting the respective system property.
mvn test -Dtest=TestMiniLlapLocalCliDriver
-Dqfile=partition_params_postgres.q -Dtest.metastore.db=postgres
The second is to use our brand new and shiny docker images contributed by
Zhihua in HIVE-26400 and get the real feel of HS2 and HMS in a prod-like
setup. I haven't played much with this in apache/master but I did use some
similar images in our internal forks and it's pretty easy to get it up and
running.
The third is the old and classic hive-dev-box [1] started by Zoltan. With
two/three commands you have a Hive cluster like environment and of course
you can choose which DBMS you want for the metastore.
Regarding point 3, I assume that it is pretty easy to fix by adding the
postgresql (or other JDBC driver) dependency inside the hive-it-unit module.
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<optional>true</optional>
</dependency>
Given that we have other alternatives do we really need to go into this
direction? In fact, do we really need the StartMiniHS2Cluster class?
Best,
Stamatis
[1] https://github.com/kgyrtkirk/hive-dev-box
On Tue, Aug 15, 2023, 6:08 PM Zsolt Miskolczi <zs...@gmail.com>
wrote:
> Hey there!Do you know how it is possible to use minihs2 with embedded
> metastore service but postgresql as the metastore database?
>
> Since I'm pretty sure it broke with that change
> https://github.com/apache/hive/pull/4317, handling drivers is changed:
>
> We don't bundle postgresql drivers with the hive build.Firstly, when I copy
> the postgresql driver jar into the
>
> packaging/target/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/apache-hive-4.0.0-beta-1-SNAPSHOT-bin/lib
> directory, I can run the schematool as it sees the driver.
> The real issue is with miniHS2Cluster: it is basically a maven test and I
> have really no idea how I can pass the driver to the test. If it would be a
> simple java command, I would try to pass the -cp argument but I see no
> other way to add extra jar file to the test execution.I want to run minihs2
> with this command: mvn test -Dtest=StartMiniHS2Cluster -DminiHS2.run=true
> -DminiHS2.usePortsFromConf=true -DminiHS2.clusterType=LLAP
> -DminiHS2.conf="../../data/conf/llap/hive-site.xml"
> -Dpackaging.minimizeJar=false -DskipShade -Dremoteresources.skip=true
> -Dmaven.javadoc.skip=true -Denforcer.skip=trueAnd get that error (it seems
> the metastore client cannot find the proper driver):
>
> java.lang.RuntimeException: Error applying authorization policy on hive
> configuration: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
> instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at org.apache.hive.service.cli.CLIService.init(CLIService.java:122)
> at org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:243)
> at org.apache.hive.jdbc.miniHS2.MiniHS2.start(MiniHS2.java:394)
> at
>
> org.apache.hive.jdbc.miniHS2.StartMiniHS2Cluster.testRunCluster(StartMiniHS2Cluster.java:78)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
>
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> at
>
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
>
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> at
>
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
>
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
>
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> at
>
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
>
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
>
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
>
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
>
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:377)
> at
>
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:138)
> at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:465)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:451)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
>
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:1001)
> at
>
> org.apache.hadoop.hive.ql.session.SessionState.applyAuthorizationPolicy(SessionState.java:2003)
> at
>
> org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:135)
> at org.apache.hive.service.cli.CLIService.init(CLIService.java:119)
> ... 32 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Unable to
> instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
>
> org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1033)
> at
>
> org.apache.hadoop.hive.ql.session.SessionState.setupAuth(SessionState.java:993)
> ... 35 more
> Caused by: java.lang.RuntimeException: Unable to instantiate
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
> at
>
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:88)
> at
>
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:96)
> at
>
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:149)
> at
>
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:120)
> at
>
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:5769)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5847)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:5827)
> at
>
> org.apache.hadoop.hive.ql.session.SessionState.setAuthorizerV2Config(SessionState.java:1029)
> ... 36 more
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
>
> org.apache.hadoop.hive.metastore.utils.JavaUtils.newInstance(JavaUtils.java:86)
> ... 43 more
> Caused by: MetaException(message:Got exception:
> org.apache.hadoop.hive.metastore.api.MetaException java.sql.SQLException:
> No suitable driver)
> at
>
> org.apache.hadoop.hive.metastore.utils.MetaStoreUtils.throwMetaException(MetaStoreUtils.java:150)
> at
>
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.callEmbeddedMetastore(HiveMetaStoreClient.java:309)
> at
>
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:220)
> at
>
> org.apache.hadoop.hive.ql.metadata.HiveMetaStoreClientWithLocalCache.<init>(HiveMetaStoreClientWithLocalCache.java:120)
> at
>
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:154)
> ... 48 more
>
>
> Any help is appreciated,
> Thank you
>