You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Alan Gates <ga...@hortonworks.com> on 2013/12/10 00:14:41 UTC
Maven unit test question
I was attempting to write unit tests for changes I'm making to HiveMetaStoreClient as part of the ACID transaction work (see https://issues.apache.org/jira/browse/HIVE-5843). When I added the tests and attempted to run them using
mvn tests -Dtest=TestHiveMetaStoreClient -Phadoop-1
it failed with:
java.lang.NoClassDefFoundError: org/apache/hadoop/hive/thrift/TUGIContainingTransport$Factory
This class is contained in the hive-shims jar. The error surprised me because according to metastore/pom.xml, hive-shims is a dependency of hive-metastore. When I ran maven with -X to get debug information, I found that in the classpath it was including /Users/gates/git/apache/hive/shims/assembly/target/classes. I'm guessing that rather than use the shims jar (which has been built by this time) it's trying to use the compiled classes, but failing in this case because the shims jar is actually constructed not by directly conglomerating a set of class files but by picking and choosing from several shim jar versions and then constructing a single jar. But I could not figure out how to communicate to maven that is should use the already built shims jar rather than the classes. To test my theory I took the shims jar and unpacked in the path maven was looking in, and sure enough my tests ran once I did that.
The existing unit test TestMetastoreExpr in ql seems to have the same issue. I tried to use it as a model, but when I ran it it failed with the same error, and unpacking the jar resolved it in the same way.
Am I doing something wrong, or is there a change needed in the pom.xml to get it to look in the jar instead of the .class files for shims dependencies?
Alan.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Maven unit test question
Posted by Alan Gates <ga...@hortonworks.com>.
There's a patch on https://issues.apache.org/jira/browse/HIVE-5843 that has the code. Unfortunately the patch is huge because my work involves changes to the thrift interface. The git repo I'm working off of last pulled from Apache on Nov 20th (commit 1de88eedc69af1e7e618fc4f5eac045f69c02973 is the last one it has) so you may need to go back a bit in your repo to get a version that the patch will apply against.
The test in question is TestHiveMetaStoreClient.
Also, I had to move the class from metastore to ql as it turns out instantiating the HiveMetaStoreClient needs the hive-exec jar. I couldn't add a dependency on hive-exec in the metastore package as hive-exec depends on hive-metastore.
Thanks for your help.
Alan.
On Dec 9, 2013, at 3:53 PM, Brock Noland wrote:
> Can you share the change with me so I can debug?
> On Dec 9, 2013 5:15 PM, "Alan Gates" <ga...@hortonworks.com> wrote:
>
>> I was attempting to write unit tests for changes I'm making to
>> HiveMetaStoreClient as part of the ACID transaction work (see
>> https://issues.apache.org/jira/browse/HIVE-5843). When I added the tests
>> and attempted to run them using
>> mvn tests -Dtest=TestHiveMetaStoreClient -Phadoop-1
>>
>> it failed with:
>>
>> java.lang.NoClassDefFoundError:
>> org/apache/hadoop/hive/thrift/TUGIContainingTransport$Factory
>>
>> This class is contained in the hive-shims jar. The error surprised me
>> because according to metastore/pom.xml, hive-shims is a dependency of
>> hive-metastore. When I ran maven with -X to get debug information, I found
>> that in the classpath it was including
>> /Users/gates/git/apache/hive/shims/assembly/target/classes. I'm guessing
>> that rather than use the shims jar (which has been built by this time) it's
>> trying to use the compiled classes, but failing in this case because the
>> shims jar is actually constructed not by directly conglomerating a set of
>> class files but by picking and choosing from several shim jar versions and
>> then constructing a single jar. But I could not figure out how to
>> communicate to maven that is should use the already built shims jar rather
>> than the classes. To test my theory I took the shims jar and unpacked in
>> the path maven was looking in, and sure enough my tests ran once I did that.
>>
>> The existing unit test TestMetastoreExpr in ql seems to have the same
>> issue. I tried to use it as a model, but when I ran it it failed with the
>> same error, and unpacking the jar resolved it in the same way.
>>
>> Am I doing something wrong, or is there a change needed in the pom.xml to
>> get it to look in the jar instead of the .class files for shims
>> dependencies?
>>
>> Alan.
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Maven unit test question
Posted by Brock Noland <br...@cloudera.com>.
Can you share the change with me so I can debug?
On Dec 9, 2013 5:15 PM, "Alan Gates" <ga...@hortonworks.com> wrote:
> I was attempting to write unit tests for changes I'm making to
> HiveMetaStoreClient as part of the ACID transaction work (see
> https://issues.apache.org/jira/browse/HIVE-5843). When I added the tests
> and attempted to run them using
> mvn tests -Dtest=TestHiveMetaStoreClient -Phadoop-1
>
> it failed with:
>
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/hive/thrift/TUGIContainingTransport$Factory
>
> This class is contained in the hive-shims jar. The error surprised me
> because according to metastore/pom.xml, hive-shims is a dependency of
> hive-metastore. When I ran maven with -X to get debug information, I found
> that in the classpath it was including
> /Users/gates/git/apache/hive/shims/assembly/target/classes. I'm guessing
> that rather than use the shims jar (which has been built by this time) it's
> trying to use the compiled classes, but failing in this case because the
> shims jar is actually constructed not by directly conglomerating a set of
> class files but by picking and choosing from several shim jar versions and
> then constructing a single jar. But I could not figure out how to
> communicate to maven that is should use the already built shims jar rather
> than the classes. To test my theory I took the shims jar and unpacked in
> the path maven was looking in, and sure enough my tests ran once I did that.
>
> The existing unit test TestMetastoreExpr in ql seems to have the same
> issue. I tried to use it as a model, but when I ran it it failed with the
> same error, and unpacking the jar resolved it in the same way.
>
> Am I doing something wrong, or is there a change needed in the pom.xml to
> get it to look in the jar instead of the .class files for shims
> dependencies?
>
> Alan.
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>