You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by Sunitha Beeram via Review Board <no...@reviews.apache.org> on 2017/06/21 22:02:26 UTC

Review Request 60303: HIVE-16908: Update table and partition replication tests to not use 2nd HCat instance

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60303/
-----------------------------------------------------------

Review request for hive, Carl Steinbach, Anthony Hsu, and Ratandeep Ratti.

Bugs: HIVE-16908
https://issues.apache.org/jira/browse/HIVE-16908

Repository: hive-git

Description
-------

HIVE-16908: Update table and partition replication tests to not use 2nd HCat instance

HIVE-16844 fixed a connection leak issue which subsequently exposed failures in TestHCatClient. The connection leak gets triggered if a metastore instance is updated with a different JDO configuration. TestHCatClient uses 2 metastore instances to test replication related methods. Unfortunately, it does so by providing a different derby db name for the second instance. Since the 2 metastores run in the same JVM, the path fixed in HIVE-16844 gets triggered, resulting in "sourceMetastore"'s connection being closed and thus resulting in failures.

It appears to me that running 2 metastore instances within the same JVM is error prone as there could be unintentional side-effects due to statics in the code (as was exposed by fixing HIVE-16844). This patch provides a way to test the replication related methods without involving a second instance. The changes mainly validate the serialize/deserialize methods. One of the tests, testPartitionRegistrationWithCustomSchema, uses addPartitions method to verify propogation of changes and it appeared that addPartitions wasn't covered by other tests in TestHCatClient and there wasn't a better way to verify the intended path, so I used an approach where the original database and table are dropped and recreated using the serialized-string and captured partition spec.

Diffs
-----

hcatalog/webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java 86d3acbcb462d244fa2dc2f48923aab1e3ccee66

Diff: https://reviews.apache.org/r/60303/diff/1/

Testing
-------

mvn test -DTest=TestHCatClient now passes.

Thanks,

Sunitha Beeram

Re: Review Request 60303: HIVE-16908: Update table and partition replication tests to not use 2nd HCat instance

Posted by Anthony Hsu via Review Board <no...@reviews.apache.org>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60303/#review178710
-----------------------------------------------------------




hcatalog/webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java
Lines 796-807 (original)
<https://reviews.apache.org/r/60303/#comment252859>

    Instead of deleting this, what about just starting the second metastore in a separate process? Then we can preserve the end-to-end integration-esque nature of the tests.



hcatalog/webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java
Line 1011 (original), 996 (patched)
<https://reviews.apache.org/r/60303/#comment252858>

    its -> it's


- Anthony Hsu


On 六月 22, 2017, 12:59 a.m., Sunitha Beeram wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60303/
> -----------------------------------------------------------
> 
> (Updated 六月 22, 2017, 12:59 a.m.)
> 
> 
> Review request for hive, Carl Steinbach, Anthony Hsu, and Ratandeep Ratti.
> 
> 
> Bugs: HIVE-16908
>     https://issues.apache.org/jira/browse/HIVE-16908
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> HIVE-16908: Update table and partition replication tests to not use 2nd HCat instance
> 
> HIVE-16844 fixed a connection leak issue which subsequently exposed failures in TestHCatClient. The connection leak gets triggered if a metastore instance is updated with a different JDO configuration. TestHCatClient uses 2 metastore instances to test replication related methods. Unfortunately, it does so by providing a different derby db name for the second instance. Since the 2 metastores run in the same JVM, the path fixed in HIVE-16844 gets triggered, resulting in "sourceMetastore"'s connection being closed and thus resulting in failures.
> 
> It appears to me that running 2 metastore instances within the same JVM is error prone as there could be unintentional side-effects due to statics in the code (as was exposed by fixing HIVE-16844). This patch provides a way to test the replication related methods without involving a second instance. The changes mainly validate the serialize/deserialize methods. One of the tests, testPartitionRegistrationWithCustomSchema, uses addPartitions method to verify propogation of changes and it appeared that addPartitions wasn't covered by other tests in TestHCatClient and there wasn't a better way to verify the intended path, so I used an approach where the original database and table are dropped and recreated using the serialized-string and captured partition spec.
> 
> 
> Diffs
> -----
> 
>   hcatalog/webhcat/java-client/src/test/java/org/apache/hive/hcatalog/api/TestHCatClient.java 86d3acbcb462d244fa2dc2f48923aab1e3ccee66 
> 
> 
> Diff: https://reviews.apache.org/r/60303/diff/2/
> 
> 
> Testing
> -------
> 
> mvn test -DTest=TestHCatClient now passes.
> 
> 
> Thanks,
> 
> Sunitha Beeram
> 
>

Re: Review Request 60303: HIVE-16908: Update table and partition replication tests to not use 2nd HCat instance

Posted by Sunitha Beeram via Review Board <no...@reviews.apache.org>.

(Updated June 22, 2017, 12:59 a.m.)