You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Peter Vary via Review Board <no...@reviews.apache.org> on 2018/05/31 13:53:55 UTC
Review Request 67399: HIVE-19503 Create a test that checks for
dropPartitions with directSql
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67399/
-----------------------------------------------------------
Review request for hive, Alexander Kolbasov and Vihang Karajgaonkar.
Bugs: HIVE-19503
https://issues.apache.org/jira/browse/HIVE-19503
Repository: hive-git
Description
-------
The patch contains 2 tests:
Test which checks if the JDO cache is able to handle directSql partition drops
Test which checks if the directSQL partition drop removes every connected data from the RDBMS tables.
To create these tests we have 2 helper methods:
Method to create the partitioned table
Method to check the number of rows in a given RDBMS table
Added a new ObjectStore.dropPartitionsInternal method which only visible for testing so we can make sure that the dropPartition is using directSql and does not fall back to JDO.
Fixed a problem where some of the tables are not created automatically by the tests, adding new init queries to MetaStoreDirectSql.ensureDbInit method
Diffs
-----
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 5bb1985
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java b15d89d
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java 9912213
Diff: https://reviews.apache.org/r/67399/diff/1/
Testing
-------
Run the new tests
Thanks,
Peter Vary
Re: Review Request 67399: HIVE-19503 Create a test that checks for
dropPartitions with directSql
Posted by Vihang Karajgaonkar via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67399/#review204195
-----------------------------------------------------------
Ship it!
Thanks for the changes. LGTM
- Vihang Karajgaonkar
On June 1, 2018, 10:23 a.m., Peter Vary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67399/
> -----------------------------------------------------------
>
> (Updated June 1, 2018, 10:23 a.m.)
>
>
> Review request for hive, Alexander Kolbasov and Vihang Karajgaonkar.
>
>
> Bugs: HIVE-19503
> https://issues.apache.org/jira/browse/HIVE-19503
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> The patch contains 2 tests:
>
> Test which checks if the JDO cache is able to handle directSql partition drops
> Test which checks if the directSQL partition drop removes every connected data from the RDBMS tables.
> To create these tests we have 2 helper methods:
>
> Method to create the partitioned table
> Method to check the number of rows in a given RDBMS table
> Added a new ObjectStore.dropPartitionsInternal method which only visible for testing so we can make sure that the dropPartition is using directSql and does not fall back to JDO.
>
> Fixed a problem where some of the tables are not created automatically by the tests, adding new init queries to MetaStoreDirectSql.ensureDbInit method
>
>
> Diffs
> -----
>
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 5bb1985
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java b15d89d
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java 9912213
>
>
> Diff: https://reviews.apache.org/r/67399/diff/2/
>
>
> Testing
> -------
>
> Run the new tests
>
>
> Thanks,
>
> Peter Vary
>
>
Re: Review Request 67399: HIVE-19503 Create a test that checks for
dropPartitions with directSql
Posted by Peter Vary via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67399/
-----------------------------------------------------------
(Updated June 4, 2018, 9:51 a.m.)
Review request for hive, Alexander Kolbasov and Vihang Karajgaonkar.
Changes
-------
The PreCommit test failed because the ObjectStore is not run it test mode, so the initializing queries are not called, thus the tables are not created.
Set the HIVE_IN_TEST flag before starting the ObjectStore
Bugs: HIVE-19503
https://issues.apache.org/jira/browse/HIVE-19503
Repository: hive-git
Description
-------
The patch contains 2 tests:
Test which checks if the JDO cache is able to handle directSql partition drops
Test which checks if the directSQL partition drop removes every connected data from the RDBMS tables.
To create these tests we have 2 helper methods:
Method to create the partitioned table
Method to check the number of rows in a given RDBMS table
Added a new ObjectStore.dropPartitionsInternal method which only visible for testing so we can make sure that the dropPartition is using directSql and does not fall back to JDO.
Fixed a problem where some of the tables are not created automatically by the tests, adding new init queries to MetaStoreDirectSql.ensureDbInit method
Diffs (updated)
-----
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 5bb1985
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java b15d89d
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java 9912213
Diff: https://reviews.apache.org/r/67399/diff/3/
Changes: https://reviews.apache.org/r/67399/diff/2-3/
Testing
-------
Run the new tests
Thanks,
Peter Vary
Re: Review Request 67399: HIVE-19503 Create a test that checks for
dropPartitions with directSql
Posted by Peter Vary via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67399/
-----------------------------------------------------------
(Updated June 1, 2018, 10:23 a.m.)
Review request for hive, Alexander Kolbasov and Vihang Karajgaonkar.
Changes
-------
Addressed Vihang's comments
Bugs: HIVE-19503
https://issues.apache.org/jira/browse/HIVE-19503
Repository: hive-git
Description
-------
The patch contains 2 tests:
Test which checks if the JDO cache is able to handle directSql partition drops
Test which checks if the directSQL partition drop removes every connected data from the RDBMS tables.
To create these tests we have 2 helper methods:
Method to create the partitioned table
Method to check the number of rows in a given RDBMS table
Added a new ObjectStore.dropPartitionsInternal method which only visible for testing so we can make sure that the dropPartition is using directSql and does not fall back to JDO.
Fixed a problem where some of the tables are not created automatically by the tests, adding new init queries to MetaStoreDirectSql.ensureDbInit method
Diffs (updated)
-----
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 5bb1985
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java b15d89d
standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java 9912213
Diff: https://reviews.apache.org/r/67399/diff/2/
Changes: https://reviews.apache.org/r/67399/diff/1-2/
Testing
-------
Run the new tests
Thanks,
Peter Vary
Re: Review Request 67399: HIVE-19503 Create a test that checks for
dropPartitions with directSql
Posted by Peter Vary via Review Board <no...@reviews.apache.org>.
> On May 31, 2018, 10:36 p.m., Vihang Karajgaonkar wrote:
> > Thanks for the tests. Can we add a one more test which repeats the same test accross multiple sessions (can be simulated by using objectstore instances)? So for example:
> >
> > 1. Session 1 : list partitions with jdo, Session 2 : list partitions with jdo
> > 2. Session 1 : drop partitions with directSQL
> > 3. Session 2 : list partitions with jdo
Added the extra test.
> On May 31, 2018, 10:36 p.m., Vihang Karajgaonkar wrote:
> > standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> > Lines 2562 (patched)
> > <https://reviews.apache.org/r/67399/diff/1/?file=2033271#file2033271line2562>
> >
> > why do we need it to be protected? Can we use package private instead?
Valid point :)
Fixed.
Thanks!
- Peter
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67399/#review204139
-----------------------------------------------------------
On June 1, 2018, 10:23 a.m., Peter Vary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67399/
> -----------------------------------------------------------
>
> (Updated June 1, 2018, 10:23 a.m.)
>
>
> Review request for hive, Alexander Kolbasov and Vihang Karajgaonkar.
>
>
> Bugs: HIVE-19503
> https://issues.apache.org/jira/browse/HIVE-19503
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> The patch contains 2 tests:
>
> Test which checks if the JDO cache is able to handle directSql partition drops
> Test which checks if the directSQL partition drop removes every connected data from the RDBMS tables.
> To create these tests we have 2 helper methods:
>
> Method to create the partitioned table
> Method to check the number of rows in a given RDBMS table
> Added a new ObjectStore.dropPartitionsInternal method which only visible for testing so we can make sure that the dropPartition is using directSql and does not fall back to JDO.
>
> Fixed a problem where some of the tables are not created automatically by the tests, adding new init queries to MetaStoreDirectSql.ensureDbInit method
>
>
> Diffs
> -----
>
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 5bb1985
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java b15d89d
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java 9912213
>
>
> Diff: https://reviews.apache.org/r/67399/diff/2/
>
>
> Testing
> -------
>
> Run the new tests
>
>
> Thanks,
>
> Peter Vary
>
>
Re: Review Request 67399: HIVE-19503 Create a test that checks for
dropPartitions with directSql
Posted by Vihang Karajgaonkar via Review Board <no...@reviews.apache.org>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/67399/#review204139
-----------------------------------------------------------
Thanks for the tests. Can we add a one more test which repeats the same test accross multiple sessions (can be simulated by using objectstore instances)? So for example:
1. Session 1 : list partitions with jdo, Session 2 : list partitions with jdo
2. Session 1 : drop partitions with directSQL
3. Session 2 : list partitions with jdo
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java
Lines 2562 (patched)
<https://reviews.apache.org/r/67399/#comment286535>
why do we need it to be protected? Can we use package private instead?
- Vihang Karajgaonkar
On May 31, 2018, 1:53 p.m., Peter Vary wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/67399/
> -----------------------------------------------------------
>
> (Updated May 31, 2018, 1:53 p.m.)
>
>
> Review request for hive, Alexander Kolbasov and Vihang Karajgaonkar.
>
>
> Bugs: HIVE-19503
> https://issues.apache.org/jira/browse/HIVE-19503
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> The patch contains 2 tests:
>
> Test which checks if the JDO cache is able to handle directSql partition drops
> Test which checks if the directSQL partition drop removes every connected data from the RDBMS tables.
> To create these tests we have 2 helper methods:
>
> Method to create the partitioned table
> Method to check the number of rows in a given RDBMS table
> Added a new ObjectStore.dropPartitionsInternal method which only visible for testing so we can make sure that the dropPartition is using directSql and does not fall back to JDO.
>
> Fixed a problem where some of the tables are not created automatically by the tests, adding new init queries to MetaStoreDirectSql.ensureDbInit method
>
>
> Diffs
> -----
>
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java 5bb1985
> standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java b15d89d
> standalone-metastore/src/test/java/org/apache/hadoop/hive/metastore/TestObjectStore.java 9912213
>
>
> Diff: https://reviews.apache.org/r/67399/diff/1/
>
>
> Testing
> -------
>
> Run the new tests
>
>
> Thanks,
>
> Peter Vary
>
>