You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Prasad Mujumdar <pr...@cloudera.com> on 2013/09/13 09:53:10 UTC
Review Request 14120: HIVE-3764: Support metastore version consistency check
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14120/
-----------------------------------------------------------
Review request for hive.
Bugs: HIVE-3764
https://issues.apache.org/jira/browse/HIVE-3764
Repository: hive-git
Description
-------
- Added a new table in the metastore schema to store the Hive version in the metastore.
- Metastore handler compare the version stored in the schema with its own version. If there's a mismatch, then it can either record the correct version or raise error. The behavior is configurable via a new Hive config. This config when set, also restrict dataNucleus to auto upgrade the schema.
- The new schema creation and upgrade scripts record the new version in the metastore version table.
- Added 0.12 upgrade scripts for all supported DBs to creates the new table version tables in 0.12 metastore schema
- Added a new schemaTool that can perform new schema initialization or upgrade based on the schema version and product version.
Diffs
-----
beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java PRE-CREATION
beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java PRE-CREATION
beeline/src/test/org/apache/hive/beeline/src/test/TestSchemaTool.java PRE-CREATION
bin/ext/schemaTool.sh PRE-CREATION
bin/schematool PRE-CREATION
build-common.xml ad5ac23
build.xml 3e87163
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 22149e4
conf/hive-default.xml.template 9a3fc1d
metastore/scripts/upgrade/derby/014-HIVE-3764.derby.sql PRE-CREATION
metastore/scripts/upgrade/derby/hive-schema-0.12.0.derby.sql cce544f
metastore/scripts/upgrade/derby/upgrade-0.10.0-to-0.11.0.derby.sql cae7936
metastore/scripts/upgrade/derby/upgrade-0.11.0-to-0.12.0.derby.sql 492cc93
metastore/scripts/upgrade/derby/upgrade.order.derby PRE-CREATION
metastore/scripts/upgrade/mysql/014-HIVE-3764.mysql.sql PRE-CREATION
metastore/scripts/upgrade/mysql/hive-schema-0.12.0.mysql.sql 22a77fe
metastore/scripts/upgrade/mysql/upgrade-0.11.0-to-0.12.0.mysql.sql 375a05f
metastore/scripts/upgrade/mysql/upgrade.order.mysql PRE-CREATION
metastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql PRE-CREATION
metastore/scripts/upgrade/oracle/hive-schema-0.12.0.oracle.sql 85a0178
metastore/scripts/upgrade/oracle/upgrade-0.10.0-to-0.11.0.mysql.sql PRE-CREATION
metastore/scripts/upgrade/oracle/upgrade-0.11.0-to-0.12.0.oracle.sql a2d0901
metastore/scripts/upgrade/oracle/upgrade.order.oracle PRE-CREATION
metastore/scripts/upgrade/postgres/014-HIVE-3764.postgres.sql PRE-CREATION
metastore/scripts/upgrade/postgres/hive-schema-0.12.0.postgres.sql 7b319ba
metastore/scripts/upgrade/postgres/upgrade-0.11.0-to-0.12.0.postgres.sql 9da0a1b
metastore/scripts/upgrade/postgres/upgrade.order.postgres PRE-CREATION
metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 39dda92
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java PRE-CREATION
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java a08c728
metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java e410c3a
metastore/src/model/org/apache/hadoop/hive/metastore/model/MVersionTable.java PRE-CREATION
metastore/src/model/package.jdo c42b5b0
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 8066784
metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 0f9b16c
metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java PRE-CREATION
Diff: https://reviews.apache.org/r/14120/diff/
Testing
-------
Added new tests for schema verification and schemaTool.
Thanks,
Prasad Mujumdar
Re: Review Request 14120: HIVE-3764: Support metastore version consistency
check
Posted by Prasad Mujumdar <pr...@cloudera.com>.
> On Sept. 13, 2013, 1:35 p.m., Brock Noland wrote:
> > Prasad, this looks really good! I just had two people email me directly yesterday and both were using the incorrect metastore version.
> >
> > Have you ran the new unit tests a couple of times? Have you done any other testing?
Addressed the comments. The MetaException doesn't support nesting, so changed that to HiveMetaException.
Added more tests.
manually tested the init and upgrade operations using derby and MySQL.
As discussed on the ticket, I am going to split the patch into two separate tickets. will close this review
- Prasad
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14120/#review26079
-----------------------------------------------------------
On Sept. 13, 2013, 7:53 a.m., Prasad Mujumdar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14120/
> -----------------------------------------------------------
>
> (Updated Sept. 13, 2013, 7:53 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-3764
> https://issues.apache.org/jira/browse/HIVE-3764
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> - Added a new table in the metastore schema to store the Hive version in the metastore.
> - Metastore handler compare the version stored in the schema with its own version. If there's a mismatch, then it can either record the correct version or raise error. The behavior is configurable via a new Hive config. This config when set, also restrict dataNucleus to auto upgrade the schema.
> - The new schema creation and upgrade scripts record the new version in the metastore version table.
> - Added 0.12 upgrade scripts for all supported DBs to creates the new table version tables in 0.12 metastore schema
> - Added a new schemaTool that can perform new schema initialization or upgrade based on the schema version and product version.
>
>
> Diffs
> -----
>
> beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java PRE-CREATION
> beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java PRE-CREATION
> beeline/src/test/org/apache/hive/beeline/src/test/TestSchemaTool.java PRE-CREATION
> bin/ext/schemaTool.sh PRE-CREATION
> bin/schematool PRE-CREATION
> build-common.xml ad5ac23
> build.xml 3e87163
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 22149e4
> conf/hive-default.xml.template 9a3fc1d
> metastore/scripts/upgrade/derby/014-HIVE-3764.derby.sql PRE-CREATION
> metastore/scripts/upgrade/derby/hive-schema-0.12.0.derby.sql cce544f
> metastore/scripts/upgrade/derby/upgrade-0.10.0-to-0.11.0.derby.sql cae7936
> metastore/scripts/upgrade/derby/upgrade-0.11.0-to-0.12.0.derby.sql 492cc93
> metastore/scripts/upgrade/derby/upgrade.order.derby PRE-CREATION
> metastore/scripts/upgrade/mysql/014-HIVE-3764.mysql.sql PRE-CREATION
> metastore/scripts/upgrade/mysql/hive-schema-0.12.0.mysql.sql 22a77fe
> metastore/scripts/upgrade/mysql/upgrade-0.11.0-to-0.12.0.mysql.sql 375a05f
> metastore/scripts/upgrade/mysql/upgrade.order.mysql PRE-CREATION
> metastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql PRE-CREATION
> metastore/scripts/upgrade/oracle/hive-schema-0.12.0.oracle.sql 85a0178
> metastore/scripts/upgrade/oracle/upgrade-0.10.0-to-0.11.0.mysql.sql PRE-CREATION
> metastore/scripts/upgrade/oracle/upgrade-0.11.0-to-0.12.0.oracle.sql a2d0901
> metastore/scripts/upgrade/oracle/upgrade.order.oracle PRE-CREATION
> metastore/scripts/upgrade/postgres/014-HIVE-3764.postgres.sql PRE-CREATION
> metastore/scripts/upgrade/postgres/hive-schema-0.12.0.postgres.sql 7b319ba
> metastore/scripts/upgrade/postgres/upgrade-0.11.0-to-0.12.0.postgres.sql 9da0a1b
> metastore/scripts/upgrade/postgres/upgrade.order.postgres PRE-CREATION
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 39dda92
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java PRE-CREATION
> metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java a08c728
> metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java e410c3a
> metastore/src/model/org/apache/hadoop/hive/metastore/model/MVersionTable.java PRE-CREATION
> metastore/src/model/package.jdo c42b5b0
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 8066784
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 0f9b16c
> metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/14120/diff/
>
>
> Testing
> -------
>
> Added new tests for schema verification and schemaTool.
>
>
> Thanks,
>
> Prasad Mujumdar
>
>
Re: Review Request 14120: HIVE-3764: Support metastore version consistency
check
Posted by Brock Noland <br...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/14120/#review26079
-----------------------------------------------------------
Prasad, this looks really good! I just had two people email me directly yesterday and both were using the incorrect metastore version.
Have you ran the new unit tests a couple of times? Have you done any other testing?
beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java
<https://reviews.apache.org/r/14120/#comment50906>
nit: Misspelled Oracle
beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java
<https://reviews.apache.org/r/14120/#comment50907>
Please nest the exceptin
beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java
<https://reviews.apache.org/r/14120/#comment50904>
Please next the exception
beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java
<https://reviews.apache.org/r/14120/#comment50905>
This won't print the stack trace
metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java
<https://reviews.apache.org/r/14120/#comment50908>
Please nest this exception
- Brock Noland
On Sept. 13, 2013, 7:53 a.m., Prasad Mujumdar wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/14120/
> -----------------------------------------------------------
>
> (Updated Sept. 13, 2013, 7:53 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-3764
> https://issues.apache.org/jira/browse/HIVE-3764
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> - Added a new table in the metastore schema to store the Hive version in the metastore.
> - Metastore handler compare the version stored in the schema with its own version. If there's a mismatch, then it can either record the correct version or raise error. The behavior is configurable via a new Hive config. This config when set, also restrict dataNucleus to auto upgrade the schema.
> - The new schema creation and upgrade scripts record the new version in the metastore version table.
> - Added 0.12 upgrade scripts for all supported DBs to creates the new table version tables in 0.12 metastore schema
> - Added a new schemaTool that can perform new schema initialization or upgrade based on the schema version and product version.
>
>
> Diffs
> -----
>
> beeline/src/java/org/apache/hive/beeline/HiveSchemaHelper.java PRE-CREATION
> beeline/src/java/org/apache/hive/beeline/HiveSchemaTool.java PRE-CREATION
> beeline/src/test/org/apache/hive/beeline/src/test/TestSchemaTool.java PRE-CREATION
> bin/ext/schemaTool.sh PRE-CREATION
> bin/schematool PRE-CREATION
> build-common.xml ad5ac23
> build.xml 3e87163
> common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 22149e4
> conf/hive-default.xml.template 9a3fc1d
> metastore/scripts/upgrade/derby/014-HIVE-3764.derby.sql PRE-CREATION
> metastore/scripts/upgrade/derby/hive-schema-0.12.0.derby.sql cce544f
> metastore/scripts/upgrade/derby/upgrade-0.10.0-to-0.11.0.derby.sql cae7936
> metastore/scripts/upgrade/derby/upgrade-0.11.0-to-0.12.0.derby.sql 492cc93
> metastore/scripts/upgrade/derby/upgrade.order.derby PRE-CREATION
> metastore/scripts/upgrade/mysql/014-HIVE-3764.mysql.sql PRE-CREATION
> metastore/scripts/upgrade/mysql/hive-schema-0.12.0.mysql.sql 22a77fe
> metastore/scripts/upgrade/mysql/upgrade-0.11.0-to-0.12.0.mysql.sql 375a05f
> metastore/scripts/upgrade/mysql/upgrade.order.mysql PRE-CREATION
> metastore/scripts/upgrade/oracle/014-HIVE-3764.oracle.sql PRE-CREATION
> metastore/scripts/upgrade/oracle/hive-schema-0.12.0.oracle.sql 85a0178
> metastore/scripts/upgrade/oracle/upgrade-0.10.0-to-0.11.0.mysql.sql PRE-CREATION
> metastore/scripts/upgrade/oracle/upgrade-0.11.0-to-0.12.0.oracle.sql a2d0901
> metastore/scripts/upgrade/oracle/upgrade.order.oracle PRE-CREATION
> metastore/scripts/upgrade/postgres/014-HIVE-3764.postgres.sql PRE-CREATION
> metastore/scripts/upgrade/postgres/hive-schema-0.12.0.postgres.sql 7b319ba
> metastore/scripts/upgrade/postgres/upgrade-0.11.0-to-0.12.0.postgres.sql 9da0a1b
> metastore/scripts/upgrade/postgres/upgrade.order.postgres PRE-CREATION
> metastore/src/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java 39dda92
> metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreSchemaInfo.java PRE-CREATION
> metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java a08c728
> metastore/src/java/org/apache/hadoop/hive/metastore/RawStore.java e410c3a
> metastore/src/model/org/apache/hadoop/hive/metastore/model/MVersionTable.java PRE-CREATION
> metastore/src/model/package.jdo c42b5b0
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreControlledCommit.java 8066784
> metastore/src/test/org/apache/hadoop/hive/metastore/DummyRawStoreForJdoConnection.java 0f9b16c
> metastore/src/test/org/apache/hadoop/hive/metastore/TestMetastoreVersion.java PRE-CREATION
>
> Diff: https://reviews.apache.org/r/14120/diff/
>
>
> Testing
> -------
>
> Added new tests for schema verification and schemaTool.
>
>
> Thanks,
>
> Prasad Mujumdar
>
>