[jira] [Comment Edited] (HIVE-21954) QTest: support for running qtests on various metastore DBs

["László Bodor (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-21954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16965365#comment-16965365 ] 

László Bodor edited comment on HIVE-21954 at 11/2/19 1:34 PM:
--------------------------------------------------------------

[~kgyrtkirk], [~pvary]: could you please take a look? the patch contains the following:
1.  Database rule refactoring in standalone-metastore project
2. Qtest infrastructure: preparing qtest infra for being able to run any qtest against any kind of metastores
3. A bunch of cleanups in the current qtest mess

With the patch, you can run the following (if your docker is set up properly, meaning that you can run the metastore integration tests, already described in standalone-metastore/DEV-README):
{code}
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q -Dtest.metastore.db=mssql
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q -Dtest.metastore.db=mysql
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q -Dtest.metastore.db=postgres
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q -Dtest.metastore.db=oracle -Ditest.jdbc.jars=/path/to/your/god/damn/oracle/jdbc/driver/ojdbc6.jar          
{code}

looking forward for comments: https://reviews.apache.org/r/71711/


was (Author: abstractdog):
[~kgyrtkirk], [~pvary]: could you please take a look? the patch contains the following:
1.  Database rule refactoring in standalone-metastore project
2. Qtest infrastructure: preparing qtest infra for being able to run any qtest against any kind of metastores
3. A bunch of cleanups in the current qtest mess

With the patch, you can run the following (if your docker is set up properly, meaning that you can run the metastore integration tests, already described in standalone-metastore/DEV-README):
{code}
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q -Dtest.metastore.db=mssql
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q -Dtest.metastore.db=mysql
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q -Dtest.metastore.db=postgres
mvn test -Dtest.output.overwrite=true -Pitests -pl itests/qtest -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q -Dtest.metastore.db=oracle -Ditest.jdbc.jars=/path/to/your/god/damn/oracle/jdbc/driver/ojdbc6.jar          
{code}

> QTest: support for running qtests on various metastore DBs
> ----------------------------------------------------------
>
>                 Key: HIVE-21954
>                 URL: https://issues.apache.org/jira/browse/HIVE-21954
>             Project: Hive
>          Issue Type: Improvement
>          Components: Standalone Metastore, Testing Infrastructure
>            Reporter: László Bodor
>            Assignee: László Bodor
>            Priority: Major
>             Fix For: 4.0.0
>
>         Attachments: HIVE-21954.01.patch, HIVE-21954.02.patch, HIVE-21954.03.patch, HIVE-21954.03.patch, HIVE-21954.03.patch, HIVE-21954.04.patch, HIVE-21954.05.patch, HIVE-21954.07.patch, HIVE-21954.07.patch, HIVE-21954.08.patch
>
>
> In HIVE-21940, a postgres metastore related issue has been fixed, and a local reproduction has been provided.
> {code}
> export QTEST_LEAVE_FILES=true
> docker kill metastore-test-postgres-install
> docker rm metastore-test-postgres-install
> cd standalone-metastore
> mvn verify -DskipITests=false -Dit.test=ITestPostgres#install -Dtest=nosuch -Dmetastore.itest.no.stop.container=true
> cd ..
> mvn test -Dtest.output.overwrite=true -Pitests,hadoop-2 -pl itests/qtest -Dtest=TestCliDriver -Dqfile=partition_params_postgres.q -Dhive.metastore.rawstore.impl=org.apache.hadoop.hive.metastore.ObjectStore
> {code}
> The problem with this solution is that data/conf/hive-site.xml has to be edited manually. My proposal is to introduce a property (-Dmetastore.db=postgres), which can take care of the parameters on the fly. 2 supported solutions could be:
> 1. simple parameters: -Dmetastore.db=postgres
> In this case, tests depend on settings from ITestPostgres class (password, db, etc.)
> 2. verbose but flexible parameters: [see hive-site.xml HIVE-21940's repro patch|https://issues.apache.org/jira/secure/attachment/12973534/HIVE-21940.repro.patch] 
> In the first implementation, I would not start metastore db automatically (which is done be 'mvn verify ...'), but it's still under planning. 
> In the long term, we should consider running this kind of tests in precommit phase, so maybe -Dmetastore.db=postgres could start metastore db automatically. Also we should consider running some qtests on various metastores. I would not pick randomly, but choose some "metastore-heavy" ones instead.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)