You are viewing a plain text version of this content. The canonical link for it is here.

Posted to gitbox@hive.apache.org by "zabetak (via GitHub)" <gi...@apache.org> on 2023/09/27 20:09:35 UTC

[GitHub] [hive] zabetak opened a new pull request, #4754: HIVE-27747: Generalize TestSchemaToolForMetastore to run on every suported DBMS

zabetak opened a new pull request, #4754:
URL: https://github.com/apache/hive/pull/4754

   ### What changes were proposed in this pull request?
   1. Parameterize the test using all available dockerized DBMS (except Mariadb which currently fails due to HIVE-27749)
   2. Refactor hardcoded references to derby with the appropriate database type
   3. Add missing semicolons in some statements in the test scripts
   4. Use appropriate DDL statements for renaming a table based on the underlying database
   5. Post-process DML statements in test scripts to:
   * quote table/column identifiers when necessary (only Postgres at the moment)
   * pick appropriate literal for indicating false in boolean/bit datatype based on the database
   
   ### Why are the changes needed?
   The goal is to make the tests run on every supported metastore DBMS (not only Derby), to increase code coverage for the SchemaTool. Working on this PR uncovered the following bugs:
   * https://issues.apache.org/jira/browse/HIVE-27748
   * https://issues.apache.org/jira/browse/HIVE-27749
   
   ### Does this PR introduce _any_ user-facing change?
   No
   
   ### Is the change a dependency upgrade?
   No
   
   ### How was this patch tested?
   N/A


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

Re: [PR] HIVE-27747: Generalize TestSchemaToolForMetastore to run on every suported DBMS [hive]

Posted by "zabetak (via GitHub)" <gi...@apache.org>.

zabetak closed pull request #4754: HIVE-27747: Generalize TestSchemaToolForMetastore to run on every suported DBMS
URL: https://github.com/apache/hive/pull/4754


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

Re: [PR] HIVE-27747: Generalize TestSchemaToolForMetastore to run on every suported DBMS [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.

dengzhhu653 commented on PR #4754:
URL: https://github.com/apache/hive/pull/4754#issuecomment-1754619038

   Thanks @zabetak!
   I don't realize we have `DbInstallBase` to test default installation and validation, this `TestSchemaToolForMetastore` can act as a validator for some corner cases, so it makes sense to me.
   Regarding the different dbs, I think it's ok to run them all in a bulk, as we don't disable any type of `DbInstallBase`.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

[GitHub] [hive] sonarcloud[bot] commented on pull request #4754: HIVE-27747: Generalize TestSchemaToolForMetastore to run on every suported DBMS

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.

sonarcloud[bot] commented on PR #4754:
URL: https://github.com/apache/hive/pull/4754#issuecomment-1738379750

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4754)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4754&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4754&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4754&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4754&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4754&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4754&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4754&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4754&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4754&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4754&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4754&resolved=false&types=CODE_SMELL) [0 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4754&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4754&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4754&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

Re: [PR] HIVE-27747: Generalize TestSchemaToolForMetastore to run on every suported DBMS [hive]

Posted by "dengzhhu653 (via GitHub)" <gi...@apache.org>.

dengzhhu653 commented on PR #4754:
URL: https://github.com/apache/hive/pull/4754#issuecomment-1752442822

   Changes look good to me. The only concern is do we need to run this test against multiple db every time? we also automatically trigger a Metastore initialization against these DBMS in Jenkins build, is there a chance to move the SchemaTool validation at the same place?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org

Re: [PR] HIVE-27747: Generalize TestSchemaToolForMetastore to run on every suported DBMS [hive]

Posted by "zabetak (via GitHub)" <gi...@apache.org>.

zabetak commented on PR #4754:
URL: https://github.com/apache/hive/pull/4754#issuecomment-1752964093

> Changes look good to me. The only concern is do we need to run this test against multiple dbs every time?

Indeed the tests are somewhat lengthy when using the docker containers. I am fine to let's say run them only for one but I am afraid they will be broken quite soon. It would be nice if we can find a way to keep them active and functional.

One option to reduce the duration and still have regular coverage is to run them for only one DB but in a randomized manner. For instance, MySQL runs on Monday, Derby runs on Tuesday, MSSQL runs on Wednesday, etc., or something along these lines.

Another idea would be to run them all but only once daily. This probably would require setting more CI jobs etc., but it is still an option.

I am also open to other ideas and suggestions if any.

> we also automatically trigger a Metastore initialization against these DBMS in Jenkins build, is there a chance to move the SchemaTool validation to the same place?

This is would be a completely different PR since there is no something that we could use from here. Going into this direction would probably be a PR for https://github.com/kgyrtkirk/hive-dev-box not for Hive.

Note that under `DbInstallBase` we already have default installation and validation for most supported DBMS. These tests however, do not check what happens when there are errors.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org