You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "zabetak (via GitHub)" <gi...@apache.org> on 2025/07/04 10:06:20 UTC
[PR] HIVE-24419: Refactor junit database rules to exploit testcontainers [hive]
zabetak opened a new pull request, #5928:
URL: https://github.com/apache/hive/pull/5928
### What changes were proposed in this pull request?
1. Use testcontainers library to manage the lifecycle of dockerized databases
2. Use the newest version of the library (1.21.3) everywhere
### Why are the changes needed?
1. Reduce maintenance burden since we no longer have to manually craft and fix docker handling logic
2. Stabilize CI and by avoiding common problems as port conflicts, container leaks, etc., which are handled automatically by the library
3. Enhance OS independence since we no longer call Runtime.exec methods. The underlying docker-java library takes care of using the appropriate APIs (REST, etc.)
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
```
mvn test -Dtest=TestMssql,TestDerby,TestMysql,TestOracle,TestPostgres -Dtest.groups= -Dverbose.schematool
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-24419: Refactor junit database rules to exploit testcontainers [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5928:
URL: https://github.com/apache/hive/pull/5928#discussion_r2218320945
##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/schematool/TestSchemaToolForMetastore.java:
##########
@@ -75,20 +69,13 @@ public class TestSchemaToolForMetastore {
private PrintStream outStream;
private SchemaToolTaskValidate validator;
- public TestSchemaToolForMetastore(DatabaseRule dbms){
- this.dbms = dbms;
+ public TestSchemaToolForMetastore(String dbType){
+ this.dbms = MetastoreRuleFactory.create(dbType);
}
@Parameterized.Parameters(name = "{0}")
- public static Collection<Object[]> databases() {
- List<Object[]> dbs = new ArrayList<>();
- dbs.add(new Object[] { new Derby(true) });
- dbs.add(new Object[] { new Mysql() });
- dbs.add(new Object[] { new Oracle() });
- dbs.add(new Object[] { new Postgres() });
- dbs.add(new Object[] { new Mariadb() });
- dbs.add(new Object[] { new Mssql() });
- return dbs;
+ public static Collection<String> databases() {
+ return Arrays.asList("derby.clean", "mysql", "oracle", "postgres", "mariadb", "mssql");
Review Comment:
why not simply "derby"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-24419: Refactor junit database rules to exploit testcontainers [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5928:
URL: https://github.com/apache/hive/pull/5928#issuecomment-3049481229
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5928) **Quality Gate passed**
Issues
 [8 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5928&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5928&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5928&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5928&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5928&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5928)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-24419: Refactor junit database rules to exploit testcontainers [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5928:
URL: https://github.com/apache/hive/pull/5928#discussion_r2218329959
##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/MetastoreRuleFactory.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.dbinstall.rules;
+
+/**
+ * A factory for creating a Metastore database rule for use in tests.
+ */
+public final class MetastoreRuleFactory {
+ private MetastoreRuleFactory() {
+ throw new IllegalStateException("Factory class should not be instantiated");
+ }
+
+ /**
+ * Creates a new Metastore rule based on the provided database type.
+ *
+ * @param dbType the type of database (e.g., "mysql", "postgres", etc.)
+ * @return a DatabaseRule instance for the specified database type
+ * @throws IllegalArgumentException if the database type is unsupported
+ */
+ public static DatabaseRule create(String dbType) {
+ return switch (dbType.toLowerCase()) {
+ case "mysql" -> new Mysql();
+ case "postgres" -> new Postgres();
+ case "postgres.tpcds" -> new PostgresTPCDS();
+ case "mariadb" -> new Mariadb();
+ case "derby" -> new Derby();
+ case "derby.clean" -> new Derby(true);
Review Comment:
why do we need `purgeInAfter` only for Derby?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-24419: Refactor junit database rules to exploit testcontainers [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5928:
URL: https://github.com/apache/hive/pull/5928#discussion_r2218320945
##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/tools/schematool/TestSchemaToolForMetastore.java:
##########
@@ -75,20 +69,13 @@ public class TestSchemaToolForMetastore {
private PrintStream outStream;
private SchemaToolTaskValidate validator;
- public TestSchemaToolForMetastore(DatabaseRule dbms){
- this.dbms = dbms;
+ public TestSchemaToolForMetastore(String dbType){
+ this.dbms = MetastoreRuleFactory.create(dbType);
}
@Parameterized.Parameters(name = "{0}")
- public static Collection<Object[]> databases() {
- List<Object[]> dbs = new ArrayList<>();
- dbs.add(new Object[] { new Derby(true) });
- dbs.add(new Object[] { new Mysql() });
- dbs.add(new Object[] { new Oracle() });
- dbs.add(new Object[] { new Postgres() });
- dbs.add(new Object[] { new Mariadb() });
- dbs.add(new Object[] { new Mssql() });
- return dbs;
+ public static Collection<String> databases() {
+ return Arrays.asList("derby.clean", "mysql", "oracle", "postgres", "mariadb", "mssql");
Review Comment:
why not simply "derby"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-24419: Refactor junit database rules to exploit testcontainers [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5928:
URL: https://github.com/apache/hive/pull/5928#issuecomment-3063775234
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5928) **Quality Gate passed**
Issues
 [2 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5928&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5928&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5928&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5928&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5928&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5928)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-24419: Refactor junit database rules to exploit testcontainers [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #5928:
URL: https://github.com/apache/hive/pull/5928#discussion_r2218465720
##########
standalone-metastore/metastore-server/src/test/java/org/apache/hadoop/hive/metastore/dbinstall/rules/MetastoreRuleFactory.java:
##########
@@ -0,0 +1,48 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.metastore.dbinstall.rules;
+
+/**
+ * A factory for creating a Metastore database rule for use in tests.
+ */
+public final class MetastoreRuleFactory {
+ private MetastoreRuleFactory() {
+ throw new IllegalStateException("Factory class should not be instantiated");
+ }
+
+ /**
+ * Creates a new Metastore rule based on the provided database type.
+ *
+ * @param dbType the type of database (e.g., "mysql", "postgres", etc.)
+ * @return a DatabaseRule instance for the specified database type
+ * @throws IllegalArgumentException if the database type is unsupported
+ */
+ public static DatabaseRule create(String dbType) {
+ return switch (dbType.toLowerCase()) {
+ case "mysql" -> new Mysql();
+ case "postgres" -> new Postgres();
+ case "postgres.tpcds" -> new PostgresTPCDS();
+ case "mariadb" -> new Mariadb();
+ case "derby" -> new Derby();
+ case "derby.clean" -> new Derby(true);
Review Comment:
In some places (e.g., in [QTestMetaStoreHandler](https://github.com/apache/hive/blob/93c765df0272523d5d70614b8ff422f1f2f2168a/itests/util/src/main/java/org/apache/hadoop/hive/ql/QTestMetaStoreHandler.java#L100)) there are some Derby specific optimizations to only truncate some tables instead of deleting the entire database for performance reasons.
Implementing similar functionality for other DBMS may be useful but out-of-scope of the current PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-24419: Refactor junit database rules to exploit testcontainers [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #5928:
URL: https://github.com/apache/hive/pull/5928#issuecomment-3095666780
@soumyakanti3578 Yes, we don't use containers for Derby because it runs in the same JVM with the tests for performance reasons. I am not sure if it makes sense to isolate Derby in its own container but this is a much bigger discussion that probably deserves its own ticket/PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-24419: Refactor junit database rules to exploit testcontainers [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak merged PR #5928:
URL: https://github.com/apache/hive/pull/5928
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-24419: Refactor junit database rules to exploit testcontainers [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5928:
URL: https://github.com/apache/hive/pull/5928#issuecomment-3036055536
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5928) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5928&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5928&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5928&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5928&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5928&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5928)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-24419: Refactor junit database rules to exploit testcontainers [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #5928:
URL: https://github.com/apache/hive/pull/5928#issuecomment-3101442406
Thanks for the reviews @soumyakanti3578 @deniskuzZ !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org