You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "mszurap (via GitHub)" <gi...@apache.org> on 2025/04/11 13:46:30 UTC
[PR] HIVE-28900: Make the objectstore-secondary connection pool size configurable [hive]
mszurap opened a new pull request, #5764:
URL: https://github.com/apache/hive/pull/5764
### What changes were proposed in this pull request?
The secondary connection factory is used for schema generation, and for value generation operations. Under normal circumstances 2 connections should be sufficient, and currently it is hardcoded to that value. We have found in [HIVE-28839](https://issues.apache.org/jira/browse/HIVE-28839) that connection starvation may still happen. (in that case due to a datanucleus bug)
We should make the "objectstore-secondary" pool size configurable. This is needed at least until a final fix is not available for [HIVE-28839](https://issues.apache.org/jira/browse/HIVE-28839), but probably it is good to have on the long run if we hit other edge cases.
### Why are the changes needed?
The "objectstore-secondary" pool size is currently hardcoded to 2.
The "datanucleus.connectionPool.maxPoolSize" controls the "objectstore" poolsize, I propose we introduce a new configuration as "datanucleus.connectionPool.maxSecondaryPoolSize" which is not an existing Datanucleus config, however we could still use that only in Hive to control the number of connections in the "objectstore-secondary" pool. If that config name is not desired, we can use a different name like we have for "metastore.compactor.connectionPool.maxPoolSize".
### Does this PR introduce _any_ user-facing change?
No
### Is the change a dependency upgrade?
No
### How was this patch tested?
As I see there are no automated tests for the pool handling, so I made a manual test:
- Set the "datanucleus.connectionPool.maxSecondaryPoolSize" new config item in the HMS hive-site.xml, and observed the new size is now is logged out:
`2025-04-11T13:09:25,881 INFO [main] datasource.HikariCPDataSourceProvider: Creating Hikari connection pool for the MetaStore, maxPoolSize: 4, name: objectstore-secondary`
- Observed that the number of database connections increased on my MySQL side ("show processlist" output)
- During testing found from the logs that with this change the "objectstore-compactor-secondary" pool size is also increased from 2 to the new size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28900: Make the objectstore-secondary connection pool size configurable [hive]
Posted by "mszurap (via GitHub)" <gi...@apache.org>.
mszurap commented on code in PR #5764:
URL: https://github.com/apache/hive/pull/5764#discussion_r2042127429
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java:
##########
@@ -315,9 +315,12 @@ private static PersistenceManagerFactory initPMF(Configuration conf, boolean for
databaseProduct = DatabaseProduct.determineDatabaseProduct(ds, conf);
// The secondary connection factory is used for schema generation, and for value generation operations.
// We should use a different pool for the secondary connection factory to avoid resource starvation.
- // Since DataNucleus uses locks for schema generation and value generation, 2 connections should be sufficient.
+ // DataNucleus uses locks for schema generation and value generation, under normal circumstances 2 connections
+ // should be sufficient. However, as found in HIVE-28839 in certain situations connection starvation may happen,
+ // so we need to make this configurable until a final fix is not available.
Review Comment:
Thanks @zabetak for the suggestions. I agree, removed the confusing comments/descriptions and renamed the property to "datanucleus.connectionPool.secondary.maxPoolSize".
Added a new testcase TestDataSourceProviderFactory#testHikariCpMaxPoolSize, it tests the HikariCP pool sizes only. I could not write a testcase for the DBCP as it does not expose the pool's properties and did not want to use reflection, but let me know if that is also okay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28900: Make the objectstore-secondary connection pool size configurable [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on PR #5764:
URL: https://github.com/apache/hive/pull/5764#issuecomment-2812605690
Thanks for the PR @mszurap !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28900: Make the objectstore-secondary connection pool size configurable [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5764:
URL: https://github.com/apache/hive/pull/5764#issuecomment-2801954442
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5764) **Quality Gate passed**
Issues
 [1 New issue](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5764&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5764&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5764&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5764&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5764&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5764)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28900: Make the objectstore-secondary connection pool size configurable [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5764:
URL: https://github.com/apache/hive/pull/5764#issuecomment-2802739317
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5764) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5764&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5764&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5764&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5764&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5764&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5764)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28900: Make the objectstore-secondary connection pool size configurable [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5764:
URL: https://github.com/apache/hive/pull/5764#issuecomment-2797737192
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5764) **Quality Gate passed**
Issues
 [0 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5764&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5764&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5764&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5764&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5764&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5764)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28900: Make the objectstore-secondary connection pool size configurable [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak merged PR #5764:
URL: https://github.com/apache/hive/pull/5764
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28900: Make the objectstore-secondary connection pool size configurable [hive]
Posted by "zabetak (via GitHub)" <gi...@apache.org>.
zabetak commented on code in PR #5764:
URL: https://github.com/apache/hive/pull/5764#discussion_r2041704820
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -700,6 +700,12 @@ public enum ConfVars {
"configured with embedded metastore. To get optimal performance, set config to meet the following condition\n"+
"(2 * pool_size * metastore_instances + 2 * pool_size * HS2_instances_with_embedded_metastore) = \n" +
"(2 * physical_core_count + hard_disk_count)."),
+ CONNECTION_POOLING_MAX_SECONDARY_CONNECTIONS("datanucleus.connectionPool.maxSecondaryPoolSize",
+ "datanucleus.connectionPool.maxSecondaryPoolSize", 2,
+ "Specify the maximum number of connections in the objectstore-secondary connection pool.\n" +
+ "Same as datanucleus.connectionPool.maxPoolSize, but for the objectstore-secondary pool. \n" +
+ "Note that this is not a valid datanucleus configuration, this is used only in Hive.\n" +
Review Comment:
The comment it is not a valid configuration is a bit confusing. Note that `datanucleus.connectionPool.maxPoolSize` is not passed directly to datanucleus and in fact it is only used by Hive. It may be better to not mention anything.
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/PersistenceManagerProvider.java:
##########
@@ -315,9 +315,12 @@ private static PersistenceManagerFactory initPMF(Configuration conf, boolean for
databaseProduct = DatabaseProduct.determineDatabaseProduct(ds, conf);
// The secondary connection factory is used for schema generation, and for value generation operations.
// We should use a different pool for the secondary connection factory to avoid resource starvation.
- // Since DataNucleus uses locks for schema generation and value generation, 2 connections should be sufficient.
+ // DataNucleus uses locks for schema generation and value generation, under normal circumstances 2 connections
+ // should be sufficient. However, as found in HIVE-28839 in certain situations connection starvation may happen,
+ // so we need to make this configurable until a final fix is not available.
Review Comment:
As far as I understand the intention is to keep this config around even after fixing HIVE-28839. If my understanding is correct then it's better to remove the comment "However...is not available".
If it is really meant to be a temporary config till we merge HIVE-28839 then it's fine to leave things as is. Maybe just fix the typo "fix is not available" -> "fix is available".
##########
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/conf/MetastoreConf.java:
##########
@@ -700,6 +700,12 @@ public enum ConfVars {
"configured with embedded metastore. To get optimal performance, set config to meet the following condition\n"+
"(2 * pool_size * metastore_instances + 2 * pool_size * HS2_instances_with_embedded_metastore) = \n" +
"(2 * physical_core_count + hard_disk_count)."),
+ CONNECTION_POOLING_MAX_SECONDARY_CONNECTIONS("datanucleus.connectionPool.maxSecondaryPoolSize",
Review Comment:
I would prefer slightly more a name like `datanucleus.connectionPool.secondary.maxPoolSize` or `datanucleus.connectionPool.nontx.maxPoolSize` but don't feel very strong about it.
The datanucleus prefix is somewhat misleading since we are not propagating the property to datanucleus but given the current situation and the other props don't have a better suggestion so we can leave with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org