You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "David Mollitor (Jira)" <ji...@apache.org> on 2019/11/07 20:10:00 UTC
[jira] [Comment Edited] (HIVE-22469) Lower Metastore DB Connection
Pool Size in QTests
[ https://issues.apache.org/jira/browse/HIVE-22469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969533#comment-16969533 ]
David Mollitor edited comment on HIVE-22469 at 11/7/19 8:09 PM:
----------------------------------------------------------------
Managing 10 connections, bunch of extra noise in the logs.
{code:none}
2019-11-06T11:45:30,741 DEBUG [HikariPool-3 housekeeper] pool.HikariPool: HikariPool-3 - Pool stats (total=10, active=0, idle=10, waiting=0)
2019-11-06T11:45:30,741 DEBUG [HikariPool-2 housekeeper] pool.HikariPool: HikariPool-2 - Pool stats (total=10, active=1, idle=9, waiting=0)
2019-11-06T11:45:40,809 DEBUG [HikariPool-1 connection closer] pool.PoolBase: HikariPool-1 - Closing connection org.apache.derby.impl.jdbc.EmbedConnection@442851558 (XID = 184), (SESSIONID = 19), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:45:40,812 DEBUG [HikariPool-1 connection adder] pool.HikariPool: HikariPool-1 - Added connection org.apache.derby.impl.jdbc.EmbedConnection@2095252197 (XID = 17066), (SESSIONID = 3), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null)
2019-11-06T11:45:49,020 DEBUG [HikariPool-1 connection closer] pool.PoolBase: HikariPool-1 - Closing connection org.apache.derby.impl.jdbc.EmbedConnection@416110958 (XID = 174), (SESSIONID = 9), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:45:49,022 DEBUG [HikariPool-1 connection adder] pool.HikariPool: HikariPool-1 - Added connection org.apache.derby.impl.jdbc.EmbedConnection@73417930 (XID = 17068), (SESSIONID = 5), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null)
2019-11-06T11:45:49,363 DEBUG [HikariPool-1 connection closer] pool.PoolBase: HikariPool-1 - Closing connection org.apache.derby.impl.jdbc.EmbedConnection@1998766497 (XID = 180), (SESSIONID = 15), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:45:49,364 DEBUG [HikariPool-1 connection adder] pool.HikariPool: HikariPool-1 - Added connection org.apache.derby.impl.jdbc.EmbedConnection@1163742961 (XID = 17070), (SESSIONID = 7), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null)
2019-11-06T11:45:52,762 DEBUG [HikariPool-1 housekeeper] pool.HikariPool: HikariPool-1 - Pool stats (total=10, active=0, idle=10, waiting=0)
2019-11-06T11:45:53,260 DEBUG [HikariPool-1 connection closer] pool.PoolBase: HikariPool-1 - Closing connection org.apache.derby.impl.jdbc.EmbedConnection@478356754 (XID = 178), (SESSIONID = 13), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:45:53,261 DEBUG [HikariPool-1 connection adder] pool.HikariPool: HikariPool-1 - Added connection org.apache.derby.impl.jdbc.EmbedConnection@1959963107 (XID = 17072), (SESSIONID = 9), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null)
2019-11-06T11:46:00,298 DEBUG [HikariPool-1 connection closer] pool.PoolBase: HikariPool-1 - Closing connection org.apache.derby.impl.jdbc.EmbedConnection@605147393 (XID = 182), (SESSIONID = 17), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:46:00,299 DEBUG [HikariPool-1 connection adder] pool.HikariPool: HikariPool-1 - Added connection org.apache.derby.impl.jdbc.EmbedConnection@245074122 (XID = 17074), (SESSIONID = 11), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null)
2019-11-06T11:46:00,741 DEBUG [HikariPool-3 housekeeper] pool.HikariPool: HikariPool-3 - Pool stats (total=10, active=0, idle=10, waiting=0)
2019-11-06T11:46:00,741 DEBUG [HikariPool-2 housekeeper] pool.HikariPool: HikariPool-2 - Pool stats (total=10, active=1, idle=9, waiting=0)
{code}
was (Author: belugabehr):
Managing 10 connections, bunch of extra noise in the logs.
{code}
2019-11-06T11:45:30,741 DEBUG [HikariPool-3 housekeeper] pool.HikariPool: HikariPool-3 - Pool stats (total=10, active=0, idle=10, waiting=0)
2019-11-06T11:45:30,741 DEBUG [HikariPool-2 housekeeper] pool.HikariPool: HikariPool-2 - Pool stats (total=10, active=1, idle=9, waiting=0)
2019-11-06T11:45:40,809 DEBUG [HikariPool-1 connection closer] pool.PoolBase: HikariPool-1 - Closing connection org.apache.derby.impl.jdbc.EmbedConnection@442851558 (XID = 184), (SESSIONID = 19), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:45:40,812 DEBUG [HikariPool-1 connection adder] pool.HikariPool: HikariPool-1 - Added connection org.apache.derby.impl.jdbc.EmbedConnection@2095252197 (XID = 17066), (SESSIONID = 3), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null)
2019-11-06T11:45:49,020 DEBUG [HikariPool-1 connection closer] pool.PoolBase: HikariPool-1 - Closing connection org.apache.derby.impl.jdbc.EmbedConnection@416110958 (XID = 174), (SESSIONID = 9), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:45:49,022 DEBUG [HikariPool-1 connection adder] pool.HikariPool: HikariPool-1 - Added connection org.apache.derby.impl.jdbc.EmbedConnection@73417930 (XID = 17068), (SESSIONID = 5), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null)
2019-11-06T11:45:49,363 DEBUG [HikariPool-1 connection closer] pool.PoolBase: HikariPool-1 - Closing connection org.apache.derby.impl.jdbc.EmbedConnection@1998766497 (XID = 180), (SESSIONID = 15), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:45:49,364 DEBUG [HikariPool-1 connection adder] pool.HikariPool: HikariPool-1 - Added connection org.apache.derby.impl.jdbc.EmbedConnection@1163742961 (XID = 17070), (SESSIONID = 7), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null)
2019-11-06T11:45:52,762 DEBUG [HikariPool-1 housekeeper] pool.HikariPool: HikariPool-1 - Pool stats (total=10, active=0, idle=10, waiting=0)
2019-11-06T11:45:53,260 DEBUG [HikariPool-1 connection closer] pool.PoolBase: HikariPool-1 - Closing connection org.apache.derby.impl.jdbc.EmbedConnection@478356754 (XID = 178), (SESSIONID = 13), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:45:53,261 DEBUG [HikariPool-1 connection adder] pool.HikariPool: HikariPool-1 - Added connection org.apache.derby.impl.jdbc.EmbedConnection@1959963107 (XID = 17072), (SESSIONID = 9), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null)
2019-11-06T11:46:00,298 DEBUG [HikariPool-1 connection closer] pool.PoolBase: HikariPool-1 - Closing connection org.apache.derby.impl.jdbc.EmbedConnection@605147393 (XID = 182), (SESSIONID = 17), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null) : (connection has passed maxLifetime)
2019-11-06T11:46:00,299 DEBUG [HikariPool-1 connection adder] pool.HikariPool: HikariPool-1 - Added connection org.apache.derby.impl.jdbc.EmbedConnection@245074122 (XID = 17074), (SESSIONID = 11), (DATABASE = /home/hiveptest/35.193.82.55-hiveptest-2/apache-github-source-source/itests/qtest/target/tmp/junit_metastore_db), (DRDAID = null)
2019-11-06T11:46:00,741 DEBUG [HikariPool-3 housekeeper] pool.HikariPool: HikariPool-3 - Pool stats (total=10, active=0, idle=10, waiting=0)
2019-11-06T11:46:00,741 DEBUG [HikariPool-2 housekeeper] pool.HikariPool: HikariPool-2 - Pool stats (total=10, active=1, idle=9, waiting=0)
{code}
> Lower Metastore DB Connection Pool Size in QTests
> -------------------------------------------------
>
> Key: HIVE-22469
> URL: https://issues.apache.org/jira/browse/HIVE-22469
> Project: Hive
> Issue Type: Improvement
> Components: Test, Tests
> Affects Versions: 3.2.0
> Reporter: David Mollitor
> Assignee: David Mollitor
> Priority: Minor
> Attachments: HIVE-22469.1.patch, HIVE-22469.2.patch
>
>
> Hive Metastore uses the 'HikariCP' database connection pool. The default number of connections to the database is 10. For the Qtests, connecting to a local DerbyDB, there need not be more than 2 connections. Anymore simply adds undo overhead and by looking at the QTest logs, I see a bunch of 'connection refused' from HikariCP. It may be the case that the standalone DB does not support that many concurrent connections anyway.
> [https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-MetaStore]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)