You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Chris Nauroth <cn...@apache.org> on 2022/10/28 06:41:37 UTC
Updating Wiki about Hikari Configuration Properties
Hi everyone,
Regarding this page:
https://cwiki.apache.org/confluence/display/hive/configuration+properties#ConfigurationProperties-HiveMetastoreConnectionPoolingConfiguration
It states that the metastore's Hikari connection pool can be configured by
specifying properties prefixed as "hikari". This is not quite correct. In
HIVE-17317, there was a bug fix made to the Hikari integration such that
the proper prefix is "hikaricp". For example:
<property>
<name>hikaricp.minimumIdle</name>
<value>4</value>
<final>false</final>
<source>Dataproc Cluster Properties</source>
</property>
Could you please grant access to me (cnauroth@apache.org) to update the
page? If you prefer not to grant access, could a Hive committer make the
change for me?
BTW, the reason I discovered this is that I recently upgraded a cluster
from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). After the
upgrade, I found that HiveMetaStore was generating far more database
connections at baseline, putting extra burden on the database. It appears
that BoneCP default behavior (4 idle connections) is different from
HikariCP default behavior (idle connections equal to max connections which
is 10). This put me down the path of wanting to control Hikari's
minimumIdle setting and then finding this discrepancy in the documentation.
Passing on this information in case others are seeing unusually high
connection counts after an upgrade to 3.x.
Chris Nauroth
Re: Updating Wiki about Hikari Configuration Properties
Posted by Chris Nauroth <cn...@apache.org>.
That did the trick! Thank you. I have updated the page.
Chris Nauroth
On Fri, Oct 28, 2022 at 11:47 PM Denys Kuzmenko
<dk...@cloudera.com.invalid> wrote:
> Hi Chris,
>
> Previously I just gave you edit permissions for the requested page, but now
> granted Hive space permissions. Please check again and let me know.
>
> On Fri, Oct 28, 2022 at 7:51 PM Chris Nauroth <cn...@apache.org> wrote:
>
> > Thank you, Denys. Unfortunately, I still don't see the Edit button like I
> > do on other projects where I have access (Hadoop, Zookeeper). I just
> > realized my Confluence login is actually just "cnauroth", so maybe I
> caused
> > some confusion by stating it as "cnauroth@apache.org"?
> >
> > Ayush, to further clarify, yes, I am planning to backport HIVE-17317 onto
> > versions 3.1.2 and 3.1.3 of Dataproc's Hive distro. I've already tested
> > locally and confirmed that I can use hikaricp.minimumIdle to limit the
> > number of connections. I'd also be happy to send an upstream pull request
> > against branch-3.1 for the backport if the community would find that
> > valuable. Let me know.
> >
> > When I make the Confluence edit, I'll clarify which versions have this
> > support.
> >
> > Chris Nauroth
> >
> >
> > On Fri, Oct 28, 2022 at 4:37 AM Ayush Saxena <ay...@gmail.com> wrote:
> >
> > > Hi Chris,
> > >
> > > Just curious:
> > >>
> > >> BTW, the reason I discovered this is that I recently upgraded a
> cluster
> > >> from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP).
> > >
> > >
> > > As you said the prefix change happened in HIVE-17317 which is in
> > > 4.0.0-alpha-1 [1] and you migrated to Hive 3.x, Then how did you face
> > > this problem? Some patched version of Hive, or some missing FixVersion
> in
> > > the Jira?
> > >
> > > Good to mention the version post which the change in prefix happens in
> > the
> > > wiki as well, to avoid future confusions due to versions.
> > >
> > > -Ayush
> > >
> > > [1] https://issues.apache.org/jira/browse/HIVE-17317
> > >
> > > On Fri, 28 Oct 2022 at 17:02, Denys Kuzmenko
> > > <dk...@cloudera.com.invalid> wrote:
> > >
> > >> Hi Chris,
> > >>
> > >> Please try now
> > >>
> > >> On Fri, Oct 28, 2022 at 12:43 PM Stamatis Zampetakis <
> zabetak@gmail.com
> > >
> > >> wrote:
> > >>
> > >> > Hive PMC members can provide edit rights to the wiki.
> > >> >
> > >> > @Naveen, Dennys, Adam: Can someone please give write privileges to
> > >> Chris?
> > >> >
> > >> > Best,
> > >> > Stamatis
> > >> >
> > >> > On Fri, Oct 28, 2022 at 8:41 AM Chris Nauroth <cn...@apache.org>
> > >> wrote:
> > >> >
> > >> > > Hi everyone,
> > >> > >
> > >> > > Regarding this page:
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/hive/configuration+properties#ConfigurationProperties-HiveMetastoreConnectionPoolingConfiguration
> > >> > >
> > >> > > It states that the metastore's Hikari connection pool can be
> > >> configured
> > >> > by
> > >> > > specifying properties prefixed as "hikari". This is not quite
> > >> correct. In
> > >> > > HIVE-17317, there was a bug fix made to the Hikari integration
> such
> > >> that
> > >> > > the proper prefix is "hikaricp". For example:
> > >> > >
> > >> > > <property>
> > >> > > <name>hikaricp.minimumIdle</name>
> > >> > > <value>4</value>
> > >> > > <final>false</final>
> > >> > > <source>Dataproc Cluster Properties</source>
> > >> > > </property>
> > >> > >
> > >> > > Could you please grant access to me (cnauroth@apache.org) to
> update
> > >> the
> > >> > > page? If you prefer not to grant access, could a Hive committer
> make
> > >> the
> > >> > > change for me?
> > >> > >
> > >> > > BTW, the reason I discovered this is that I recently upgraded a
> > >> cluster
> > >> > > from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP).
> After
> > >> the
> > >> > > upgrade, I found that HiveMetaStore was generating far more
> database
> > >> > > connections at baseline, putting extra burden on the database. It
> > >> appears
> > >> > > that BoneCP default behavior (4 idle connections) is different
> from
> > >> > > HikariCP default behavior (idle connections equal to max
> connections
> > >> > which
> > >> > > is 10). This put me down the path of wanting to control Hikari's
> > >> > > minimumIdle setting and then finding this discrepancy in the
> > >> > documentation.
> > >> > >
> > >> > > Passing on this information in case others are seeing unusually
> high
> > >> > > connection counts after an upgrade to 3.x.
> > >> > >
> > >> > > Chris Nauroth
> > >> > >
> > >> >
> > >>
> > >
> >
>
Re: Updating Wiki about Hikari Configuration Properties
Posted by Denys Kuzmenko <dk...@cloudera.com.INVALID>.
Hi Chris,
Previously I just gave you edit permissions for the requested page, but now
granted Hive space permissions. Please check again and let me know.
On Fri, Oct 28, 2022 at 7:51 PM Chris Nauroth <cn...@apache.org> wrote:
> Thank you, Denys. Unfortunately, I still don't see the Edit button like I
> do on other projects where I have access (Hadoop, Zookeeper). I just
> realized my Confluence login is actually just "cnauroth", so maybe I caused
> some confusion by stating it as "cnauroth@apache.org"?
>
> Ayush, to further clarify, yes, I am planning to backport HIVE-17317 onto
> versions 3.1.2 and 3.1.3 of Dataproc's Hive distro. I've already tested
> locally and confirmed that I can use hikaricp.minimumIdle to limit the
> number of connections. I'd also be happy to send an upstream pull request
> against branch-3.1 for the backport if the community would find that
> valuable. Let me know.
>
> When I make the Confluence edit, I'll clarify which versions have this
> support.
>
> Chris Nauroth
>
>
> On Fri, Oct 28, 2022 at 4:37 AM Ayush Saxena <ay...@gmail.com> wrote:
>
> > Hi Chris,
> >
> > Just curious:
> >>
> >> BTW, the reason I discovered this is that I recently upgraded a cluster
> >> from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP).
> >
> >
> > As you said the prefix change happened in HIVE-17317 which is in
> > 4.0.0-alpha-1 [1] and you migrated to Hive 3.x, Then how did you face
> > this problem? Some patched version of Hive, or some missing FixVersion in
> > the Jira?
> >
> > Good to mention the version post which the change in prefix happens in
> the
> > wiki as well, to avoid future confusions due to versions.
> >
> > -Ayush
> >
> > [1] https://issues.apache.org/jira/browse/HIVE-17317
> >
> > On Fri, 28 Oct 2022 at 17:02, Denys Kuzmenko
> > <dk...@cloudera.com.invalid> wrote:
> >
> >> Hi Chris,
> >>
> >> Please try now
> >>
> >> On Fri, Oct 28, 2022 at 12:43 PM Stamatis Zampetakis <zabetak@gmail.com
> >
> >> wrote:
> >>
> >> > Hive PMC members can provide edit rights to the wiki.
> >> >
> >> > @Naveen, Dennys, Adam: Can someone please give write privileges to
> >> Chris?
> >> >
> >> > Best,
> >> > Stamatis
> >> >
> >> > On Fri, Oct 28, 2022 at 8:41 AM Chris Nauroth <cn...@apache.org>
> >> wrote:
> >> >
> >> > > Hi everyone,
> >> > >
> >> > > Regarding this page:
> >> > >
> >> > >
> >> > >
> >> >
> >>
> https://cwiki.apache.org/confluence/display/hive/configuration+properties#ConfigurationProperties-HiveMetastoreConnectionPoolingConfiguration
> >> > >
> >> > > It states that the metastore's Hikari connection pool can be
> >> configured
> >> > by
> >> > > specifying properties prefixed as "hikari". This is not quite
> >> correct. In
> >> > > HIVE-17317, there was a bug fix made to the Hikari integration such
> >> that
> >> > > the proper prefix is "hikaricp". For example:
> >> > >
> >> > > <property>
> >> > > <name>hikaricp.minimumIdle</name>
> >> > > <value>4</value>
> >> > > <final>false</final>
> >> > > <source>Dataproc Cluster Properties</source>
> >> > > </property>
> >> > >
> >> > > Could you please grant access to me (cnauroth@apache.org) to update
> >> the
> >> > > page? If you prefer not to grant access, could a Hive committer make
> >> the
> >> > > change for me?
> >> > >
> >> > > BTW, the reason I discovered this is that I recently upgraded a
> >> cluster
> >> > > from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). After
> >> the
> >> > > upgrade, I found that HiveMetaStore was generating far more database
> >> > > connections at baseline, putting extra burden on the database. It
> >> appears
> >> > > that BoneCP default behavior (4 idle connections) is different from
> >> > > HikariCP default behavior (idle connections equal to max connections
> >> > which
> >> > > is 10). This put me down the path of wanting to control Hikari's
> >> > > minimumIdle setting and then finding this discrepancy in the
> >> > documentation.
> >> > >
> >> > > Passing on this information in case others are seeing unusually high
> >> > > connection counts after an upgrade to 3.x.
> >> > >
> >> > > Chris Nauroth
> >> > >
> >> >
> >>
> >
>
Re: Updating Wiki about Hikari Configuration Properties
Posted by Chris Nauroth <cn...@apache.org>.
Thank you, Denys. Unfortunately, I still don't see the Edit button like I
do on other projects where I have access (Hadoop, Zookeeper). I just
realized my Confluence login is actually just "cnauroth", so maybe I caused
some confusion by stating it as "cnauroth@apache.org"?
Ayush, to further clarify, yes, I am planning to backport HIVE-17317 onto
versions 3.1.2 and 3.1.3 of Dataproc's Hive distro. I've already tested
locally and confirmed that I can use hikaricp.minimumIdle to limit the
number of connections. I'd also be happy to send an upstream pull request
against branch-3.1 for the backport if the community would find that
valuable. Let me know.
When I make the Confluence edit, I'll clarify which versions have this
support.
Chris Nauroth
On Fri, Oct 28, 2022 at 4:37 AM Ayush Saxena <ay...@gmail.com> wrote:
> Hi Chris,
>
> Just curious:
>>
>> BTW, the reason I discovered this is that I recently upgraded a cluster
>> from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP).
>
>
> As you said the prefix change happened in HIVE-17317 which is in
> 4.0.0-alpha-1 [1] and you migrated to Hive 3.x, Then how did you face
> this problem? Some patched version of Hive, or some missing FixVersion in
> the Jira?
>
> Good to mention the version post which the change in prefix happens in the
> wiki as well, to avoid future confusions due to versions.
>
> -Ayush
>
> [1] https://issues.apache.org/jira/browse/HIVE-17317
>
> On Fri, 28 Oct 2022 at 17:02, Denys Kuzmenko
> <dk...@cloudera.com.invalid> wrote:
>
>> Hi Chris,
>>
>> Please try now
>>
>> On Fri, Oct 28, 2022 at 12:43 PM Stamatis Zampetakis <za...@gmail.com>
>> wrote:
>>
>> > Hive PMC members can provide edit rights to the wiki.
>> >
>> > @Naveen, Dennys, Adam: Can someone please give write privileges to
>> Chris?
>> >
>> > Best,
>> > Stamatis
>> >
>> > On Fri, Oct 28, 2022 at 8:41 AM Chris Nauroth <cn...@apache.org>
>> wrote:
>> >
>> > > Hi everyone,
>> > >
>> > > Regarding this page:
>> > >
>> > >
>> > >
>> >
>> https://cwiki.apache.org/confluence/display/hive/configuration+properties#ConfigurationProperties-HiveMetastoreConnectionPoolingConfiguration
>> > >
>> > > It states that the metastore's Hikari connection pool can be
>> configured
>> > by
>> > > specifying properties prefixed as "hikari". This is not quite
>> correct. In
>> > > HIVE-17317, there was a bug fix made to the Hikari integration such
>> that
>> > > the proper prefix is "hikaricp". For example:
>> > >
>> > > <property>
>> > > <name>hikaricp.minimumIdle</name>
>> > > <value>4</value>
>> > > <final>false</final>
>> > > <source>Dataproc Cluster Properties</source>
>> > > </property>
>> > >
>> > > Could you please grant access to me (cnauroth@apache.org) to update
>> the
>> > > page? If you prefer not to grant access, could a Hive committer make
>> the
>> > > change for me?
>> > >
>> > > BTW, the reason I discovered this is that I recently upgraded a
>> cluster
>> > > from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). After
>> the
>> > > upgrade, I found that HiveMetaStore was generating far more database
>> > > connections at baseline, putting extra burden on the database. It
>> appears
>> > > that BoneCP default behavior (4 idle connections) is different from
>> > > HikariCP default behavior (idle connections equal to max connections
>> > which
>> > > is 10). This put me down the path of wanting to control Hikari's
>> > > minimumIdle setting and then finding this discrepancy in the
>> > documentation.
>> > >
>> > > Passing on this information in case others are seeing unusually high
>> > > connection counts after an upgrade to 3.x.
>> > >
>> > > Chris Nauroth
>> > >
>> >
>>
>
Re: Updating Wiki about Hikari Configuration Properties
Posted by Ayush Saxena <ay...@gmail.com>.
Hi Chris,
Just curious:
>
> BTW, the reason I discovered this is that I recently upgraded a cluster
> from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP).
As you said the prefix change happened in HIVE-17317 which is in
4.0.0-alpha-1 [1] and you migrated to Hive 3.x, Then how did you face
this problem? Some patched version of Hive, or some missing FixVersion in
the Jira?
Good to mention the version post which the change in prefix happens in the
wiki as well, to avoid future confusions due to versions.
-Ayush
[1] https://issues.apache.org/jira/browse/HIVE-17317
On Fri, 28 Oct 2022 at 17:02, Denys Kuzmenko <dk...@cloudera.com.invalid>
wrote:
> Hi Chris,
>
> Please try now
>
> On Fri, Oct 28, 2022 at 12:43 PM Stamatis Zampetakis <za...@gmail.com>
> wrote:
>
> > Hive PMC members can provide edit rights to the wiki.
> >
> > @Naveen, Dennys, Adam: Can someone please give write privileges to Chris?
> >
> > Best,
> > Stamatis
> >
> > On Fri, Oct 28, 2022 at 8:41 AM Chris Nauroth <cn...@apache.org>
> wrote:
> >
> > > Hi everyone,
> > >
> > > Regarding this page:
> > >
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/hive/configuration+properties#ConfigurationProperties-HiveMetastoreConnectionPoolingConfiguration
> > >
> > > It states that the metastore's Hikari connection pool can be configured
> > by
> > > specifying properties prefixed as "hikari". This is not quite correct.
> In
> > > HIVE-17317, there was a bug fix made to the Hikari integration such
> that
> > > the proper prefix is "hikaricp". For example:
> > >
> > > <property>
> > > <name>hikaricp.minimumIdle</name>
> > > <value>4</value>
> > > <final>false</final>
> > > <source>Dataproc Cluster Properties</source>
> > > </property>
> > >
> > > Could you please grant access to me (cnauroth@apache.org) to update
> the
> > > page? If you prefer not to grant access, could a Hive committer make
> the
> > > change for me?
> > >
> > > BTW, the reason I discovered this is that I recently upgraded a cluster
> > > from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). After
> the
> > > upgrade, I found that HiveMetaStore was generating far more database
> > > connections at baseline, putting extra burden on the database. It
> appears
> > > that BoneCP default behavior (4 idle connections) is different from
> > > HikariCP default behavior (idle connections equal to max connections
> > which
> > > is 10). This put me down the path of wanting to control Hikari's
> > > minimumIdle setting and then finding this discrepancy in the
> > documentation.
> > >
> > > Passing on this information in case others are seeing unusually high
> > > connection counts after an upgrade to 3.x.
> > >
> > > Chris Nauroth
> > >
> >
>
Re: Updating Wiki about Hikari Configuration Properties
Posted by Denys Kuzmenko <dk...@cloudera.com.INVALID>.
Hi Chris,
Please try now
On Fri, Oct 28, 2022 at 12:43 PM Stamatis Zampetakis <za...@gmail.com>
wrote:
> Hive PMC members can provide edit rights to the wiki.
>
> @Naveen, Dennys, Adam: Can someone please give write privileges to Chris?
>
> Best,
> Stamatis
>
> On Fri, Oct 28, 2022 at 8:41 AM Chris Nauroth <cn...@apache.org> wrote:
>
> > Hi everyone,
> >
> > Regarding this page:
> >
> >
> >
> https://cwiki.apache.org/confluence/display/hive/configuration+properties#ConfigurationProperties-HiveMetastoreConnectionPoolingConfiguration
> >
> > It states that the metastore's Hikari connection pool can be configured
> by
> > specifying properties prefixed as "hikari". This is not quite correct. In
> > HIVE-17317, there was a bug fix made to the Hikari integration such that
> > the proper prefix is "hikaricp". For example:
> >
> > <property>
> > <name>hikaricp.minimumIdle</name>
> > <value>4</value>
> > <final>false</final>
> > <source>Dataproc Cluster Properties</source>
> > </property>
> >
> > Could you please grant access to me (cnauroth@apache.org) to update the
> > page? If you prefer not to grant access, could a Hive committer make the
> > change for me?
> >
> > BTW, the reason I discovered this is that I recently upgraded a cluster
> > from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). After the
> > upgrade, I found that HiveMetaStore was generating far more database
> > connections at baseline, putting extra burden on the database. It appears
> > that BoneCP default behavior (4 idle connections) is different from
> > HikariCP default behavior (idle connections equal to max connections
> which
> > is 10). This put me down the path of wanting to control Hikari's
> > minimumIdle setting and then finding this discrepancy in the
> documentation.
> >
> > Passing on this information in case others are seeing unusually high
> > connection counts after an upgrade to 3.x.
> >
> > Chris Nauroth
> >
>
Re: Updating Wiki about Hikari Configuration Properties
Posted by Stamatis Zampetakis <za...@gmail.com>.
Hive PMC members can provide edit rights to the wiki.
@Naveen, Dennys, Adam: Can someone please give write privileges to Chris?
Best,
Stamatis
On Fri, Oct 28, 2022 at 8:41 AM Chris Nauroth <cn...@apache.org> wrote:
> Hi everyone,
>
> Regarding this page:
>
>
> https://cwiki.apache.org/confluence/display/hive/configuration+properties#ConfigurationProperties-HiveMetastoreConnectionPoolingConfiguration
>
> It states that the metastore's Hikari connection pool can be configured by
> specifying properties prefixed as "hikari". This is not quite correct. In
> HIVE-17317, there was a bug fix made to the Hikari integration such that
> the proper prefix is "hikaricp". For example:
>
> <property>
> <name>hikaricp.minimumIdle</name>
> <value>4</value>
> <final>false</final>
> <source>Dataproc Cluster Properties</source>
> </property>
>
> Could you please grant access to me (cnauroth@apache.org) to update the
> page? If you prefer not to grant access, could a Hive committer make the
> change for me?
>
> BTW, the reason I discovered this is that I recently upgraded a cluster
> from Hive 2.x (default BoneCP) to Hive 3.x (default HikariCP). After the
> upgrade, I found that HiveMetaStore was generating far more database
> connections at baseline, putting extra burden on the database. It appears
> that BoneCP default behavior (4 idle connections) is different from
> HikariCP default behavior (idle connections equal to max connections which
> is 10). This put me down the path of wanting to control Hikari's
> minimumIdle setting and then finding this discrepancy in the documentation.
>
> Passing on this information in case others are seeing unusually high
> connection counts after an upgrade to 3.x.
>
> Chris Nauroth
>