You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Nicolas Vazquez <Ni...@shapeblue.com> on 2018/03/13 01:32:08 UTC
[DISCUSS] CloudStack Connection Pools
Hi all,
I would like to introduce a topic for discussion, regarding DB connection pools used in CloudStack, currently Apache Commons DBCP 1.4 (http://commons.apache.org/) is used. I've been investigating this topic as we are having complains of random issues on MySQL connection pool on large environments. Please let me know if this topic has already been discussed before.
First of all, DBCP 1.4 has been released on 2010 (https://commons.apache.org/proper/commons-dbcp/changes-report.html), and no minor/patch version has been released since then. It seems to work in high performance with relatively low traffic and low load applications. However, it is single threaded, and in order to be thread-safe, the entire pool needs to be locked. It is also reported that an CPU and concurrent threads increases, the performance gets affected. This is a serious issue on highly concurrent systems, such as CloudStack.
I've been investigating some options to replace it:
- The first option can be upgrading to version 2.x. Issues on performance and concurrency could be solved using this version.
- Tomcat JDBC Connection Pool. Please check: https://tomcat.apache.org/tomcat-7.0-doc/jdbc-pool.html.
- Other replacement options found: BoneCP, C3P0, HikariCP
Given these options, I've been looking for benchmarks to compare them (*). Looks like HikariCP (http://brettwooldridge.github.io/HikariCP/) could be the best replacement, improving performance and stability. Another good replacement option could be Tomcat.
I've been also examining the codebase, data source initialization is done on TransactionLegacy class under the cloud-framework-db project. Replacement work should be done on this class. Instead of pure replacement, a global setting can be introduced to make the admins able to select which connection pool to use.
What do you think? Any possitive/negative feedback is welcome as well as new ideas. As mentioned before, I don't know if it has been discussed before, sorry in advance if it has.
Kind regards,
Nicolas
(*) Links to benchmarks and comparissons:
https://www.wix.engineering/single-post/how-many-threads-does-it-take-to-fill-a-pool
https://www.wix.engineering/single-post/how-does-hikaricp-compare-to-other-connection-pools
<https://www.wix.engineering/single-post/how-does-hikaricp-compare-to-other-connection-pools>https://beansroasted.wordpress.com/2017/07/29/connection-pool-analysis/
https://beansroasted.wordpress.com/tag/connection-pool-comparison/
<https://beansroasted.wordpress.com/tag/connection-pool-comparison/>https://github.com/brettwooldridge/HikariCP/wiki/%22My-benchmark-doesn't-show-a-difference.%22
http://www.trustiv.co.uk/2014/06/battle-connection-pools
Nicolas.Vazquez@shapeblue.com
www.shapeblue.com
,
@shapeblue
Re: [DISCUSS] CloudStack Connection Pools
Posted by ilya musayev <il...@gmail.com>.
Rafael and Khosrow,
I actually think quite the opposite on DBCP development. Please pull up
latest commits and release notes on DBCP.
An assumption that its an Apache project and therefore will live and
flourish - is a dangerous one. It comes down to who supports the project
and if there funding to back it.
https://github.com/apache/commons-dbcp
While there are some commits - there is no major innovation happening and
its limited to few individuals.
In terms of performance HikariCP blows all other contenders by leaps and
bounds. It also holds ASF license - so we should have no issues
intergrating.
HikariCP-bench-2.6.0.png
<https://github.com/brettwooldridge/HikariCP/wiki/HikariCP-bench-2.6.0.png>
You can see active development and its backed by a company that is funded
sufficiently well.
C3P0 - is GNU licensed based - which if i understand correctly - cannot be
used with ASF project.
The reason it was proposed to allow a flexibility for connection pool
providers - is to allow for user to switch between one or the other - in
case you come across an issue with one of the two.
It would be something an advanced user knows and understands. Not providing
the flexibility to choose between different CP - makes troubleshooting
extremely difficult.
Lastly - DBCP 1.4 log4j support is none existent - therefore you cant
troubleshoot the connection pool issues with ease. We can upgrade to 2.x
and see if it helps - but given the slow development and performance
limitation - i dont see a huge benefit. Nevertheless, we know DBCP 2.x is
stable - it would be nice to have a fallback as an option in case the user
suspects his issues are due to connection pool handling.
Just point of reference, internally - some of apps switched from DBCP to
HikariCP and we saw tremendous boost in performance (10x+) and
troubleshooting became much easier. I dont want to assume we will get 10x
boost - but i think we will certainly do much better than what 8 year old
DBCP has to offer.
Regards,
ilya
On Wed, Mar 14, 2018 at 10:58 AM, Rafael Weingärtner <
rafaelweingartner@gmail.com> wrote:
> I agree with Khosrow. Even though the idea of externalizing a configuration
> like this seems interesting, I believe that it would bring more
> complications than benefits. And, at the end of the day operators would
> only use the default.
>
> On Wed, Mar 14, 2018 at 2:50 PM, Khosrow Moossavi <km...@cloudops.com>
> wrote:
>
> > Why would we want to expose this choice to administrator of Cloudstack
> > whose responsibility
> > is to keep it running and not knowing about the inner-mechanic of how it
> > works. right? It's not
> > like that we're giving them a choice of which database to connect to.
> >
> > So on that note, I would say we need to agree on any of those CP
> libraries
> > and implement, the
> > same way we chose for example log4j or slf4j over one another, or any
> other
> > _library_ we use.
> >
> > Khosrow Moossavi
> >
> > CloudOps
> >
> >
> >
> > On Wed, Mar 14, 2018 at 10:36 AM, Nicolas Vazquez <
> > Nicolas.Vazquez@shapeblue.com> wrote:
> >
> > > Thanks Khosrow and Rafael. You both agree on Spring Data as the best
> > > option, I see it would require a big effort and commitment to migrate
> to
> > > it, therefore it can take some (long) time to achieve it.
> > >
> > > As a more viable option, would you agree on supporting different
> > > connection pool management libraries and letting the administrator
> choose
> > > which one to use? (DBCP 1.4 as default)
> > >
> > > ________________________________
> > > From: Rafael Weingärtner <ra...@gmail.com>
> > > Sent: Tuesday, March 13, 2018 8:52:50 AM
> > > To: dev
> > > Subject: Re: [DISCUSS] CloudStack Connection Pools
> > >
> > > Spring data would be awesome. It is very flexible and has a very good
> > API.
> > > However, this would require commitment from our side to slowly migrate
> > > things to it.
> > >
> > > Regarding the connection pool management libraries; I would prefer
> either
> > > C3P0 or 2.* DBCP. The other two sound trendy, but I worry about this
> type
> > > of project in the long run. Both DBCP from Apache and C3P0 from
> Hibernate
> > > (RedHat) sound a more reasonable selection for me. They have been
> around
> > > for years, and have a solid community base already.
> > >
> > > On Mon, Mar 12, 2018 at 11:31 PM, Khosrow Moossavi <
> > kmoossavi@cloudops.com
> > > >
> > > wrote:
> > >
> > > > Hi Nicolas
> > > >
> > > > From my past experiences, I prefer 1) HikariCP 2) Tomcat Pool 3) C3P0
> > 4)
> > > > DBCP in that order. Although I don't have
> > > > any benchmark of my own to provide, and the ones you mentioned are
> > really
> > > > informative anyway.
> > > >
> > > > To me the broader subject is the _one_ who uses the pool, I mean if
> the
> > > > transactions are handled in a faster way and
> > > > released sooner and with shorter locks, generally speaking if it's
> more
> > > > efficient, I don't think from ACS point of view
> > > > there won't be much difference between the above mentioned options.
> > > >
> > > > On the same subject, it might be more interesting to use Spring Boot
> in
> > > > general and Spring Boot Data in particular
> > > > rather than only changing the CP functionality, and slowly
> > > refactor/retire
> > > > the DAO layer in favor of Spring Boot equivalent
> > > > implementation.
> > > >
> > > >
> > > > Khosrow Moossavi
> > > >
> > > > CloudOps
> > > >
> > > >
> > > >
> > > > On Mon, Mar 12, 2018 at 9:32 PM, Nicolas Vazquez <
> > > > Nicolas.Vazquez@shapeblue.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > >
> > > > > I would like to introduce a topic for discussion, regarding DB
> > > connection
> > > > > pools used in CloudStack, currently Apache Commons DBCP 1.4 (
> > > > > http://commons.apache.org/) is used. I've been investigating this
> > > topic
> > > > > as we are having complains of random issues on MySQL connection
> pool
> > on
> > > > > large environments. Please let me know if this topic has already
> been
> > > > > discussed before.
> > > > >
> > > > >
> > > > > First of all, DBCP 1.4 has been released on 2010 (
> > > > > https://commons.apache.org/proper/commons-dbcp/changes-report.html
> ),
> > > and
> > > > > no minor/patch version has been released since then. It seems to
> work
> > > in
> > > > > high performance with relatively low traffic and low load
> > applications.
> > > > > However, it is single threaded, and in order to be thread-safe, the
> > > > entire
> > > > > pool needs to be locked. It is also reported that an CPU and
> > concurrent
> > > > > threads increases, the performance gets affected. This is a serious
> > > issue
> > > > > on highly concurrent systems, such as CloudStack.
> > > > >
> > > > >
> > > > > I've been investigating some options to replace it:
> > > > > - The first option can be upgrading to version 2.x. Issues on
> > > performance
> > > > > and concurrency could be solved using this version.
> > > > > - Tomcat JDBC Connection Pool. Please check:
> > > https://tomcat.apache.org/
> > > > > tomcat-7.0-doc/jdbc-pool.html.
> > > > >
> > > > > - Other replacement options found: BoneCP, C3P0, HikariCP
> > > > >
> > > > >
> > > > > Given these options, I've been looking for benchmarks to compare
> them
> > > > (*).
> > > > > Looks like HikariCP (http://brettwooldridge.github.io/HikariCP/)
> > could
> > > > be
> > > > > the best replacement, improving performance and stability. Another
> > good
> > > > > replacement option could be Tomcat.
> > > > >
> > > > >
> > > > > I've been also examining the codebase, data source initialization
> is
> > > done
> > > > > on TransactionLegacy class under the cloud-framework-db project.
> > > > > Replacement work should be done on this class. Instead of pure
> > > > replacement,
> > > > > a global setting can be introduced to make the admins able to
> select
> > > > which
> > > > > connection pool to use.
> > > > >
> > > > > What do you think? Any possitive/negative feedback is welcome as
> well
> > > as
> > > > > new ideas. As mentioned before, I don't know if it has been
> discussed
> > > > > before, sorry in advance if it has.
> > > > >
> > > > > Kind regards,
> > > > > Nicolas
> > > > >
> > > > > (*) Links to benchmarks and comparissons:
> > > > > https://www.wix.engineering/single-post/how-many-threads-
> > > > > does-it-take-to-fill-a-pool
> > > > > https://www.wix.engineering/single-post/how-does-hikaricp-
> > > > > compare-to-other-connection-pools
> > > > > <https://www.wix.engineering/single-post/how-does-hikaricp-
> > > > > compare-to-other-connection-pools>https://beansroasted.
> > > > > wordpress.com/2017/07/29/connection-pool-analysis/
> > > > > https://beansroasted.wordpress.com/tag/connection-pool-comparison/
> > > > > <https://beansroasted.wordpress.com/tag/connection-
> pool-comparison/>
> > > > > https://github.com/brettwooldridge/HikariCP/wiki/%22My-benchmark-
> > > > > doesn't-show-a-difference.%22
> > > > > http://www.trustiv.co.uk/2014/06/battle-connection-pools
> > > > >
> > > > > Nicolas.Vazquez@shapeblue.com
> > > > > www.shapeblue.com<http://www.shapeblue.com>
> > > > > ,
> > > > > @shapeblue
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> > > Nicolas.Vazquez@shapeblue.com
> > > www.shapeblue.com
> > > ,
> > > @shapeblue
> > >
> > >
> > >
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>
Re: [DISCUSS] CloudStack Connection Pools
Posted by Rafael Weingärtner <ra...@gmail.com>.
I agree with Khosrow. Even though the idea of externalizing a configuration
like this seems interesting, I believe that it would bring more
complications than benefits. And, at the end of the day operators would
only use the default.
On Wed, Mar 14, 2018 at 2:50 PM, Khosrow Moossavi <km...@cloudops.com>
wrote:
> Why would we want to expose this choice to administrator of Cloudstack
> whose responsibility
> is to keep it running and not knowing about the inner-mechanic of how it
> works. right? It's not
> like that we're giving them a choice of which database to connect to.
>
> So on that note, I would say we need to agree on any of those CP libraries
> and implement, the
> same way we chose for example log4j or slf4j over one another, or any other
> _library_ we use.
>
> Khosrow Moossavi
>
> CloudOps
>
>
>
> On Wed, Mar 14, 2018 at 10:36 AM, Nicolas Vazquez <
> Nicolas.Vazquez@shapeblue.com> wrote:
>
> > Thanks Khosrow and Rafael. You both agree on Spring Data as the best
> > option, I see it would require a big effort and commitment to migrate to
> > it, therefore it can take some (long) time to achieve it.
> >
> > As a more viable option, would you agree on supporting different
> > connection pool management libraries and letting the administrator choose
> > which one to use? (DBCP 1.4 as default)
> >
> > ________________________________
> > From: Rafael Weingärtner <ra...@gmail.com>
> > Sent: Tuesday, March 13, 2018 8:52:50 AM
> > To: dev
> > Subject: Re: [DISCUSS] CloudStack Connection Pools
> >
> > Spring data would be awesome. It is very flexible and has a very good
> API.
> > However, this would require commitment from our side to slowly migrate
> > things to it.
> >
> > Regarding the connection pool management libraries; I would prefer either
> > C3P0 or 2.* DBCP. The other two sound trendy, but I worry about this type
> > of project in the long run. Both DBCP from Apache and C3P0 from Hibernate
> > (RedHat) sound a more reasonable selection for me. They have been around
> > for years, and have a solid community base already.
> >
> > On Mon, Mar 12, 2018 at 11:31 PM, Khosrow Moossavi <
> kmoossavi@cloudops.com
> > >
> > wrote:
> >
> > > Hi Nicolas
> > >
> > > From my past experiences, I prefer 1) HikariCP 2) Tomcat Pool 3) C3P0
> 4)
> > > DBCP in that order. Although I don't have
> > > any benchmark of my own to provide, and the ones you mentioned are
> really
> > > informative anyway.
> > >
> > > To me the broader subject is the _one_ who uses the pool, I mean if the
> > > transactions are handled in a faster way and
> > > released sooner and with shorter locks, generally speaking if it's more
> > > efficient, I don't think from ACS point of view
> > > there won't be much difference between the above mentioned options.
> > >
> > > On the same subject, it might be more interesting to use Spring Boot in
> > > general and Spring Boot Data in particular
> > > rather than only changing the CP functionality, and slowly
> > refactor/retire
> > > the DAO layer in favor of Spring Boot equivalent
> > > implementation.
> > >
> > >
> > > Khosrow Moossavi
> > >
> > > CloudOps
> > >
> > >
> > >
> > > On Mon, Mar 12, 2018 at 9:32 PM, Nicolas Vazquez <
> > > Nicolas.Vazquez@shapeblue.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > >
> > > > I would like to introduce a topic for discussion, regarding DB
> > connection
> > > > pools used in CloudStack, currently Apache Commons DBCP 1.4 (
> > > > http://commons.apache.org/) is used. I've been investigating this
> > topic
> > > > as we are having complains of random issues on MySQL connection pool
> on
> > > > large environments. Please let me know if this topic has already been
> > > > discussed before.
> > > >
> > > >
> > > > First of all, DBCP 1.4 has been released on 2010 (
> > > > https://commons.apache.org/proper/commons-dbcp/changes-report.html),
> > and
> > > > no minor/patch version has been released since then. It seems to work
> > in
> > > > high performance with relatively low traffic and low load
> applications.
> > > > However, it is single threaded, and in order to be thread-safe, the
> > > entire
> > > > pool needs to be locked. It is also reported that an CPU and
> concurrent
> > > > threads increases, the performance gets affected. This is a serious
> > issue
> > > > on highly concurrent systems, such as CloudStack.
> > > >
> > > >
> > > > I've been investigating some options to replace it:
> > > > - The first option can be upgrading to version 2.x. Issues on
> > performance
> > > > and concurrency could be solved using this version.
> > > > - Tomcat JDBC Connection Pool. Please check:
> > https://tomcat.apache.org/
> > > > tomcat-7.0-doc/jdbc-pool.html.
> > > >
> > > > - Other replacement options found: BoneCP, C3P0, HikariCP
> > > >
> > > >
> > > > Given these options, I've been looking for benchmarks to compare them
> > > (*).
> > > > Looks like HikariCP (http://brettwooldridge.github.io/HikariCP/)
> could
> > > be
> > > > the best replacement, improving performance and stability. Another
> good
> > > > replacement option could be Tomcat.
> > > >
> > > >
> > > > I've been also examining the codebase, data source initialization is
> > done
> > > > on TransactionLegacy class under the cloud-framework-db project.
> > > > Replacement work should be done on this class. Instead of pure
> > > replacement,
> > > > a global setting can be introduced to make the admins able to select
> > > which
> > > > connection pool to use.
> > > >
> > > > What do you think? Any possitive/negative feedback is welcome as well
> > as
> > > > new ideas. As mentioned before, I don't know if it has been discussed
> > > > before, sorry in advance if it has.
> > > >
> > > > Kind regards,
> > > > Nicolas
> > > >
> > > > (*) Links to benchmarks and comparissons:
> > > > https://www.wix.engineering/single-post/how-many-threads-
> > > > does-it-take-to-fill-a-pool
> > > > https://www.wix.engineering/single-post/how-does-hikaricp-
> > > > compare-to-other-connection-pools
> > > > <https://www.wix.engineering/single-post/how-does-hikaricp-
> > > > compare-to-other-connection-pools>https://beansroasted.
> > > > wordpress.com/2017/07/29/connection-pool-analysis/
> > > > https://beansroasted.wordpress.com/tag/connection-pool-comparison/
> > > > <https://beansroasted.wordpress.com/tag/connection-pool-comparison/>
> > > > https://github.com/brettwooldridge/HikariCP/wiki/%22My-benchmark-
> > > > doesn't-show-a-difference.%22
> > > > http://www.trustiv.co.uk/2014/06/battle-connection-pools
> > > >
> > > > Nicolas.Vazquez@shapeblue.com
> > > > www.shapeblue.com<http://www.shapeblue.com>
> > > > ,
> > > > @shapeblue
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
> > Nicolas.Vazquez@shapeblue.com
> > www.shapeblue.com
> > ,
> > @shapeblue
> >
> >
> >
> >
>
--
Rafael Weingärtner
Re: [DISCUSS] CloudStack Connection Pools
Posted by Nicolas Vazquez <Ni...@shapeblue.com>.
Thanks Ilya, Rafael and Khosrow for your feedback.
I think having the option to select between connection pools can be useful when experimenting DB issues and need troubleshooting. However, as an admin I would expect the connection pool decision to be made by CloudStack internally.
Upgrading to DBCP 2.x seems reasonable, DBCP 1.4 is compiled using Java 6 libraries and as we moved to Java 8 it would make sense to move to the newest version which runs on Java 7, following recommendations on: http://commons.apache.org/proper/commons-dbcp/
________________________________
From: Rafael Weingärtner <ra...@gmail.com>
Sent: Thursday, March 15, 2018 8:43:12 AM
To: dev
Subject: Re: [DISCUSS] CloudStack Connection Pools
Thanks for your feedback Ilya.
I would first say that most of the problems relating database are not due
to connection pool libraries, but rather to the DAO layer that is an ad-hoc
implementation of JPA. Not to mention transactions being managed in a not
so elegant and efficient fashion.
We can for sure externalize the selection of the database connection pool,
but my worry is how this will be implemented and maintained later. If it
becomes a neat and nice solution, it is awesome. However, if it locks down
different places in the code base, then we will have problems. If we were
using a framework to manage DAO connection and transaction, it would be a
matter of changing the connection pool bean, but I do not think that this
is the case in ACS right now.
Moreover, I have seen some of the tests between these database connection
pools. There seems to be an advantage on these new ones when you have lots
of idle connections, but when things start to get busy they seem to
converge to similar results. Therefore, the benefits between different
libraries will depend on different factors of the application.
I do prefer consolidated libraries; it does not mean that it is open for
debate and tests the selection of another one. I also do not see a problem
that DBCP is not getting thousands of commits every day. These are all very
concise libraries, it is normal to get less commit as time goes by and they
stabilize. All of Apache’s commons libraries are like this.
On Wed, Mar 14, 2018 at 4:17 PM, ilya musayev <il...@gmail.com>
wrote:
> When everything works smooth - the end user does not need to know whats
> under the hood.
>
> However, when things begin to swing and your hands are tied to just one CP
> - what do you do?
>
> On Wed, Mar 14, 2018 at 10:50 AM, Khosrow Moossavi <kmoossavi@cloudops.com
> >
> wrote:
>
> > Why would we want to expose this choice to administrator of Cloudstack
> > whose responsibility
> > is to keep it running and not knowing about the inner-mechanic of how it
> > works. right? It's not
> > like that we're giving them a choice of which database to connect to.
> >
> > So on that note, I would say we need to agree on any of those CP
> libraries
> > and implement, the
> > same way we chose for example log4j or slf4j over one another, or any
> other
> > _library_ we use.
> >
> > Khosrow Moossavi
> >
> > CloudOps
> >
> >
> >
> > On Wed, Mar 14, 2018 at 10:36 AM, Nicolas Vazquez <
> > Nicolas.Vazquez@shapeblue.com> wrote:
> >
> > > Thanks Khosrow and Rafael. You both agree on Spring Data as the best
> > > option, I see it would require a big effort and commitment to migrate
> to
> > > it, therefore it can take some (long) time to achieve it.
> > >
> > > As a more viable option, would you agree on supporting different
> > > connection pool management libraries and letting the administrator
> choose
> > > which one to use? (DBCP 1.4 as default)
> > >
> > > ________________________________
> > > From: Rafael Weingärtner <ra...@gmail.com>
> > > Sent: Tuesday, March 13, 2018 8:52:50 AM
> > > To: dev
> > > Subject: Re: [DISCUSS] CloudStack Connection Pools
> > >
> > > Spring data would be awesome. It is very flexible and has a very good
> > API.
> > > However, this would require commitment from our side to slowly migrate
> > > things to it.
> > >
> > > Regarding the connection pool management libraries; I would prefer
> either
> > > C3P0 or 2.* DBCP. The other two sound trendy, but I worry about this
> type
> > > of project in the long run. Both DBCP from Apache and C3P0 from
> Hibernate
> > > (RedHat) sound a more reasonable selection for me. They have been
> around
> > > for years, and have a solid community base already.
> > >
> > > On Mon, Mar 12, 2018 at 11:31 PM, Khosrow Moossavi <
> > kmoossavi@cloudops.com
> > > >
> > > wrote:
> > >
> > > > Hi Nicolas
> > > >
> > > > From my past experiences, I prefer 1) HikariCP 2) Tomcat Pool 3) C3P0
> > 4)
> > > > DBCP in that order. Although I don't have
> > > > any benchmark of my own to provide, and the ones you mentioned are
> > really
> > > > informative anyway.
> > > >
> > > > To me the broader subject is the _one_ who uses the pool, I mean if
> the
> > > > transactions are handled in a faster way and
> > > > released sooner and with shorter locks, generally speaking if it's
> more
> > > > efficient, I don't think from ACS point of view
> > > > there won't be much difference between the above mentioned options.
> > > >
> > > > On the same subject, it might be more interesting to use Spring Boot
> in
> > > > general and Spring Boot Data in particular
> > > > rather than only changing the CP functionality, and slowly
> > > refactor/retire
> > > > the DAO layer in favor of Spring Boot equivalent
> > > > implementation.
> > > >
> > > >
> > > > Khosrow Moossavi
> > > >
> > > > CloudOps
> > > >
> > > >
> > > >
> > > > On Mon, Mar 12, 2018 at 9:32 PM, Nicolas Vazquez <
> > > > Nicolas.Vazquez@shapeblue.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > >
> > > > > I would like to introduce a topic for discussion, regarding DB
> > > connection
> > > > > pools used in CloudStack, currently Apache Commons DBCP 1.4 (
> > > > > http://commons.apache.org/) is used. I've been investigating this
Apache Commons – Apache Commons<http://commons.apache.org/>
commons.apache.org
Apache Commons Proper. Commons Proper is dedicated to one principal goal: creating and maintaining reusable Java components. The Commons Proper is a place ...
> > > topic
> > > > > as we are having complains of random issues on MySQL connection
> pool
> > on
> > > > > large environments. Please let me know if this topic has already
> been
> > > > > discussed before.
> > > > >
> > > > >
> > > > > First of all, DBCP 1.4 has been released on 2010 (
> > > > > https://commons.apache.org/proper/commons-dbcp/changes-report.html
> ),
> > > and
> > > > > no minor/patch version has been released since then. It seems to
> work
> > > in
> > > > > high performance with relatively low traffic and low load
> > applications.
> > > > > However, it is single threaded, and in order to be thread-safe, the
> > > > entire
> > > > > pool needs to be locked. It is also reported that an CPU and
> > concurrent
> > > > > threads increases, the performance gets affected. This is a serious
> > > issue
> > > > > on highly concurrent systems, such as CloudStack.
> > > > >
> > > > >
> > > > > I've been investigating some options to replace it:
> > > > > - The first option can be upgrading to version 2.x. Issues on
> > > performance
> > > > > and concurrency could be solved using this version.
> > > > > - Tomcat JDBC Connection Pool. Please check:
> > > https://tomcat.apache.org/
> > > > > tomcat-7.0-doc/jdbc-pool.html.
> > > > >
> > > > > - Other replacement options found: BoneCP, C3P0, HikariCP
> > > > >
> > > > >
> > > > > Given these options, I've been looking for benchmarks to compare
> them
> > > > (*).
> > > > > Looks like HikariCP (http://brettwooldridge.github.io/HikariCP/)
> > could
> > > > be
> > > > > the best replacement, improving performance and stability. Another
> > good
> > > > > replacement option could be Tomcat.
> > > > >
> > > > >
> > > > > I've been also examining the codebase, data source initialization
> is
> > > done
> > > > > on TransactionLegacy class under the cloud-framework-db project.
> > > > > Replacement work should be done on this class. Instead of pure
> > > > replacement,
> > > > > a global setting can be introduced to make the admins able to
> select
> > > > which
> > > > > connection pool to use.
> > > > >
> > > > > What do you think? Any possitive/negative feedback is welcome as
> well
> > > as
> > > > > new ideas. As mentioned before, I don't know if it has been
> discussed
> > > > > before, sorry in advance if it has.
> > > > >
> > > > > Kind regards,
> > > > > Nicolas
> > > > >
> > > > > (*) Links to benchmarks and comparissons:
> > > > > https://www.wix.engineering/single-post/how-many-threads-
> > > > > does-it-take-to-fill-a-pool
> > > > > https://www.wix.engineering/single-post/how-does-hikaricp-
> > > > > compare-to-other-connection-pools
> > > > > <https://www.wix.engineering/single-post/how-does-hikaricp-
> > > > > compare-to-other-connection-pools>https://beansroasted.
> > > > > wordpress.com/2017/07/29/connection-pool-analysis/
> > > > > https://beansroasted.wordpress.com/tag/connection-pool-comparison/
> > > > > <https://beansroasted.wordpress.com/tag/connection-
> pool-comparison/>
> > > > > https://github.com/brettwooldridge/HikariCP/wiki/%22My-benchmark-
> > > > > doesn't-show-a-difference.%22
> > > > > http://www.trustiv.co.uk/2014/06/battle-connection-pools
> > > > >
> > > > > Nicolas.Vazquez@shapeblue.com
> > > > > www.shapeblue.com<http://www.shapeblue.com>
> > > > > ,
> > > > > @shapeblue
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> > > Nicolas.Vazquez@shapeblue.com
> > > www.shapeblue.com<http://www.shapeblue.com>
> > > ,
> > > @shapeblue
> > >
> > >
> > >
> > >
> >
>
--
Rafael Weingärtner
Nicolas.Vazquez@shapeblue.com
www.shapeblue.com
,
@shapeblue
Re: [DISCUSS] CloudStack Connection Pools
Posted by Rafael Weingärtner <ra...@gmail.com>.
Thanks for your feedback Ilya.
I would first say that most of the problems relating database are not due
to connection pool libraries, but rather to the DAO layer that is an ad-hoc
implementation of JPA. Not to mention transactions being managed in a not
so elegant and efficient fashion.
We can for sure externalize the selection of the database connection pool,
but my worry is how this will be implemented and maintained later. If it
becomes a neat and nice solution, it is awesome. However, if it locks down
different places in the code base, then we will have problems. If we were
using a framework to manage DAO connection and transaction, it would be a
matter of changing the connection pool bean, but I do not think that this
is the case in ACS right now.
Moreover, I have seen some of the tests between these database connection
pools. There seems to be an advantage on these new ones when you have lots
of idle connections, but when things start to get busy they seem to
converge to similar results. Therefore, the benefits between different
libraries will depend on different factors of the application.
I do prefer consolidated libraries; it does not mean that it is open for
debate and tests the selection of another one. I also do not see a problem
that DBCP is not getting thousands of commits every day. These are all very
concise libraries, it is normal to get less commit as time goes by and they
stabilize. All of Apache’s commons libraries are like this.
On Wed, Mar 14, 2018 at 4:17 PM, ilya musayev <il...@gmail.com>
wrote:
> When everything works smooth - the end user does not need to know whats
> under the hood.
>
> However, when things begin to swing and your hands are tied to just one CP
> - what do you do?
>
> On Wed, Mar 14, 2018 at 10:50 AM, Khosrow Moossavi <kmoossavi@cloudops.com
> >
> wrote:
>
> > Why would we want to expose this choice to administrator of Cloudstack
> > whose responsibility
> > is to keep it running and not knowing about the inner-mechanic of how it
> > works. right? It's not
> > like that we're giving them a choice of which database to connect to.
> >
> > So on that note, I would say we need to agree on any of those CP
> libraries
> > and implement, the
> > same way we chose for example log4j or slf4j over one another, or any
> other
> > _library_ we use.
> >
> > Khosrow Moossavi
> >
> > CloudOps
> >
> >
> >
> > On Wed, Mar 14, 2018 at 10:36 AM, Nicolas Vazquez <
> > Nicolas.Vazquez@shapeblue.com> wrote:
> >
> > > Thanks Khosrow and Rafael. You both agree on Spring Data as the best
> > > option, I see it would require a big effort and commitment to migrate
> to
> > > it, therefore it can take some (long) time to achieve it.
> > >
> > > As a more viable option, would you agree on supporting different
> > > connection pool management libraries and letting the administrator
> choose
> > > which one to use? (DBCP 1.4 as default)
> > >
> > > ________________________________
> > > From: Rafael Weingärtner <ra...@gmail.com>
> > > Sent: Tuesday, March 13, 2018 8:52:50 AM
> > > To: dev
> > > Subject: Re: [DISCUSS] CloudStack Connection Pools
> > >
> > > Spring data would be awesome. It is very flexible and has a very good
> > API.
> > > However, this would require commitment from our side to slowly migrate
> > > things to it.
> > >
> > > Regarding the connection pool management libraries; I would prefer
> either
> > > C3P0 or 2.* DBCP. The other two sound trendy, but I worry about this
> type
> > > of project in the long run. Both DBCP from Apache and C3P0 from
> Hibernate
> > > (RedHat) sound a more reasonable selection for me. They have been
> around
> > > for years, and have a solid community base already.
> > >
> > > On Mon, Mar 12, 2018 at 11:31 PM, Khosrow Moossavi <
> > kmoossavi@cloudops.com
> > > >
> > > wrote:
> > >
> > > > Hi Nicolas
> > > >
> > > > From my past experiences, I prefer 1) HikariCP 2) Tomcat Pool 3) C3P0
> > 4)
> > > > DBCP in that order. Although I don't have
> > > > any benchmark of my own to provide, and the ones you mentioned are
> > really
> > > > informative anyway.
> > > >
> > > > To me the broader subject is the _one_ who uses the pool, I mean if
> the
> > > > transactions are handled in a faster way and
> > > > released sooner and with shorter locks, generally speaking if it's
> more
> > > > efficient, I don't think from ACS point of view
> > > > there won't be much difference between the above mentioned options.
> > > >
> > > > On the same subject, it might be more interesting to use Spring Boot
> in
> > > > general and Spring Boot Data in particular
> > > > rather than only changing the CP functionality, and slowly
> > > refactor/retire
> > > > the DAO layer in favor of Spring Boot equivalent
> > > > implementation.
> > > >
> > > >
> > > > Khosrow Moossavi
> > > >
> > > > CloudOps
> > > >
> > > >
> > > >
> > > > On Mon, Mar 12, 2018 at 9:32 PM, Nicolas Vazquez <
> > > > Nicolas.Vazquez@shapeblue.com> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > >
> > > > > I would like to introduce a topic for discussion, regarding DB
> > > connection
> > > > > pools used in CloudStack, currently Apache Commons DBCP 1.4 (
> > > > > http://commons.apache.org/) is used. I've been investigating this
> > > topic
> > > > > as we are having complains of random issues on MySQL connection
> pool
> > on
> > > > > large environments. Please let me know if this topic has already
> been
> > > > > discussed before.
> > > > >
> > > > >
> > > > > First of all, DBCP 1.4 has been released on 2010 (
> > > > > https://commons.apache.org/proper/commons-dbcp/changes-report.html
> ),
> > > and
> > > > > no minor/patch version has been released since then. It seems to
> work
> > > in
> > > > > high performance with relatively low traffic and low load
> > applications.
> > > > > However, it is single threaded, and in order to be thread-safe, the
> > > > entire
> > > > > pool needs to be locked. It is also reported that an CPU and
> > concurrent
> > > > > threads increases, the performance gets affected. This is a serious
> > > issue
> > > > > on highly concurrent systems, such as CloudStack.
> > > > >
> > > > >
> > > > > I've been investigating some options to replace it:
> > > > > - The first option can be upgrading to version 2.x. Issues on
> > > performance
> > > > > and concurrency could be solved using this version.
> > > > > - Tomcat JDBC Connection Pool. Please check:
> > > https://tomcat.apache.org/
> > > > > tomcat-7.0-doc/jdbc-pool.html.
> > > > >
> > > > > - Other replacement options found: BoneCP, C3P0, HikariCP
> > > > >
> > > > >
> > > > > Given these options, I've been looking for benchmarks to compare
> them
> > > > (*).
> > > > > Looks like HikariCP (http://brettwooldridge.github.io/HikariCP/)
> > could
> > > > be
> > > > > the best replacement, improving performance and stability. Another
> > good
> > > > > replacement option could be Tomcat.
> > > > >
> > > > >
> > > > > I've been also examining the codebase, data source initialization
> is
> > > done
> > > > > on TransactionLegacy class under the cloud-framework-db project.
> > > > > Replacement work should be done on this class. Instead of pure
> > > > replacement,
> > > > > a global setting can be introduced to make the admins able to
> select
> > > > which
> > > > > connection pool to use.
> > > > >
> > > > > What do you think? Any possitive/negative feedback is welcome as
> well
> > > as
> > > > > new ideas. As mentioned before, I don't know if it has been
> discussed
> > > > > before, sorry in advance if it has.
> > > > >
> > > > > Kind regards,
> > > > > Nicolas
> > > > >
> > > > > (*) Links to benchmarks and comparissons:
> > > > > https://www.wix.engineering/single-post/how-many-threads-
> > > > > does-it-take-to-fill-a-pool
> > > > > https://www.wix.engineering/single-post/how-does-hikaricp-
> > > > > compare-to-other-connection-pools
> > > > > <https://www.wix.engineering/single-post/how-does-hikaricp-
> > > > > compare-to-other-connection-pools>https://beansroasted.
> > > > > wordpress.com/2017/07/29/connection-pool-analysis/
> > > > > https://beansroasted.wordpress.com/tag/connection-pool-comparison/
> > > > > <https://beansroasted.wordpress.com/tag/connection-
> pool-comparison/>
> > > > > https://github.com/brettwooldridge/HikariCP/wiki/%22My-benchmark-
> > > > > doesn't-show-a-difference.%22
> > > > > http://www.trustiv.co.uk/2014/06/battle-connection-pools
> > > > >
> > > > > Nicolas.Vazquez@shapeblue.com
> > > > > www.shapeblue.com<http://www.shapeblue.com>
> > > > > ,
> > > > > @shapeblue
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Rafael Weingärtner
> > >
> > > Nicolas.Vazquez@shapeblue.com
> > > www.shapeblue.com
> > > ,
> > > @shapeblue
> > >
> > >
> > >
> > >
> >
>
--
Rafael Weingärtner
Re: [DISCUSS] CloudStack Connection Pools
Posted by ilya musayev <il...@gmail.com>.
When everything works smooth - the end user does not need to know whats
under the hood.
However, when things begin to swing and your hands are tied to just one CP
- what do you do?
On Wed, Mar 14, 2018 at 10:50 AM, Khosrow Moossavi <km...@cloudops.com>
wrote:
> Why would we want to expose this choice to administrator of Cloudstack
> whose responsibility
> is to keep it running and not knowing about the inner-mechanic of how it
> works. right? It's not
> like that we're giving them a choice of which database to connect to.
>
> So on that note, I would say we need to agree on any of those CP libraries
> and implement, the
> same way we chose for example log4j or slf4j over one another, or any other
> _library_ we use.
>
> Khosrow Moossavi
>
> CloudOps
>
>
>
> On Wed, Mar 14, 2018 at 10:36 AM, Nicolas Vazquez <
> Nicolas.Vazquez@shapeblue.com> wrote:
>
> > Thanks Khosrow and Rafael. You both agree on Spring Data as the best
> > option, I see it would require a big effort and commitment to migrate to
> > it, therefore it can take some (long) time to achieve it.
> >
> > As a more viable option, would you agree on supporting different
> > connection pool management libraries and letting the administrator choose
> > which one to use? (DBCP 1.4 as default)
> >
> > ________________________________
> > From: Rafael Weingärtner <ra...@gmail.com>
> > Sent: Tuesday, March 13, 2018 8:52:50 AM
> > To: dev
> > Subject: Re: [DISCUSS] CloudStack Connection Pools
> >
> > Spring data would be awesome. It is very flexible and has a very good
> API.
> > However, this would require commitment from our side to slowly migrate
> > things to it.
> >
> > Regarding the connection pool management libraries; I would prefer either
> > C3P0 or 2.* DBCP. The other two sound trendy, but I worry about this type
> > of project in the long run. Both DBCP from Apache and C3P0 from Hibernate
> > (RedHat) sound a more reasonable selection for me. They have been around
> > for years, and have a solid community base already.
> >
> > On Mon, Mar 12, 2018 at 11:31 PM, Khosrow Moossavi <
> kmoossavi@cloudops.com
> > >
> > wrote:
> >
> > > Hi Nicolas
> > >
> > > From my past experiences, I prefer 1) HikariCP 2) Tomcat Pool 3) C3P0
> 4)
> > > DBCP in that order. Although I don't have
> > > any benchmark of my own to provide, and the ones you mentioned are
> really
> > > informative anyway.
> > >
> > > To me the broader subject is the _one_ who uses the pool, I mean if the
> > > transactions are handled in a faster way and
> > > released sooner and with shorter locks, generally speaking if it's more
> > > efficient, I don't think from ACS point of view
> > > there won't be much difference between the above mentioned options.
> > >
> > > On the same subject, it might be more interesting to use Spring Boot in
> > > general and Spring Boot Data in particular
> > > rather than only changing the CP functionality, and slowly
> > refactor/retire
> > > the DAO layer in favor of Spring Boot equivalent
> > > implementation.
> > >
> > >
> > > Khosrow Moossavi
> > >
> > > CloudOps
> > >
> > >
> > >
> > > On Mon, Mar 12, 2018 at 9:32 PM, Nicolas Vazquez <
> > > Nicolas.Vazquez@shapeblue.com> wrote:
> > >
> > > > Hi all,
> > > >
> > > >
> > > > I would like to introduce a topic for discussion, regarding DB
> > connection
> > > > pools used in CloudStack, currently Apache Commons DBCP 1.4 (
> > > > http://commons.apache.org/) is used. I've been investigating this
> > topic
> > > > as we are having complains of random issues on MySQL connection pool
> on
> > > > large environments. Please let me know if this topic has already been
> > > > discussed before.
> > > >
> > > >
> > > > First of all, DBCP 1.4 has been released on 2010 (
> > > > https://commons.apache.org/proper/commons-dbcp/changes-report.html),
> > and
> > > > no minor/patch version has been released since then. It seems to work
> > in
> > > > high performance with relatively low traffic and low load
> applications.
> > > > However, it is single threaded, and in order to be thread-safe, the
> > > entire
> > > > pool needs to be locked. It is also reported that an CPU and
> concurrent
> > > > threads increases, the performance gets affected. This is a serious
> > issue
> > > > on highly concurrent systems, such as CloudStack.
> > > >
> > > >
> > > > I've been investigating some options to replace it:
> > > > - The first option can be upgrading to version 2.x. Issues on
> > performance
> > > > and concurrency could be solved using this version.
> > > > - Tomcat JDBC Connection Pool. Please check:
> > https://tomcat.apache.org/
> > > > tomcat-7.0-doc/jdbc-pool.html.
> > > >
> > > > - Other replacement options found: BoneCP, C3P0, HikariCP
> > > >
> > > >
> > > > Given these options, I've been looking for benchmarks to compare them
> > > (*).
> > > > Looks like HikariCP (http://brettwooldridge.github.io/HikariCP/)
> could
> > > be
> > > > the best replacement, improving performance and stability. Another
> good
> > > > replacement option could be Tomcat.
> > > >
> > > >
> > > > I've been also examining the codebase, data source initialization is
> > done
> > > > on TransactionLegacy class under the cloud-framework-db project.
> > > > Replacement work should be done on this class. Instead of pure
> > > replacement,
> > > > a global setting can be introduced to make the admins able to select
> > > which
> > > > connection pool to use.
> > > >
> > > > What do you think? Any possitive/negative feedback is welcome as well
> > as
> > > > new ideas. As mentioned before, I don't know if it has been discussed
> > > > before, sorry in advance if it has.
> > > >
> > > > Kind regards,
> > > > Nicolas
> > > >
> > > > (*) Links to benchmarks and comparissons:
> > > > https://www.wix.engineering/single-post/how-many-threads-
> > > > does-it-take-to-fill-a-pool
> > > > https://www.wix.engineering/single-post/how-does-hikaricp-
> > > > compare-to-other-connection-pools
> > > > <https://www.wix.engineering/single-post/how-does-hikaricp-
> > > > compare-to-other-connection-pools>https://beansroasted.
> > > > wordpress.com/2017/07/29/connection-pool-analysis/
> > > > https://beansroasted.wordpress.com/tag/connection-pool-comparison/
> > > > <https://beansroasted.wordpress.com/tag/connection-pool-comparison/>
> > > > https://github.com/brettwooldridge/HikariCP/wiki/%22My-benchmark-
> > > > doesn't-show-a-difference.%22
> > > > http://www.trustiv.co.uk/2014/06/battle-connection-pools
> > > >
> > > > Nicolas.Vazquez@shapeblue.com
> > > > www.shapeblue.com<http://www.shapeblue.com>
> > > > ,
> > > > @shapeblue
> > > >
> > > >
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Rafael Weingärtner
> >
> > Nicolas.Vazquez@shapeblue.com
> > www.shapeblue.com
> > ,
> > @shapeblue
> >
> >
> >
> >
>
Re: [DISCUSS] CloudStack Connection Pools
Posted by Khosrow Moossavi <km...@cloudops.com>.
Why would we want to expose this choice to administrator of Cloudstack
whose responsibility
is to keep it running and not knowing about the inner-mechanic of how it
works. right? It's not
like that we're giving them a choice of which database to connect to.
So on that note, I would say we need to agree on any of those CP libraries
and implement, the
same way we chose for example log4j or slf4j over one another, or any other
_library_ we use.
Khosrow Moossavi
CloudOps
On Wed, Mar 14, 2018 at 10:36 AM, Nicolas Vazquez <
Nicolas.Vazquez@shapeblue.com> wrote:
> Thanks Khosrow and Rafael. You both agree on Spring Data as the best
> option, I see it would require a big effort and commitment to migrate to
> it, therefore it can take some (long) time to achieve it.
>
> As a more viable option, would you agree on supporting different
> connection pool management libraries and letting the administrator choose
> which one to use? (DBCP 1.4 as default)
>
> ________________________________
> From: Rafael Weingärtner <ra...@gmail.com>
> Sent: Tuesday, March 13, 2018 8:52:50 AM
> To: dev
> Subject: Re: [DISCUSS] CloudStack Connection Pools
>
> Spring data would be awesome. It is very flexible and has a very good API.
> However, this would require commitment from our side to slowly migrate
> things to it.
>
> Regarding the connection pool management libraries; I would prefer either
> C3P0 or 2.* DBCP. The other two sound trendy, but I worry about this type
> of project in the long run. Both DBCP from Apache and C3P0 from Hibernate
> (RedHat) sound a more reasonable selection for me. They have been around
> for years, and have a solid community base already.
>
> On Mon, Mar 12, 2018 at 11:31 PM, Khosrow Moossavi <kmoossavi@cloudops.com
> >
> wrote:
>
> > Hi Nicolas
> >
> > From my past experiences, I prefer 1) HikariCP 2) Tomcat Pool 3) C3P0 4)
> > DBCP in that order. Although I don't have
> > any benchmark of my own to provide, and the ones you mentioned are really
> > informative anyway.
> >
> > To me the broader subject is the _one_ who uses the pool, I mean if the
> > transactions are handled in a faster way and
> > released sooner and with shorter locks, generally speaking if it's more
> > efficient, I don't think from ACS point of view
> > there won't be much difference between the above mentioned options.
> >
> > On the same subject, it might be more interesting to use Spring Boot in
> > general and Spring Boot Data in particular
> > rather than only changing the CP functionality, and slowly
> refactor/retire
> > the DAO layer in favor of Spring Boot equivalent
> > implementation.
> >
> >
> > Khosrow Moossavi
> >
> > CloudOps
> >
> >
> >
> > On Mon, Mar 12, 2018 at 9:32 PM, Nicolas Vazquez <
> > Nicolas.Vazquez@shapeblue.com> wrote:
> >
> > > Hi all,
> > >
> > >
> > > I would like to introduce a topic for discussion, regarding DB
> connection
> > > pools used in CloudStack, currently Apache Commons DBCP 1.4 (
> > > http://commons.apache.org/) is used. I've been investigating this
> topic
> > > as we are having complains of random issues on MySQL connection pool on
> > > large environments. Please let me know if this topic has already been
> > > discussed before.
> > >
> > >
> > > First of all, DBCP 1.4 has been released on 2010 (
> > > https://commons.apache.org/proper/commons-dbcp/changes-report.html),
> and
> > > no minor/patch version has been released since then. It seems to work
> in
> > > high performance with relatively low traffic and low load applications.
> > > However, it is single threaded, and in order to be thread-safe, the
> > entire
> > > pool needs to be locked. It is also reported that an CPU and concurrent
> > > threads increases, the performance gets affected. This is a serious
> issue
> > > on highly concurrent systems, such as CloudStack.
> > >
> > >
> > > I've been investigating some options to replace it:
> > > - The first option can be upgrading to version 2.x. Issues on
> performance
> > > and concurrency could be solved using this version.
> > > - Tomcat JDBC Connection Pool. Please check:
> https://tomcat.apache.org/
> > > tomcat-7.0-doc/jdbc-pool.html.
> > >
> > > - Other replacement options found: BoneCP, C3P0, HikariCP
> > >
> > >
> > > Given these options, I've been looking for benchmarks to compare them
> > (*).
> > > Looks like HikariCP (http://brettwooldridge.github.io/HikariCP/) could
> > be
> > > the best replacement, improving performance and stability. Another good
> > > replacement option could be Tomcat.
> > >
> > >
> > > I've been also examining the codebase, data source initialization is
> done
> > > on TransactionLegacy class under the cloud-framework-db project.
> > > Replacement work should be done on this class. Instead of pure
> > replacement,
> > > a global setting can be introduced to make the admins able to select
> > which
> > > connection pool to use.
> > >
> > > What do you think? Any possitive/negative feedback is welcome as well
> as
> > > new ideas. As mentioned before, I don't know if it has been discussed
> > > before, sorry in advance if it has.
> > >
> > > Kind regards,
> > > Nicolas
> > >
> > > (*) Links to benchmarks and comparissons:
> > > https://www.wix.engineering/single-post/how-many-threads-
> > > does-it-take-to-fill-a-pool
> > > https://www.wix.engineering/single-post/how-does-hikaricp-
> > > compare-to-other-connection-pools
> > > <https://www.wix.engineering/single-post/how-does-hikaricp-
> > > compare-to-other-connection-pools>https://beansroasted.
> > > wordpress.com/2017/07/29/connection-pool-analysis/
> > > https://beansroasted.wordpress.com/tag/connection-pool-comparison/
> > > <https://beansroasted.wordpress.com/tag/connection-pool-comparison/>
> > > https://github.com/brettwooldridge/HikariCP/wiki/%22My-benchmark-
> > > doesn't-show-a-difference.%22
> > > http://www.trustiv.co.uk/2014/06/battle-connection-pools
> > >
> > > Nicolas.Vazquez@shapeblue.com
> > > www.shapeblue.com<http://www.shapeblue.com>
> > > ,
> > > @shapeblue
> > >
> > >
> > >
> > >
> >
>
>
>
> --
> Rafael Weingärtner
>
> Nicolas.Vazquez@shapeblue.com
> www.shapeblue.com
> ,
> @shapeblue
>
>
>
>
Re: [DISCUSS] CloudStack Connection Pools
Posted by Nicolas Vazquez <Ni...@shapeblue.com>.
Thanks Khosrow and Rafael. You both agree on Spring Data as the best option, I see it would require a big effort and commitment to migrate to it, therefore it can take some (long) time to achieve it.
As a more viable option, would you agree on supporting different connection pool management libraries and letting the administrator choose which one to use? (DBCP 1.4 as default)
________________________________
From: Rafael Weingärtner <ra...@gmail.com>
Sent: Tuesday, March 13, 2018 8:52:50 AM
To: dev
Subject: Re: [DISCUSS] CloudStack Connection Pools
Spring data would be awesome. It is very flexible and has a very good API.
However, this would require commitment from our side to slowly migrate
things to it.
Regarding the connection pool management libraries; I would prefer either
C3P0 or 2.* DBCP. The other two sound trendy, but I worry about this type
of project in the long run. Both DBCP from Apache and C3P0 from Hibernate
(RedHat) sound a more reasonable selection for me. They have been around
for years, and have a solid community base already.
On Mon, Mar 12, 2018 at 11:31 PM, Khosrow Moossavi <km...@cloudops.com>
wrote:
> Hi Nicolas
>
> From my past experiences, I prefer 1) HikariCP 2) Tomcat Pool 3) C3P0 4)
> DBCP in that order. Although I don't have
> any benchmark of my own to provide, and the ones you mentioned are really
> informative anyway.
>
> To me the broader subject is the _one_ who uses the pool, I mean if the
> transactions are handled in a faster way and
> released sooner and with shorter locks, generally speaking if it's more
> efficient, I don't think from ACS point of view
> there won't be much difference between the above mentioned options.
>
> On the same subject, it might be more interesting to use Spring Boot in
> general and Spring Boot Data in particular
> rather than only changing the CP functionality, and slowly refactor/retire
> the DAO layer in favor of Spring Boot equivalent
> implementation.
>
>
> Khosrow Moossavi
>
> CloudOps
>
>
>
> On Mon, Mar 12, 2018 at 9:32 PM, Nicolas Vazquez <
> Nicolas.Vazquez@shapeblue.com> wrote:
>
> > Hi all,
> >
> >
> > I would like to introduce a topic for discussion, regarding DB connection
> > pools used in CloudStack, currently Apache Commons DBCP 1.4 (
> > http://commons.apache.org/) is used. I've been investigating this topic
> > as we are having complains of random issues on MySQL connection pool on
> > large environments. Please let me know if this topic has already been
> > discussed before.
> >
> >
> > First of all, DBCP 1.4 has been released on 2010 (
> > https://commons.apache.org/proper/commons-dbcp/changes-report.html), and
> > no minor/patch version has been released since then. It seems to work in
> > high performance with relatively low traffic and low load applications.
> > However, it is single threaded, and in order to be thread-safe, the
> entire
> > pool needs to be locked. It is also reported that an CPU and concurrent
> > threads increases, the performance gets affected. This is a serious issue
> > on highly concurrent systems, such as CloudStack.
> >
> >
> > I've been investigating some options to replace it:
> > - The first option can be upgrading to version 2.x. Issues on performance
> > and concurrency could be solved using this version.
> > - Tomcat JDBC Connection Pool. Please check: https://tomcat.apache.org/
> > tomcat-7.0-doc/jdbc-pool.html.
> >
> > - Other replacement options found: BoneCP, C3P0, HikariCP
> >
> >
> > Given these options, I've been looking for benchmarks to compare them
> (*).
> > Looks like HikariCP (http://brettwooldridge.github.io/HikariCP/) could
> be
> > the best replacement, improving performance and stability. Another good
> > replacement option could be Tomcat.
> >
> >
> > I've been also examining the codebase, data source initialization is done
> > on TransactionLegacy class under the cloud-framework-db project.
> > Replacement work should be done on this class. Instead of pure
> replacement,
> > a global setting can be introduced to make the admins able to select
> which
> > connection pool to use.
> >
> > What do you think? Any possitive/negative feedback is welcome as well as
> > new ideas. As mentioned before, I don't know if it has been discussed
> > before, sorry in advance if it has.
> >
> > Kind regards,
> > Nicolas
> >
> > (*) Links to benchmarks and comparissons:
> > https://www.wix.engineering/single-post/how-many-threads-
> > does-it-take-to-fill-a-pool
> > https://www.wix.engineering/single-post/how-does-hikaricp-
> > compare-to-other-connection-pools
> > <https://www.wix.engineering/single-post/how-does-hikaricp-
> > compare-to-other-connection-pools>https://beansroasted.
> > wordpress.com/2017/07/29/connection-pool-analysis/
> > https://beansroasted.wordpress.com/tag/connection-pool-comparison/
> > <https://beansroasted.wordpress.com/tag/connection-pool-comparison/>
> > https://github.com/brettwooldridge/HikariCP/wiki/%22My-benchmark-
> > doesn't-show-a-difference.%22
> > http://www.trustiv.co.uk/2014/06/battle-connection-pools
> >
> > Nicolas.Vazquez@shapeblue.com
> > www.shapeblue.com<http://www.shapeblue.com>
> > ,
> > @shapeblue
> >
> >
> >
> >
>
--
Rafael Weingärtner
Nicolas.Vazquez@shapeblue.com
www.shapeblue.com
,
@shapeblue
Re: [DISCUSS] CloudStack Connection Pools
Posted by Rafael Weingärtner <ra...@gmail.com>.
Spring data would be awesome. It is very flexible and has a very good API.
However, this would require commitment from our side to slowly migrate
things to it.
Regarding the connection pool management libraries; I would prefer either
C3P0 or 2.* DBCP. The other two sound trendy, but I worry about this type
of project in the long run. Both DBCP from Apache and C3P0 from Hibernate
(RedHat) sound a more reasonable selection for me. They have been around
for years, and have a solid community base already.
On Mon, Mar 12, 2018 at 11:31 PM, Khosrow Moossavi <km...@cloudops.com>
wrote:
> Hi Nicolas
>
> From my past experiences, I prefer 1) HikariCP 2) Tomcat Pool 3) C3P0 4)
> DBCP in that order. Although I don't have
> any benchmark of my own to provide, and the ones you mentioned are really
> informative anyway.
>
> To me the broader subject is the _one_ who uses the pool, I mean if the
> transactions are handled in a faster way and
> released sooner and with shorter locks, generally speaking if it's more
> efficient, I don't think from ACS point of view
> there won't be much difference between the above mentioned options.
>
> On the same subject, it might be more interesting to use Spring Boot in
> general and Spring Boot Data in particular
> rather than only changing the CP functionality, and slowly refactor/retire
> the DAO layer in favor of Spring Boot equivalent
> implementation.
>
>
> Khosrow Moossavi
>
> CloudOps
>
>
>
> On Mon, Mar 12, 2018 at 9:32 PM, Nicolas Vazquez <
> Nicolas.Vazquez@shapeblue.com> wrote:
>
> > Hi all,
> >
> >
> > I would like to introduce a topic for discussion, regarding DB connection
> > pools used in CloudStack, currently Apache Commons DBCP 1.4 (
> > http://commons.apache.org/) is used. I've been investigating this topic
> > as we are having complains of random issues on MySQL connection pool on
> > large environments. Please let me know if this topic has already been
> > discussed before.
> >
> >
> > First of all, DBCP 1.4 has been released on 2010 (
> > https://commons.apache.org/proper/commons-dbcp/changes-report.html), and
> > no minor/patch version has been released since then. It seems to work in
> > high performance with relatively low traffic and low load applications.
> > However, it is single threaded, and in order to be thread-safe, the
> entire
> > pool needs to be locked. It is also reported that an CPU and concurrent
> > threads increases, the performance gets affected. This is a serious issue
> > on highly concurrent systems, such as CloudStack.
> >
> >
> > I've been investigating some options to replace it:
> > - The first option can be upgrading to version 2.x. Issues on performance
> > and concurrency could be solved using this version.
> > - Tomcat JDBC Connection Pool. Please check: https://tomcat.apache.org/
> > tomcat-7.0-doc/jdbc-pool.html.
> >
> > - Other replacement options found: BoneCP, C3P0, HikariCP
> >
> >
> > Given these options, I've been looking for benchmarks to compare them
> (*).
> > Looks like HikariCP (http://brettwooldridge.github.io/HikariCP/) could
> be
> > the best replacement, improving performance and stability. Another good
> > replacement option could be Tomcat.
> >
> >
> > I've been also examining the codebase, data source initialization is done
> > on TransactionLegacy class under the cloud-framework-db project.
> > Replacement work should be done on this class. Instead of pure
> replacement,
> > a global setting can be introduced to make the admins able to select
> which
> > connection pool to use.
> >
> > What do you think? Any possitive/negative feedback is welcome as well as
> > new ideas. As mentioned before, I don't know if it has been discussed
> > before, sorry in advance if it has.
> >
> > Kind regards,
> > Nicolas
> >
> > (*) Links to benchmarks and comparissons:
> > https://www.wix.engineering/single-post/how-many-threads-
> > does-it-take-to-fill-a-pool
> > https://www.wix.engineering/single-post/how-does-hikaricp-
> > compare-to-other-connection-pools
> > <https://www.wix.engineering/single-post/how-does-hikaricp-
> > compare-to-other-connection-pools>https://beansroasted.
> > wordpress.com/2017/07/29/connection-pool-analysis/
> > https://beansroasted.wordpress.com/tag/connection-pool-comparison/
> > <https://beansroasted.wordpress.com/tag/connection-pool-comparison/>
> > https://github.com/brettwooldridge/HikariCP/wiki/%22My-benchmark-
> > doesn't-show-a-difference.%22
> > http://www.trustiv.co.uk/2014/06/battle-connection-pools
> >
> > Nicolas.Vazquez@shapeblue.com
> > www.shapeblue.com
> > ,
> > @shapeblue
> >
> >
> >
> >
>
--
Rafael Weingärtner
Re: [DISCUSS] CloudStack Connection Pools
Posted by Khosrow Moossavi <km...@cloudops.com>.
Hi Nicolas
From my past experiences, I prefer 1) HikariCP 2) Tomcat Pool 3) C3P0 4)
DBCP in that order. Although I don't have
any benchmark of my own to provide, and the ones you mentioned are really
informative anyway.
To me the broader subject is the _one_ who uses the pool, I mean if the
transactions are handled in a faster way and
released sooner and with shorter locks, generally speaking if it's more
efficient, I don't think from ACS point of view
there won't be much difference between the above mentioned options.
On the same subject, it might be more interesting to use Spring Boot in
general and Spring Boot Data in particular
rather than only changing the CP functionality, and slowly refactor/retire
the DAO layer in favor of Spring Boot equivalent
implementation.
Khosrow Moossavi
CloudOps
On Mon, Mar 12, 2018 at 9:32 PM, Nicolas Vazquez <
Nicolas.Vazquez@shapeblue.com> wrote:
> Hi all,
>
>
> I would like to introduce a topic for discussion, regarding DB connection
> pools used in CloudStack, currently Apache Commons DBCP 1.4 (
> http://commons.apache.org/) is used. I've been investigating this topic
> as we are having complains of random issues on MySQL connection pool on
> large environments. Please let me know if this topic has already been
> discussed before.
>
>
> First of all, DBCP 1.4 has been released on 2010 (
> https://commons.apache.org/proper/commons-dbcp/changes-report.html), and
> no minor/patch version has been released since then. It seems to work in
> high performance with relatively low traffic and low load applications.
> However, it is single threaded, and in order to be thread-safe, the entire
> pool needs to be locked. It is also reported that an CPU and concurrent
> threads increases, the performance gets affected. This is a serious issue
> on highly concurrent systems, such as CloudStack.
>
>
> I've been investigating some options to replace it:
> - The first option can be upgrading to version 2.x. Issues on performance
> and concurrency could be solved using this version.
> - Tomcat JDBC Connection Pool. Please check: https://tomcat.apache.org/
> tomcat-7.0-doc/jdbc-pool.html.
>
> - Other replacement options found: BoneCP, C3P0, HikariCP
>
>
> Given these options, I've been looking for benchmarks to compare them (*).
> Looks like HikariCP (http://brettwooldridge.github.io/HikariCP/) could be
> the best replacement, improving performance and stability. Another good
> replacement option could be Tomcat.
>
>
> I've been also examining the codebase, data source initialization is done
> on TransactionLegacy class under the cloud-framework-db project.
> Replacement work should be done on this class. Instead of pure replacement,
> a global setting can be introduced to make the admins able to select which
> connection pool to use.
>
> What do you think? Any possitive/negative feedback is welcome as well as
> new ideas. As mentioned before, I don't know if it has been discussed
> before, sorry in advance if it has.
>
> Kind regards,
> Nicolas
>
> (*) Links to benchmarks and comparissons:
> https://www.wix.engineering/single-post/how-many-threads-
> does-it-take-to-fill-a-pool
> https://www.wix.engineering/single-post/how-does-hikaricp-
> compare-to-other-connection-pools
> <https://www.wix.engineering/single-post/how-does-hikaricp-
> compare-to-other-connection-pools>https://beansroasted.
> wordpress.com/2017/07/29/connection-pool-analysis/
> https://beansroasted.wordpress.com/tag/connection-pool-comparison/
> <https://beansroasted.wordpress.com/tag/connection-pool-comparison/>
> https://github.com/brettwooldridge/HikariCP/wiki/%22My-benchmark-
> doesn't-show-a-difference.%22
> http://www.trustiv.co.uk/2014/06/battle-connection-pools
>
> Nicolas.Vazquez@shapeblue.com
> www.shapeblue.com
> ,
> @shapeblue
>
>
>
>