You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Alex Chew <al...@msn.com> on 2016/06/08 11:06:06 UTC
Too many connections error while using storm-JDBC
Hi all,
I got Too Many Connections error while running my topology in test environment. I am checking below codes,
//prepare JDBC configuration
Map jdbcConfigMap = Maps.newHashMap();
jdbcConfigMap.put("dataSourceClassName", prop.getProperty("mysql.dataSource.className"));//com.mysql.jdbc.jdbc2.optional.MysqlDataSource
jdbcConfigMap.put("dataSource.url", prop.getProperty("mysql.url"));//jdbc:mysql://localhost/test
jdbcConfigMap.put("dataSource.user", prop.getProperty("mysql.user"));//root
jdbcConfigMap.put("dataSource.password", prop.getProperty("mysql.password"));//password
config.put(JDBC_CONF, jdbcConfigMap);
this.connectionProvider = new HikariCPConnectionProvider(jdbcConfigMap);
This will create a new HikariCPConnectionProvider each Spout/Bolt task. By default each connectionProvider will initial 10 connections. Totally there have 48 tasks. This will create 480 connections while MySQL max-conenctions is 400. I am going to change this, but I don't know how to force all tasks running on my 4 supervisor nodes to use the same dataSource?
Thanks in advance.
Alex
回复: Too many connections error while using storm-JDBC
Posted by Alex Chew <al...@msn.com>.
Thanks a lot. because topologies are running in seperated JVMs I'll let them use different datasources. I think I can change DB pool configuration to reduce connections for a single dataProvider. I can also let those tasks running in the same JVM share one datasource.
Best Regards,
Alex
-------- 原始邮件 --------
主题：Re: Too many connections error while using storm-JDBC
发件人：Alberto S鉶 Marcos
收件人：user@storm.apache.org
抄送：
I dont think is possible to use the same datasource across topologies/bolts
how would u share such a resource across multiple jvms and threads?
On Wed, Jun 8, 2016 at 2:49 PM, Jacob Johansen <jo...@gmail.com>> wrote:
In the HikariCPConnectionProvider you need to set the connection pool
smaller or in your MySQL DB configure, increase the number of
available connections.
Jacob Johansen
On Wed, Jun 8, 2016 at 6:06 AM, Alex Chew <al...@msn.com>> wrote:
> Hi all,
>
>
> I got Too Many Connections error while running my topology in test
> environment. I am checking below codes,
>
>
> //prepare JDBC configuration
> Map jdbcConfigMap = Maps.newHashMap();
> jdbcConfigMap.put("dataSourceClassName",
> prop.getProperty("mysql.dataSource.className"));//com.mysql.jdbc.jdbc2.optional.MysqlDataSource
> jdbcConfigMap.put("dataSource.url",
> prop.getProperty("mysql.url"));//jdbc:mysql://localhost/test
> jdbcConfigMap.put("dataSource.user",
> prop.getProperty("mysql.user"));//root
> jdbcConfigMap.put("dataSource.password",
> prop.getProperty("mysql.password"));//password
> config.put(JDBC_CONF, jdbcConfigMap);
> this.connectionProvider = new
> HikariCPConnectionProvider(jdbcConfigMap);
>
> This will create a new HikariCPConnectionProvider each Spout/Bolt task. By
> default each connectionProvider will initial 10 connections. Totally there
> have 48 tasks. This will create 480 connections while MySQL max-conenctions
> is 400. I am going to change this, but I don't know how to force all tasks
> running on my 4 supervisor nodes to use the same dataSource?
>
>
> Thanks in advance.
>
> Alex
>
>
>
Re: Too many connections error while using storm-JDBC
Posted by Alberto São Marcos <al...@gmail.com>.
I dont think is possible to use the same datasource across topologies/bolts
how would u share such a resource across multiple jvms and threads?
On Wed, Jun 8, 2016 at 2:49 PM, Jacob Johansen <jo...@gmail.com>
wrote:
> In the HikariCPConnectionProvider you need to set the connection pool
> smaller or in your MySQL DB configure, increase the number of
> available connections.
> Jacob Johansen
>
>
> On Wed, Jun 8, 2016 at 6:06 AM, Alex Chew <al...@msn.com> wrote:
> > Hi all,
> >
> >
> > I got Too Many Connections error while running my topology in test
> > environment. I am checking below codes,
> >
> >
> > //prepare JDBC configuration
> > Map jdbcConfigMap = Maps.newHashMap();
> > jdbcConfigMap.put("dataSourceClassName",
> >
> prop.getProperty("mysql.dataSource.className"));//com.mysql.jdbc.jdbc2.optional.MysqlDataSource
> > jdbcConfigMap.put("dataSource.url",
> > prop.getProperty("mysql.url"));//jdbc:mysql://localhost/test
> > jdbcConfigMap.put("dataSource.user",
> > prop.getProperty("mysql.user"));//root
> > jdbcConfigMap.put("dataSource.password",
> > prop.getProperty("mysql.password"));//password
> > config.put(JDBC_CONF, jdbcConfigMap);
> > this.connectionProvider = new
> > HikariCPConnectionProvider(jdbcConfigMap);
> >
> > This will create a new HikariCPConnectionProvider each Spout/Bolt task.
> By
> > default each connectionProvider will initial 10 connections. Totally
> there
> > have 48 tasks. This will create 480 connections while MySQL
> max-conenctions
> > is 400. I am going to change this, but I don't know how to force all
> tasks
> > running on my 4 supervisor nodes to use the same dataSource?
> >
> >
> > Thanks in advance.
> >
> > Alex
> >
> >
> >
>
Re: Too many connections error while using storm-JDBC
Posted by Jacob Johansen <jo...@gmail.com>.
In the HikariCPConnectionProvider you need to set the connection pool
smaller or in your MySQL DB configure, increase the number of
available connections.
Jacob Johansen
On Wed, Jun 8, 2016 at 6:06 AM, Alex Chew <al...@msn.com> wrote:
> Hi all,
>
>
> I got Too Many Connections error while running my topology in test
> environment. I am checking below codes,
>
>
> //prepare JDBC configuration
> Map jdbcConfigMap = Maps.newHashMap();
> jdbcConfigMap.put("dataSourceClassName",
> prop.getProperty("mysql.dataSource.className"));//com.mysql.jdbc.jdbc2.optional.MysqlDataSource
> jdbcConfigMap.put("dataSource.url",
> prop.getProperty("mysql.url"));//jdbc:mysql://localhost/test
> jdbcConfigMap.put("dataSource.user",
> prop.getProperty("mysql.user"));//root
> jdbcConfigMap.put("dataSource.password",
> prop.getProperty("mysql.password"));//password
> config.put(JDBC_CONF, jdbcConfigMap);
> this.connectionProvider = new
> HikariCPConnectionProvider(jdbcConfigMap);
>
> This will create a new HikariCPConnectionProvider each Spout/Bolt task. By
> default each connectionProvider will initial 10 connections. Totally there
> have 48 tasks. This will create 480 connections while MySQL max-conenctions
> is 400. I am going to change this, but I don't know how to force all tasks
> running on my 4 supervisor nodes to use the same dataSource?
>
>
> Thanks in advance.
>
> Alex
>
>
>