You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Sai Dilip Reddy Kiralam <dk...@aadhya-analytics.com> on 2016/05/03 12:50:27 UTC
Re: Storm topology using all the Max connections of db
I'm using storm jdbc for connecting to db.how can I use pooling on codes on
topology.please share information on connection pooling used on storm
topologies.
*Best regards,*
*K.Sai Dilip Reddy.*
On Thu, Apr 28, 2016 at 12:28 PM, Sai Dilip Reddy Kiralam <
dkiralam@aadhya-analytics.com> wrote:
> Hi Spico,
>
> I use 9 bolts with parallelism of 1 and with 1 task for each bolt(default)
> and I'm not using any ConnectionPool for connecting to postgres.Just using
> jdbc classe examples here is source example
> http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-jdbc.html.
> I will check by using the ConnectionPool.
>
> Thank you.
>
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>
> On Thu, Apr 28, 2016 at 12:19 PM, Spico Florin <sp...@gmail.com>
> wrote:
>
>> Hello!
>> How many tasks do you have for inserting the data to your database?
>> Are you using ConnectionPool for connecting to Postgres? If your number of
>> task superseeds the number of max connections provided in connection pool
>> then your have a problem.
>> Please also check the number of max connections that your db accepts.
>> I hope that these help.
>> Regards,
>> Florin
>>
>> On Thu, Apr 28, 2016 at 7:09 AM, Sai Dilip Reddy Kiralam <
>> dkiralam@aadhya-analytics.com> wrote:
>>
>>> Hi,
>>>
>>> I have topology that make connection with postgresdb and insert the
>>> fields into tables of a test db.my topology is working fine but when I
>>> submit the topology it is establishing all the connections of db.but I
>>> don’t know why it is taking all the max connections.
>>>
>>>
>>> below attached the pics of pg_stat_activity.
>>>
>>>
>>> *Best regards,*
>>>
>>> *K.Sai Dilip Reddy.*
>>>
>>
>>
>
Re: Storm topology using all the Max connections of db
Posted by Spico Florin <sp...@gmail.com>.
Hi!
Well is seems that you have to contact the team that created the jdbc
storm project. It seems that the issue is deeper than my knowledge.
I hope that my advises helped you. If you have a solution, please share it
here that the others can benefit from your experience.
Good luck!
Regards,
Florin
On Thu, May 5, 2016 at 1:57 PM, Sai Dilip Reddy Kiralam <
dkiralam@aadhya-analytics.com> wrote:
> Hi,
>
> my supervisor logs shows the following error :
>
> 2016-05-05T15:45:15.365+0530 b.s.d.executor [INFO] Prepared bolt
> __acker:(67)
> 2016-05-05T15:45:15.369+0530 b.s.d.executor [INFO] Prepared bolt
> __system:(-1)
> 2016-05-05T15:45:18.481+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-0 is starting.
> 2016-05-05T15:45:18.486+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-1 is starting.
> 2016-05-05T15:45:18.486+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-3 is starting.
> 2016-05-05T15:45:18.488+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-2 is starting.
> 2016-05-05T15:45:18.488+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-3 is starting.
> 2016-05-05T15:45:18.489+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-4 is starting.
> 2016-05-05T15:45:18.489+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-5 is starting.
> 2016-05-05T15:45:18.490+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-6 is starting.
> 2016-05-05T15:45:18.490+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-7 is starting.
> 2016-05-05T15:45:18.490+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-8 is starting.
> 2016-05-05T15:45:18.490+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-9 is starting.
> 2016-05-05T15:45:18.491+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-10 is starting.
> 2016-05-05T15:45:18.491+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-11 is starting.
> 2016-05-05T15:45:18.492+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-12 is starting.
> 2016-05-05T15:45:18.493+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-13 is starting.
> 2016-05-05T15:45:18.493+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
> HikariPool-4 is starting.
> 2016-05-05T15:45:19.310+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.311+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.312+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.312+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.314+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.314+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.314+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.313+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
> 2016-05-05T15:45:19.313+0530 c.z.h.p.HikariPool [ERROR] JDBC4
> Connection.isValid() method not supported, connection test query must be
> configured
>
> do postgres is using all the connections because this error ?
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>
> On Thu, May 5, 2016 at 10:57 AM, Sai Dilip Reddy Kiralam <
> dkiralam@aadhya-analytics.com> wrote:
>
>> Hi,
>>
>> I'm closing up and opening the connections after execution of insert
>> query by my bolt.Do I need to increase the max connections db to a level of
>> 500 to 1000? Here in log file of supervisor it tells me the connection
>> made by bolt is closed and again taking new connection.I think after
>> closing it is going into ideal and not used for new connections.
>>
>> 2016-05-05T10:44:37.761+0530 STDIO [INFO] COMMAND TO CLOSE CONNECTION IS
>> DONE
>> 2016-05-05T10:44:37.761+0530 STDIO [INFO] DB connected....
>>
>> if I'm right what is the solution for those ideal connections?
>>
>>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>> On Thu, May 5, 2016 at 1:01 AM, Spico Florin <sp...@gmail.com>
>> wrote:
>>
>>> hi!
>>> you have 9 bolts with 50 max db connections. so for each bolt you get a
>>> conection pool. try to decresae this number for example to 5 and check if
>>> your performance if fine with your db
>>> regards,
>>> florin
>>>
>>> On Wednesday, May 4, 2016, Sai Dilip Reddy Kiralam <
>>> dkiralam@aadhya-analytics.com> wrote:
>>>
>>>> sorry ! my db is not yet started, so it given me the error ! but when I
>>>> gave that statement and run the topology then it is using more connections
>>>> than the specified number.
>>>>
>>>>
>>>>
>>>> *Best regards,*
>>>>
>>>> *K.Sai Dilip Reddy.*
>>>>
>>>> On Wed, May 4, 2016 at 11:22 AM, Sai Dilip Reddy Kiralam <
>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I just added
>>>>>
>>>>> a statement
>>>>> * // config.setMaximumPoolSize(50);*
>>>>> public synchronized void prepare() {
>>>>> if(dataSource == null) {
>>>>> Properties properties = new Properties();
>>>>> properties.putAll(configMap);
>>>>> HikariConfig config = new HikariConfig(properties);
>>>>> * config.setMaximumPoolSize(50);*
>>>>> this.dataSource = new HikariDataSource(config);
>>>>> this.dataSource.setAutoCommit(false);
>>>>> }
>>>>> }
>>>>>
>>>>>
>>>>> and I'm getting the following error
>>>>>
>>>>> java.lang.RuntimeException: Fail-fast during pool initialization at
>>>>> com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:475) at
>>>>> com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:159) at
>>>>> com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112) at
>>>>> com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:78) at
>>>>> com.aail.config.MConnectionProvider.prepare(MConnectionProvider.java:27) at
>>>>> com.aail.storm.bolts.AbstractJdbcBolt.prepare(AbstractJdbcBolt.java:34) at
>>>>> com.aail.storm.bolts.Inserts.prepare(Inserts.java:290) at
>>>>> backtype.storm.daemon.executor$fn__6647$fn__6659.invoke(executor.clj:692)
>>>>> at backtype.storm.util$async_loop$fn__459.invoke(util.clj:461) at
>>>>> clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)
>>>>> Caused by: org.postgresql.util.PSQLException: Connection attempt timed out.
>>>>> at org.postgresql.Driver$ConnectThread.getResult(Driver.java:372) at
>>>>> org.postgresql.Driver.connect(Driver.java:284) at
>>>>> java.sql.DriverManager.getConnection(DriverManager.java:664) at
>>>>> java.sql.DriverManager.getConnection(DriverManager.java:247) at
>>>>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:99)
>>>>> at
>>>>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:82)
>>>>> at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:398) at
>>>>> com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:474) ... 10 more
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Best regards,*
>>>>>
>>>>> *K.Sai Dilip Reddy.*
>>>>>
>>>>> On Wed, May 4, 2016 at 9:53 AM, Sai Dilip Reddy Kiralam <
>>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Thank you Spico Florin.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Best regards,*
>>>>>>
>>>>>> *K.Sai Dilip Reddy.*
>>>>>>
>>>>>> On Wed, May 4, 2016 at 1:18 AM, Spico Florin <sp...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> hi!
>>>>>>> please have a look
>>>>>>> https://github.com/apache/storm/tree/master/external/storm-jdbc
>>>>>>> where you hve to implemnt thw interface connectionprovider interface. you
>>>>>>> have to dins a third paty librry that provides conection pooloing for
>>>>>>> postgresql and use that library to implement the interface connection
>>>>>>> provider. i hope that other community member will give you some other ideas.
>>>>>>> i hope that these help.
>>>>>>> regards,
>>>>>>> florin
>>>>>>>
>>>>>>> On Tuesday, May 3, 2016, Sai Dilip Reddy Kiralam <
>>>>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>>>>
>>>>>>>> to implemI'm using storm jdbc for connecting to db.how can I use
>>>>>>>> pooling on codes on topology.please share information on connection pooling
>>>>>>>> used on storm topologies.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *Best regards,*
>>>>>>>>
>>>>>>>> *K.Sai Dilip Reddy.*
>>>>>>>>
>>>>>>>> On Thu, Apr 28, 2016 at 12:28 PM, Sai Dilip Reddy Kiralam <
>>>>>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Spico,
>>>>>>>>>
>>>>>>>>> I use 9 bolts with parallelism of 1 and with 1 task for each
>>>>>>>>> bolt(default) and I'm not using any ConnectionPool for connecting to
>>>>>>>>> postgres.Just using jdbc classe examples here is source example
>>>>>>>>> http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-jdbc.html.
>>>>>>>>> I will check by using the ConnectionPool.
>>>>>>>>>
>>>>>>>>> Thank you.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *Best regards,*
>>>>>>>>>
>>>>>>>>> *K.Sai Dilip Reddy.*
>>>>>>>>>
>>>>>>>>> On Thu, Apr 28, 2016 at 12:19 PM, Spico Florin <
>>>>>>>>> spicoflorin@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello!
>>>>>>>>>> How many tasks do you have for inserting the data to your
>>>>>>>>>> database? Are you using ConnectionPool for connecting to Postgres? If your
>>>>>>>>>> number of task superseeds the number of max connections provided in
>>>>>>>>>> connection pool then your have a problem.
>>>>>>>>>> Please also check the number of max connections that your db
>>>>>>>>>> accepts.
>>>>>>>>>> I hope that these help.
>>>>>>>>>> Regards,
>>>>>>>>>> Florin
>>>>>>>>>>
>>>>>>>>>> On Thu, Apr 28, 2016 at 7:09 AM, Sai Dilip Reddy Kiralam <
>>>>>>>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> I have topology that make connection with postgresdb and insert
>>>>>>>>>>> the fields into tables of a test db.my topology is working fine but when I
>>>>>>>>>>> submit the topology it is establishing all the connections of db.but I
>>>>>>>>>>> don’t know why it is taking all the max connections.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> below attached the pics of pg_stat_activity.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> *Best regards,*
>>>>>>>>>>>
>>>>>>>>>>> *K.Sai Dilip Reddy.*
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>
>>>>>
>>>>
>>
>
Re: Storm topology using all the Max connections of db
Posted by Sai Dilip Reddy Kiralam <dk...@aadhya-analytics.com>.
Hi,
my supervisor logs shows the following error :
2016-05-05T15:45:15.365+0530 b.s.d.executor [INFO] Prepared bolt
__acker:(67)
2016-05-05T15:45:15.369+0530 b.s.d.executor [INFO] Prepared bolt
__system:(-1)
2016-05-05T15:45:18.481+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-0 is starting.
2016-05-05T15:45:18.486+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-1 is starting.
2016-05-05T15:45:18.486+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-3 is starting.
2016-05-05T15:45:18.488+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-2 is starting.
2016-05-05T15:45:18.488+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-3 is starting.
2016-05-05T15:45:18.489+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-4 is starting.
2016-05-05T15:45:18.489+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-5 is starting.
2016-05-05T15:45:18.490+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-6 is starting.
2016-05-05T15:45:18.490+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-7 is starting.
2016-05-05T15:45:18.490+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-8 is starting.
2016-05-05T15:45:18.490+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-9 is starting.
2016-05-05T15:45:18.491+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-10 is starting.
2016-05-05T15:45:18.491+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-11 is starting.
2016-05-05T15:45:18.492+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-12 is starting.
2016-05-05T15:45:18.493+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-13 is starting.
2016-05-05T15:45:18.493+0530 c.z.h.HikariDataSource [INFO] HikariCP pool
HikariPool-4 is starting.
2016-05-05T15:45:19.310+0530 c.z.h.p.HikariPool [ERROR] JDBC4
Connection.isValid() method not supported, connection test query must be
configured
2016-05-05T15:45:19.311+0530 c.z.h.p.HikariPool [ERROR] JDBC4
Connection.isValid() method not supported, connection test query must be
configured
2016-05-05T15:45:19.312+0530 c.z.h.p.HikariPool [ERROR] JDBC4
Connection.isValid() method not supported, connection test query must be
configured
2016-05-05T15:45:19.312+0530 c.z.h.p.HikariPool [ERROR] JDBC4
Connection.isValid() method not supported, connection test query must be
configured
2016-05-05T15:45:19.314+0530 c.z.h.p.HikariPool [ERROR] JDBC4
Connection.isValid() method not supported, connection test query must be
configured
2016-05-05T15:45:19.314+0530 c.z.h.p.HikariPool [ERROR] JDBC4
Connection.isValid() method not supported, connection test query must be
configured
2016-05-05T15:45:19.314+0530 c.z.h.p.HikariPool [ERROR] JDBC4
Connection.isValid() method not supported, connection test query must be
configured
2016-05-05T15:45:19.313+0530 c.z.h.p.HikariPool [ERROR] JDBC4
Connection.isValid() method not supported, connection test query must be
configured
2016-05-05T15:45:19.313+0530 c.z.h.p.HikariPool [ERROR] JDBC4
Connection.isValid() method not supported, connection test query must be
configured
do postgres is using all the connections because this error ?
*Best regards,*
*K.Sai Dilip Reddy.*
On Thu, May 5, 2016 at 10:57 AM, Sai Dilip Reddy Kiralam <
dkiralam@aadhya-analytics.com> wrote:
> Hi,
>
> I'm closing up and opening the connections after execution of insert query
> by my bolt.Do I need to increase the max connections db to a level of 500
> to 1000? Here in log file of supervisor it tells me the connection made by
> bolt is closed and again taking new connection.I think after closing it is
> going into ideal and not used for new connections.
>
> 2016-05-05T10:44:37.761+0530 STDIO [INFO] COMMAND TO CLOSE CONNECTION IS
> DONE
> 2016-05-05T10:44:37.761+0530 STDIO [INFO] DB connected....
>
> if I'm right what is the solution for those ideal connections?
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>
> On Thu, May 5, 2016 at 1:01 AM, Spico Florin <sp...@gmail.com>
> wrote:
>
>> hi!
>> you have 9 bolts with 50 max db connections. so for each bolt you get a
>> conection pool. try to decresae this number for example to 5 and check if
>> your performance if fine with your db
>> regards,
>> florin
>>
>> On Wednesday, May 4, 2016, Sai Dilip Reddy Kiralam <
>> dkiralam@aadhya-analytics.com> wrote:
>>
>>> sorry ! my db is not yet started, so it given me the error ! but when I
>>> gave that statement and run the topology then it is using more connections
>>> than the specified number.
>>>
>>>
>>>
>>> *Best regards,*
>>>
>>> *K.Sai Dilip Reddy.*
>>>
>>> On Wed, May 4, 2016 at 11:22 AM, Sai Dilip Reddy Kiralam <
>>> dkiralam@aadhya-analytics.com> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> I just added
>>>>
>>>> a statement
>>>> * // config.setMaximumPoolSize(50);*
>>>> public synchronized void prepare() {
>>>> if(dataSource == null) {
>>>> Properties properties = new Properties();
>>>> properties.putAll(configMap);
>>>> HikariConfig config = new HikariConfig(properties);
>>>> * config.setMaximumPoolSize(50);*
>>>> this.dataSource = new HikariDataSource(config);
>>>> this.dataSource.setAutoCommit(false);
>>>> }
>>>> }
>>>>
>>>>
>>>> and I'm getting the following error
>>>>
>>>> java.lang.RuntimeException: Fail-fast during pool initialization at
>>>> com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:475) at
>>>> com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:159) at
>>>> com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112) at
>>>> com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:78) at
>>>> com.aail.config.MConnectionProvider.prepare(MConnectionProvider.java:27) at
>>>> com.aail.storm.bolts.AbstractJdbcBolt.prepare(AbstractJdbcBolt.java:34) at
>>>> com.aail.storm.bolts.Inserts.prepare(Inserts.java:290) at
>>>> backtype.storm.daemon.executor$fn__6647$fn__6659.invoke(executor.clj:692)
>>>> at backtype.storm.util$async_loop$fn__459.invoke(util.clj:461) at
>>>> clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)
>>>> Caused by: org.postgresql.util.PSQLException: Connection attempt timed out.
>>>> at org.postgresql.Driver$ConnectThread.getResult(Driver.java:372) at
>>>> org.postgresql.Driver.connect(Driver.java:284) at
>>>> java.sql.DriverManager.getConnection(DriverManager.java:664) at
>>>> java.sql.DriverManager.getConnection(DriverManager.java:247) at
>>>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:99)
>>>> at
>>>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:82)
>>>> at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:398) at
>>>> com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:474) ... 10 more
>>>>
>>>>
>>>>
>>>>
>>>> *Best regards,*
>>>>
>>>> *K.Sai Dilip Reddy.*
>>>>
>>>> On Wed, May 4, 2016 at 9:53 AM, Sai Dilip Reddy Kiralam <
>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> Thank you Spico Florin.
>>>>>
>>>>>
>>>>>
>>>>> *Best regards,*
>>>>>
>>>>> *K.Sai Dilip Reddy.*
>>>>>
>>>>> On Wed, May 4, 2016 at 1:18 AM, Spico Florin <sp...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> hi!
>>>>>> please have a look
>>>>>> https://github.com/apache/storm/tree/master/external/storm-jdbc
>>>>>> where you hve to implemnt thw interface connectionprovider interface. you
>>>>>> have to dins a third paty librry that provides conection pooloing for
>>>>>> postgresql and use that library to implement the interface connection
>>>>>> provider. i hope that other community member will give you some other ideas.
>>>>>> i hope that these help.
>>>>>> regards,
>>>>>> florin
>>>>>>
>>>>>> On Tuesday, May 3, 2016, Sai Dilip Reddy Kiralam <
>>>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>>>
>>>>>>> to implemI'm using storm jdbc for connecting to db.how can I use
>>>>>>> pooling on codes on topology.please share information on connection pooling
>>>>>>> used on storm topologies.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Best regards,*
>>>>>>>
>>>>>>> *K.Sai Dilip Reddy.*
>>>>>>>
>>>>>>> On Thu, Apr 28, 2016 at 12:28 PM, Sai Dilip Reddy Kiralam <
>>>>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>>>>
>>>>>>>> Hi Spico,
>>>>>>>>
>>>>>>>> I use 9 bolts with parallelism of 1 and with 1 task for each
>>>>>>>> bolt(default) and I'm not using any ConnectionPool for connecting to
>>>>>>>> postgres.Just using jdbc classe examples here is source example
>>>>>>>> http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-jdbc.html.
>>>>>>>> I will check by using the ConnectionPool.
>>>>>>>>
>>>>>>>> Thank you.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *Best regards,*
>>>>>>>>
>>>>>>>> *K.Sai Dilip Reddy.*
>>>>>>>>
>>>>>>>> On Thu, Apr 28, 2016 at 12:19 PM, Spico Florin <
>>>>>>>> spicoflorin@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hello!
>>>>>>>>> How many tasks do you have for inserting the data to your
>>>>>>>>> database? Are you using ConnectionPool for connecting to Postgres? If your
>>>>>>>>> number of task superseeds the number of max connections provided in
>>>>>>>>> connection pool then your have a problem.
>>>>>>>>> Please also check the number of max connections that your db
>>>>>>>>> accepts.
>>>>>>>>> I hope that these help.
>>>>>>>>> Regards,
>>>>>>>>> Florin
>>>>>>>>>
>>>>>>>>> On Thu, Apr 28, 2016 at 7:09 AM, Sai Dilip Reddy Kiralam <
>>>>>>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> I have topology that make connection with postgresdb and insert
>>>>>>>>>> the fields into tables of a test db.my topology is working fine but when I
>>>>>>>>>> submit the topology it is establishing all the connections of db.but I
>>>>>>>>>> don’t know why it is taking all the max connections.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> below attached the pics of pg_stat_activity.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> *Best regards,*
>>>>>>>>>>
>>>>>>>>>> *K.Sai Dilip Reddy.*
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>
Re: Storm topology using all the Max connections of db
Posted by Sai Dilip Reddy Kiralam <dk...@aadhya-analytics.com>.
Hi,
I'm closing up and opening the connections after execution of insert query
by my bolt.Do I need to increase the max connections db to a level of 500
to 1000? Here in log file of supervisor it tells me the connection made by
bolt is closed and again taking new connection.I think after closing it is
going into ideal and not used for new connections.
2016-05-05T10:44:37.761+0530 STDIO [INFO] COMMAND TO CLOSE CONNECTION IS
DONE
2016-05-05T10:44:37.761+0530 STDIO [INFO] DB connected....
if I'm right what is the solution for those ideal connections?
*Best regards,*
*K.Sai Dilip Reddy.*
On Thu, May 5, 2016 at 1:01 AM, Spico Florin <sp...@gmail.com> wrote:
> hi!
> you have 9 bolts with 50 max db connections. so for each bolt you get a
> conection pool. try to decresae this number for example to 5 and check if
> your performance if fine with your db
> regards,
> florin
>
> On Wednesday, May 4, 2016, Sai Dilip Reddy Kiralam <
> dkiralam@aadhya-analytics.com> wrote:
>
>> sorry ! my db is not yet started, so it given me the error ! but when I
>> gave that statement and run the topology then it is using more connections
>> than the specified number.
>>
>>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>> On Wed, May 4, 2016 at 11:22 AM, Sai Dilip Reddy Kiralam <
>> dkiralam@aadhya-analytics.com> wrote:
>>
>>>
>>> Hi,
>>>
>>> I just added
>>>
>>> a statement
>>> * // config.setMaximumPoolSize(50);*
>>> public synchronized void prepare() {
>>> if(dataSource == null) {
>>> Properties properties = new Properties();
>>> properties.putAll(configMap);
>>> HikariConfig config = new HikariConfig(properties);
>>> * config.setMaximumPoolSize(50);*
>>> this.dataSource = new HikariDataSource(config);
>>> this.dataSource.setAutoCommit(false);
>>> }
>>> }
>>>
>>>
>>> and I'm getting the following error
>>>
>>> java.lang.RuntimeException: Fail-fast during pool initialization at
>>> com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:475) at
>>> com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:159) at
>>> com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112) at
>>> com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:78) at
>>> com.aail.config.MConnectionProvider.prepare(MConnectionProvider.java:27) at
>>> com.aail.storm.bolts.AbstractJdbcBolt.prepare(AbstractJdbcBolt.java:34) at
>>> com.aail.storm.bolts.Inserts.prepare(Inserts.java:290) at
>>> backtype.storm.daemon.executor$fn__6647$fn__6659.invoke(executor.clj:692)
>>> at backtype.storm.util$async_loop$fn__459.invoke(util.clj:461) at
>>> clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)
>>> Caused by: org.postgresql.util.PSQLException: Connection attempt timed out.
>>> at org.postgresql.Driver$ConnectThread.getResult(Driver.java:372) at
>>> org.postgresql.Driver.connect(Driver.java:284) at
>>> java.sql.DriverManager.getConnection(DriverManager.java:664) at
>>> java.sql.DriverManager.getConnection(DriverManager.java:247) at
>>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:99)
>>> at
>>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:82)
>>> at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:398) at
>>> com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:474) ... 10 more
>>>
>>>
>>>
>>>
>>> *Best regards,*
>>>
>>> *K.Sai Dilip Reddy.*
>>>
>>> On Wed, May 4, 2016 at 9:53 AM, Sai Dilip Reddy Kiralam <
>>> dkiralam@aadhya-analytics.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> Thank you Spico Florin.
>>>>
>>>>
>>>>
>>>> *Best regards,*
>>>>
>>>> *K.Sai Dilip Reddy.*
>>>>
>>>> On Wed, May 4, 2016 at 1:18 AM, Spico Florin <sp...@gmail.com>
>>>> wrote:
>>>>
>>>>> hi!
>>>>> please have a look
>>>>> https://github.com/apache/storm/tree/master/external/storm-jdbc where
>>>>> you hve to implemnt thw interface connectionprovider interface. you have to
>>>>> dins a third paty librry that provides conection pooloing for postgresql
>>>>> and use that library to implement the interface connection provider. i hope
>>>>> that other community member will give you some other ideas.
>>>>> i hope that these help.
>>>>> regards,
>>>>> florin
>>>>>
>>>>> On Tuesday, May 3, 2016, Sai Dilip Reddy Kiralam <
>>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>>
>>>>>> to implemI'm using storm jdbc for connecting to db.how can I use
>>>>>> pooling on codes on topology.please share information on connection pooling
>>>>>> used on storm topologies.
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Best regards,*
>>>>>>
>>>>>> *K.Sai Dilip Reddy.*
>>>>>>
>>>>>> On Thu, Apr 28, 2016 at 12:28 PM, Sai Dilip Reddy Kiralam <
>>>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>>>
>>>>>>> Hi Spico,
>>>>>>>
>>>>>>> I use 9 bolts with parallelism of 1 and with 1 task for each
>>>>>>> bolt(default) and I'm not using any ConnectionPool for connecting to
>>>>>>> postgres.Just using jdbc classe examples here is source example
>>>>>>> http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-jdbc.html.
>>>>>>> I will check by using the ConnectionPool.
>>>>>>>
>>>>>>> Thank you.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *Best regards,*
>>>>>>>
>>>>>>> *K.Sai Dilip Reddy.*
>>>>>>>
>>>>>>> On Thu, Apr 28, 2016 at 12:19 PM, Spico Florin <
>>>>>>> spicoflorin@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello!
>>>>>>>> How many tasks do you have for inserting the data to your
>>>>>>>> database? Are you using ConnectionPool for connecting to Postgres? If your
>>>>>>>> number of task superseeds the number of max connections provided in
>>>>>>>> connection pool then your have a problem.
>>>>>>>> Please also check the number of max connections that your db
>>>>>>>> accepts.
>>>>>>>> I hope that these help.
>>>>>>>> Regards,
>>>>>>>> Florin
>>>>>>>>
>>>>>>>> On Thu, Apr 28, 2016 at 7:09 AM, Sai Dilip Reddy Kiralam <
>>>>>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> I have topology that make connection with postgresdb and insert
>>>>>>>>> the fields into tables of a test db.my topology is working fine but when I
>>>>>>>>> submit the topology it is establishing all the connections of db.but I
>>>>>>>>> don’t know why it is taking all the max connections.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> below attached the pics of pg_stat_activity.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> *Best regards,*
>>>>>>>>>
>>>>>>>>> *K.Sai Dilip Reddy.*
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
Re: Storm topology using all the Max connections of db
Posted by Spico Florin <sp...@gmail.com>.
hi!
you have 9 bolts with 50 max db connections. so for each bolt you get a
conection pool. try to decresae this number for example to 5 and check if
your performance if fine with your db
regards,
florin
On Wednesday, May 4, 2016, Sai Dilip Reddy Kiralam <
dkiralam@aadhya-analytics.com> wrote:
> sorry ! my db is not yet started, so it given me the error ! but when I
> gave that statement and run the topology then it is using more connections
> than the specified number.
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>
> On Wed, May 4, 2016 at 11:22 AM, Sai Dilip Reddy Kiralam <
> dkiralam@aadhya-analytics.com
> <javascript:_e(%7B%7D,'cvml','dkiralam@aadhya-analytics.com');>> wrote:
>
>>
>> Hi,
>>
>> I just added
>>
>> a statement
>> * // config.setMaximumPoolSize(50);*
>> public synchronized void prepare() {
>> if(dataSource == null) {
>> Properties properties = new Properties();
>> properties.putAll(configMap);
>> HikariConfig config = new HikariConfig(properties);
>> * config.setMaximumPoolSize(50);*
>> this.dataSource = new HikariDataSource(config);
>> this.dataSource.setAutoCommit(false);
>> }
>> }
>>
>>
>> and I'm getting the following error
>>
>> java.lang.RuntimeException: Fail-fast during pool initialization at
>> com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:475) at
>> com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:159) at
>> com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112) at
>> com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:78) at
>> com.aail.config.MConnectionProvider.prepare(MConnectionProvider.java:27) at
>> com.aail.storm.bolts.AbstractJdbcBolt.prepare(AbstractJdbcBolt.java:34) at
>> com.aail.storm.bolts.Inserts.prepare(Inserts.java:290) at
>> backtype.storm.daemon.executor$fn__6647$fn__6659.invoke(executor.clj:692)
>> at backtype.storm.util$async_loop$fn__459.invoke(util.clj:461) at
>> clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)
>> Caused by: org.postgresql.util.PSQLException: Connection attempt timed out.
>> at org.postgresql.Driver$ConnectThread.getResult(Driver.java:372) at
>> org.postgresql.Driver.connect(Driver.java:284) at
>> java.sql.DriverManager.getConnection(DriverManager.java:664) at
>> java.sql.DriverManager.getConnection(DriverManager.java:247) at
>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:99)
>> at
>> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:82)
>> at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:398) at
>> com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:474) ... 10 more
>>
>>
>>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>> On Wed, May 4, 2016 at 9:53 AM, Sai Dilip Reddy Kiralam <
>> dkiralam@aadhya-analytics.com
>> <javascript:_e(%7B%7D,'cvml','dkiralam@aadhya-analytics.com');>> wrote:
>>
>>> Hi,
>>>
>>> Thank you Spico Florin.
>>>
>>>
>>>
>>> *Best regards,*
>>>
>>> *K.Sai Dilip Reddy.*
>>>
>>> On Wed, May 4, 2016 at 1:18 AM, Spico Florin <spicoflorin@gmail.com
>>> <javascript:_e(%7B%7D,'cvml','spicoflorin@gmail.com');>> wrote:
>>>
>>>> hi!
>>>> please have a look
>>>> https://github.com/apache/storm/tree/master/external/storm-jdbc where
>>>> you hve to implemnt thw interface connectionprovider interface. you have to
>>>> dins a third paty librry that provides conection pooloing for postgresql
>>>> and use that library to implement the interface connection provider. i hope
>>>> that other community member will give you some other ideas.
>>>> i hope that these help.
>>>> regards,
>>>> florin
>>>>
>>>> On Tuesday, May 3, 2016, Sai Dilip Reddy Kiralam <
>>>> dkiralam@aadhya-analytics.com
>>>> <javascript:_e(%7B%7D,'cvml','dkiralam@aadhya-analytics.com');>> wrote:
>>>>
>>>>> to implemI'm using storm jdbc for connecting to db.how can I use
>>>>> pooling on codes on topology.please share information on connection pooling
>>>>> used on storm topologies.
>>>>>
>>>>>
>>>>>
>>>>> *Best regards,*
>>>>>
>>>>> *K.Sai Dilip Reddy.*
>>>>>
>>>>> On Thu, Apr 28, 2016 at 12:28 PM, Sai Dilip Reddy Kiralam <
>>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>>
>>>>>> Hi Spico,
>>>>>>
>>>>>> I use 9 bolts with parallelism of 1 and with 1 task for each
>>>>>> bolt(default) and I'm not using any ConnectionPool for connecting to
>>>>>> postgres.Just using jdbc classe examples here is source example
>>>>>> http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-jdbc.html.
>>>>>> I will check by using the ConnectionPool.
>>>>>>
>>>>>> Thank you.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *Best regards,*
>>>>>>
>>>>>> *K.Sai Dilip Reddy.*
>>>>>>
>>>>>> On Thu, Apr 28, 2016 at 12:19 PM, Spico Florin <spicoflorin@gmail.com
>>>>>> > wrote:
>>>>>>
>>>>>>> Hello!
>>>>>>> How many tasks do you have for inserting the data to your
>>>>>>> database? Are you using ConnectionPool for connecting to Postgres? If your
>>>>>>> number of task superseeds the number of max connections provided in
>>>>>>> connection pool then your have a problem.
>>>>>>> Please also check the number of max connections that your db accepts.
>>>>>>> I hope that these help.
>>>>>>> Regards,
>>>>>>> Florin
>>>>>>>
>>>>>>> On Thu, Apr 28, 2016 at 7:09 AM, Sai Dilip Reddy Kiralam <
>>>>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I have topology that make connection with postgresdb and insert the
>>>>>>>> fields into tables of a test db.my topology is working fine but when I
>>>>>>>> submit the topology it is establishing all the connections of db.but I
>>>>>>>> don’t know why it is taking all the max connections.
>>>>>>>>
>>>>>>>>
>>>>>>>> below attached the pics of pg_stat_activity.
>>>>>>>>
>>>>>>>>
>>>>>>>> *Best regards,*
>>>>>>>>
>>>>>>>> *K.Sai Dilip Reddy.*
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>
>>
>
Re: Storm topology using all the Max connections of db
Posted by Sai Dilip Reddy Kiralam <dk...@aadhya-analytics.com>.
sorry ! my db is not yet started, so it given me the error ! but when I
gave that statement and run the topology then it is using more connections
than the specified number.
*Best regards,*
*K.Sai Dilip Reddy.*
On Wed, May 4, 2016 at 11:22 AM, Sai Dilip Reddy Kiralam <
dkiralam@aadhya-analytics.com> wrote:
>
> Hi,
>
> I just added
>
> a statement
> * // config.setMaximumPoolSize(50);*
> public synchronized void prepare() {
> if(dataSource == null) {
> Properties properties = new Properties();
> properties.putAll(configMap);
> HikariConfig config = new HikariConfig(properties);
> * config.setMaximumPoolSize(50);*
> this.dataSource = new HikariDataSource(config);
> this.dataSource.setAutoCommit(false);
> }
> }
>
>
> and I'm getting the following error
>
> java.lang.RuntimeException: Fail-fast during pool initialization at
> com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:475) at
> com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:159) at
> com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112) at
> com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:78) at
> com.aail.config.MConnectionProvider.prepare(MConnectionProvider.java:27) at
> com.aail.storm.bolts.AbstractJdbcBolt.prepare(AbstractJdbcBolt.java:34) at
> com.aail.storm.bolts.Inserts.prepare(Inserts.java:290) at
> backtype.storm.daemon.executor$fn__6647$fn__6659.invoke(executor.clj:692)
> at backtype.storm.util$async_loop$fn__459.invoke(util.clj:461) at
> clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)
> Caused by: org.postgresql.util.PSQLException: Connection attempt timed out.
> at org.postgresql.Driver$ConnectThread.getResult(Driver.java:372) at
> org.postgresql.Driver.connect(Driver.java:284) at
> java.sql.DriverManager.getConnection(DriverManager.java:664) at
> java.sql.DriverManager.getConnection(DriverManager.java:247) at
> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:99)
> at
> org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:82)
> at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:398) at
> com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:474) ... 10 more
>
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>
> On Wed, May 4, 2016 at 9:53 AM, Sai Dilip Reddy Kiralam <
> dkiralam@aadhya-analytics.com> wrote:
>
>> Hi,
>>
>> Thank you Spico Florin.
>>
>>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>> On Wed, May 4, 2016 at 1:18 AM, Spico Florin <sp...@gmail.com>
>> wrote:
>>
>>> hi!
>>> please have a look
>>> https://github.com/apache/storm/tree/master/external/storm-jdbc where
>>> you hve to implemnt thw interface connectionprovider interface. you have to
>>> dins a third paty librry that provides conection pooloing for postgresql
>>> and use that library to implement the interface connection provider. i hope
>>> that other community member will give you some other ideas.
>>> i hope that these help.
>>> regards,
>>> florin
>>>
>>> On Tuesday, May 3, 2016, Sai Dilip Reddy Kiralam <
>>> dkiralam@aadhya-analytics.com> wrote:
>>>
>>>> to implemI'm using storm jdbc for connecting to db.how can I use
>>>> pooling on codes on topology.please share information on connection pooling
>>>> used on storm topologies.
>>>>
>>>>
>>>>
>>>> *Best regards,*
>>>>
>>>> *K.Sai Dilip Reddy.*
>>>>
>>>> On Thu, Apr 28, 2016 at 12:28 PM, Sai Dilip Reddy Kiralam <
>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>
>>>>> Hi Spico,
>>>>>
>>>>> I use 9 bolts with parallelism of 1 and with 1 task for each
>>>>> bolt(default) and I'm not using any ConnectionPool for connecting to
>>>>> postgres.Just using jdbc classe examples here is source example
>>>>> http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-jdbc.html.
>>>>> I will check by using the ConnectionPool.
>>>>>
>>>>> Thank you.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *Best regards,*
>>>>>
>>>>> *K.Sai Dilip Reddy.*
>>>>>
>>>>> On Thu, Apr 28, 2016 at 12:19 PM, Spico Florin <sp...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hello!
>>>>>> How many tasks do you have for inserting the data to your
>>>>>> database? Are you using ConnectionPool for connecting to Postgres? If your
>>>>>> number of task superseeds the number of max connections provided in
>>>>>> connection pool then your have a problem.
>>>>>> Please also check the number of max connections that your db accepts.
>>>>>> I hope that these help.
>>>>>> Regards,
>>>>>> Florin
>>>>>>
>>>>>> On Thu, Apr 28, 2016 at 7:09 AM, Sai Dilip Reddy Kiralam <
>>>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I have topology that make connection with postgresdb and insert the
>>>>>>> fields into tables of a test db.my topology is working fine but when I
>>>>>>> submit the topology it is establishing all the connections of db.but I
>>>>>>> don’t know why it is taking all the max connections.
>>>>>>>
>>>>>>>
>>>>>>> below attached the pics of pg_stat_activity.
>>>>>>>
>>>>>>>
>>>>>>> *Best regards,*
>>>>>>>
>>>>>>> *K.Sai Dilip Reddy.*
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>
>
Re: Storm topology using all the Max connections of db
Posted by Sai Dilip Reddy Kiralam <dk...@aadhya-analytics.com>.
Hi,
I just added
a statement
* // config.setMaximumPoolSize(50);*
public synchronized void prepare() {
if(dataSource == null) {
Properties properties = new Properties();
properties.putAll(configMap);
HikariConfig config = new HikariConfig(properties);
* config.setMaximumPoolSize(50);*
this.dataSource = new HikariDataSource(config);
this.dataSource.setAutoCommit(false);
}
}
and I'm getting the following error
java.lang.RuntimeException: Fail-fast during pool initialization at
com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:475) at
com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:159) at
com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:112) at
com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:78) at
com.aail.config.MConnectionProvider.prepare(MConnectionProvider.java:27) at
com.aail.storm.bolts.AbstractJdbcBolt.prepare(AbstractJdbcBolt.java:34) at
com.aail.storm.bolts.Inserts.prepare(Inserts.java:290) at
backtype.storm.daemon.executor$fn__6647$fn__6659.invoke(executor.clj:692)
at backtype.storm.util$async_loop$fn__459.invoke(util.clj:461) at
clojure.lang.AFn.run(AFn.java:24) at java.lang.Thread.run(Thread.java:745)
Caused by: org.postgresql.util.PSQLException: Connection attempt timed out.
at org.postgresql.Driver$ConnectThread.getResult(Driver.java:372) at
org.postgresql.Driver.connect(Driver.java:284) at
java.sql.DriverManager.getConnection(DriverManager.java:664) at
java.sql.DriverManager.getConnection(DriverManager.java:247) at
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:99)
at
org.postgresql.ds.common.BaseDataSource.getConnection(BaseDataSource.java:82)
at com.zaxxer.hikari.pool.HikariPool.addConnection(HikariPool.java:398) at
com.zaxxer.hikari.pool.HikariPool.fillPool(HikariPool.java:474) ... 10 more
*Best regards,*
*K.Sai Dilip Reddy.*
On Wed, May 4, 2016 at 9:53 AM, Sai Dilip Reddy Kiralam <
dkiralam@aadhya-analytics.com> wrote:
> Hi,
>
> Thank you Spico Florin.
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>
> On Wed, May 4, 2016 at 1:18 AM, Spico Florin <sp...@gmail.com>
> wrote:
>
>> hi!
>> please have a look
>> https://github.com/apache/storm/tree/master/external/storm-jdbc where
>> you hve to implemnt thw interface connectionprovider interface. you have to
>> dins a third paty librry that provides conection pooloing for postgresql
>> and use that library to implement the interface connection provider. i hope
>> that other community member will give you some other ideas.
>> i hope that these help.
>> regards,
>> florin
>>
>> On Tuesday, May 3, 2016, Sai Dilip Reddy Kiralam <
>> dkiralam@aadhya-analytics.com> wrote:
>>
>>> to implemI'm using storm jdbc for connecting to db.how can I use pooling
>>> on codes on topology.please share information on connection pooling used on
>>> storm topologies.
>>>
>>>
>>>
>>> *Best regards,*
>>>
>>> *K.Sai Dilip Reddy.*
>>>
>>> On Thu, Apr 28, 2016 at 12:28 PM, Sai Dilip Reddy Kiralam <
>>> dkiralam@aadhya-analytics.com> wrote:
>>>
>>>> Hi Spico,
>>>>
>>>> I use 9 bolts with parallelism of 1 and with 1 task for each
>>>> bolt(default) and I'm not using any ConnectionPool for connecting to
>>>> postgres.Just using jdbc classe examples here is source example
>>>> http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-jdbc.html.
>>>> I will check by using the ConnectionPool.
>>>>
>>>> Thank you.
>>>>
>>>>
>>>>
>>>>
>>>> *Best regards,*
>>>>
>>>> *K.Sai Dilip Reddy.*
>>>>
>>>> On Thu, Apr 28, 2016 at 12:19 PM, Spico Florin <sp...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hello!
>>>>> How many tasks do you have for inserting the data to your database?
>>>>> Are you using ConnectionPool for connecting to Postgres? If your number of
>>>>> task superseeds the number of max connections provided in connection pool
>>>>> then your have a problem.
>>>>> Please also check the number of max connections that your db accepts.
>>>>> I hope that these help.
>>>>> Regards,
>>>>> Florin
>>>>>
>>>>> On Thu, Apr 28, 2016 at 7:09 AM, Sai Dilip Reddy Kiralam <
>>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> I have topology that make connection with postgresdb and insert the
>>>>>> fields into tables of a test db.my topology is working fine but when I
>>>>>> submit the topology it is establishing all the connections of db.but I
>>>>>> don’t know why it is taking all the max connections.
>>>>>>
>>>>>>
>>>>>> below attached the pics of pg_stat_activity.
>>>>>>
>>>>>>
>>>>>> *Best regards,*
>>>>>>
>>>>>> *K.Sai Dilip Reddy.*
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>
Re: Storm topology using all the Max connections of db
Posted by Sai Dilip Reddy Kiralam <dk...@aadhya-analytics.com>.
Hi,
Thank you Spico Florin.
*Best regards,*
*K.Sai Dilip Reddy.*
On Wed, May 4, 2016 at 1:18 AM, Spico Florin <sp...@gmail.com> wrote:
> hi!
> please have a look
> https://github.com/apache/storm/tree/master/external/storm-jdbc where you
> hve to implemnt thw interface connectionprovider interface. you have to
> dins a third paty librry that provides conection pooloing for postgresql
> and use that library to implement the interface connection provider. i hope
> that other community member will give you some other ideas.
> i hope that these help.
> regards,
> florin
>
> On Tuesday, May 3, 2016, Sai Dilip Reddy Kiralam <
> dkiralam@aadhya-analytics.com> wrote:
>
>> to implemI'm using storm jdbc for connecting to db.how can I use pooling
>> on codes on topology.please share information on connection pooling used on
>> storm topologies.
>>
>>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>> On Thu, Apr 28, 2016 at 12:28 PM, Sai Dilip Reddy Kiralam <
>> dkiralam@aadhya-analytics.com> wrote:
>>
>>> Hi Spico,
>>>
>>> I use 9 bolts with parallelism of 1 and with 1 task for each
>>> bolt(default) and I'm not using any ConnectionPool for connecting to
>>> postgres.Just using jdbc classe examples here is source example
>>> http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-jdbc.html.
>>> I will check by using the ConnectionPool.
>>>
>>> Thank you.
>>>
>>>
>>>
>>>
>>> *Best regards,*
>>>
>>> *K.Sai Dilip Reddy.*
>>>
>>> On Thu, Apr 28, 2016 at 12:19 PM, Spico Florin <sp...@gmail.com>
>>> wrote:
>>>
>>>> Hello!
>>>> How many tasks do you have for inserting the data to your database?
>>>> Are you using ConnectionPool for connecting to Postgres? If your number of
>>>> task superseeds the number of max connections provided in connection pool
>>>> then your have a problem.
>>>> Please also check the number of max connections that your db accepts.
>>>> I hope that these help.
>>>> Regards,
>>>> Florin
>>>>
>>>> On Thu, Apr 28, 2016 at 7:09 AM, Sai Dilip Reddy Kiralam <
>>>> dkiralam@aadhya-analytics.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have topology that make connection with postgresdb and insert the
>>>>> fields into tables of a test db.my topology is working fine but when I
>>>>> submit the topology it is establishing all the connections of db.but I
>>>>> don’t know why it is taking all the max connections.
>>>>>
>>>>>
>>>>> below attached the pics of pg_stat_activity.
>>>>>
>>>>>
>>>>> *Best regards,*
>>>>>
>>>>> *K.Sai Dilip Reddy.*
>>>>>
>>>>
>>>>
>>>
>>
Re: Storm topology using all the Max connections of db
Posted by Spico Florin <sp...@gmail.com>.
hi!
please have a look
https://github.com/apache/storm/tree/master/external/storm-jdbc where you
hve to implemnt thw interface connectionprovider interface. you have to
dins a third paty librry that provides conection pooloing for postgresql
and use that library to implement the interface connection provider. i hope
that other community member will give you some other ideas.
i hope that these help.
regards,
florin
On Tuesday, May 3, 2016, Sai Dilip Reddy Kiralam <
dkiralam@aadhya-analytics.com> wrote:
> to implemI'm using storm jdbc for connecting to db.how can I use pooling
> on codes on topology.please share information on connection pooling used on
> storm topologies.
>
>
>
> *Best regards,*
>
> *K.Sai Dilip Reddy.*
>
> On Thu, Apr 28, 2016 at 12:28 PM, Sai Dilip Reddy Kiralam <
> dkiralam@aadhya-analytics.com
> <javascript:_e(%7B%7D,'cvml','dkiralam@aadhya-analytics.com');>> wrote:
>
>> Hi Spico,
>>
>> I use 9 bolts with parallelism of 1 and with 1 task for each
>> bolt(default) and I'm not using any ConnectionPool for connecting to
>> postgres.Just using jdbc classe examples here is source example
>> http://storm.apache.org/releases/2.0.0-SNAPSHOT/storm-jdbc.html.
>> I will check by using the ConnectionPool.
>>
>> Thank you.
>>
>>
>>
>>
>> *Best regards,*
>>
>> *K.Sai Dilip Reddy.*
>>
>> On Thu, Apr 28, 2016 at 12:19 PM, Spico Florin <spicoflorin@gmail.com
>> <javascript:_e(%7B%7D,'cvml','spicoflorin@gmail.com');>> wrote:
>>
>>> Hello!
>>> How many tasks do you have for inserting the data to your database?
>>> Are you using ConnectionPool for connecting to Postgres? If your number of
>>> task superseeds the number of max connections provided in connection pool
>>> then your have a problem.
>>> Please also check the number of max connections that your db accepts.
>>> I hope that these help.
>>> Regards,
>>> Florin
>>>
>>> On Thu, Apr 28, 2016 at 7:09 AM, Sai Dilip Reddy Kiralam <
>>> dkiralam@aadhya-analytics.com
>>> <javascript:_e(%7B%7D,'cvml','dkiralam@aadhya-analytics.com');>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I have topology that make connection with postgresdb and insert the
>>>> fields into tables of a test db.my topology is working fine but when I
>>>> submit the topology it is establishing all the connections of db.but I
>>>> don’t know why it is taking all the max connections.
>>>>
>>>>
>>>> below attached the pics of pg_stat_activity.
>>>>
>>>>
>>>> *Best regards,*
>>>>
>>>> *K.Sai Dilip Reddy.*
>>>>
>>>
>>>
>>
>