You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by "Carlos Manuel Fernandes (DSI)" <ca...@cgd.pt> on 2021/04/19 20:11:17 UTC
Possible problem with DBCPConnectionPool 1.12.1
Hello,
After upgrading to Nifi 1.13.2 from Nifi 1.9.2, I notice a problem with DBCPConnectionPool, several connections remain on pool above 'Max Idle Connections' number after the 'Minimum Evictable Idle Time' is passed'.
To reproduce the issue I use a simple flow: ExecuteSql (Concurrent Tasks:10) ->LogAttribute
On Execute Sql:
Database Connection Pooling Service : XPTO-CQ
select query : select count(*) as cont from Table
On pool XPTO-CQ :
Database Connection URL jdbc:xxx:thin://xpto:10800
Database Driver Class Name org.apache.somedriver
Validation query select 1
Minimum Idle Connections 0
Max Idle Connections 8
Max Connection Lifetime -1
Time Between Eviction Runs -1
Minimum Evictable Idle Time 1 mins
Soft Minimum Evictable Idle Time -1
After run ExecuteSql several times the number of connections ESTABLISHED, using netstat -na | grep 10800 is above 8 and never be destroyed. The unique workaround I founded to avoid a rise in connections was to put 'Max Idle Connections' to 0, in practice not use the pool at all.
Thanks
Carlos
RE: Possible problem with DBCPConnectionPool 1.12.1
Posted by "Carlos Manuel Fernandes (DSI)" <ca...@cgd.pt>.
Matt, you are right , setting Max Idle to 8 and having 10 concurrent tasks, the number of idle connections rise above maxIdle and rest in this state some time but slowly return to maxidle,
Setting Max Idle to 15 and having 10 concurrent tasks the problem don’t occur.
Thanks for your help.
Carlos
-----Original Message-----
From: Matt Burgess [mailto:mattyb149@apache.org]
Sent: 19 de abril de 2021 21:51
To: users@nifi.apache.org
Subject: Re: Possible problem with DBCPConnectionPool 1.12.1
Carlos,
From the DBCP doc:
If maxIdle is set too low on heavily loaded systems it is possible you will see connections being closed and almost immediately new connections being opened. This is a result of the active threads momentarily closing connections faster than they are opening them, causing the number of idle connections to rise above maxIdle. The best value for maxIdle for heavily loaded system will vary but the default is a good starting point.
In your case, you have Max Idle set to 8 but you have 10 concurrent tasks closing connections at the same time, I suspect this may cause some mishandling of connections by the pool. Can you set Max Idle to
15 and see if the problem persists?
Thanks,
Matt
On Mon, Apr 19, 2021 at 4:11 PM Carlos Manuel Fernandes (DSI) <ca...@cgd.pt> wrote:
>
> Hello,
>
>
>
> After upgrading to Nifi 1.13.2 from Nifi 1.9.2, I notice a problem with DBCPConnectionPool, several connections remain on pool above ‘Max Idle Connections’ number after the ‘Minimum Evictable Idle Time’ is passed’.
>
>
>
> To reproduce the issue I use a simple flow: ExecuteSql (Concurrent
> Tasks:10) ->LogAttribute
>
>
>
> On Execute Sql:
>
> Database Connection Pooling Service : XPTO-CQ
>
> select query : select count(*) as cont from Table
>
>
>
> On pool XPTO-CQ :
>
> Database Connection URL jdbc:xxx:thin://xpto:10800
>
> Database Driver Class Name org.apache.somedriver
>
> Validation query select 1
>
> Minimum Idle Connections 0
>
> Max Idle Connections 8
>
> Max Connection Lifetime -1
>
> Time Between Eviction Runs -1
>
> Minimum Evictable Idle Time 1 mins
>
> Soft Minimum Evictable Idle Time -1
>
>
>
> After run ExecuteSql several times the number of connections ESTABLISHED, using netstat –na | grep 10800 is above 8 and never be destroyed. The unique workaround I founded to avoid a rise in connections was to put ‘Max Idle Connections’ to 0, in practice not use the pool at all.
>
>
>
> Thanks
>
> Carlos
>
>
Re: Possible problem with DBCPConnectionPool 1.12.1
Posted by Matt Burgess <ma...@apache.org>.
Carlos,
From the DBCP doc:
If maxIdle is set too low on heavily loaded systems it is possible you
will see connections being closed and almost immediately new
connections being opened. This is a result of the active threads
momentarily closing connections faster than they are opening them,
causing the number of idle connections to rise above maxIdle. The best
value for maxIdle for heavily loaded system will vary but the default
is a good starting point.
In your case, you have Max Idle set to 8 but you have 10 concurrent
tasks closing connections at the same time, I suspect this may cause
some mishandling of connections by the pool. Can you set Max Idle to
15 and see if the problem persists?
Thanks,
Matt
On Mon, Apr 19, 2021 at 4:11 PM Carlos Manuel Fernandes (DSI)
<ca...@cgd.pt> wrote:
>
> Hello,
>
>
>
> After upgrading to Nifi 1.13.2 from Nifi 1.9.2, I notice a problem with DBCPConnectionPool, several connections remain on pool above ‘Max Idle Connections’ number after the ‘Minimum Evictable Idle Time’ is passed’.
>
>
>
> To reproduce the issue I use a simple flow: ExecuteSql (Concurrent Tasks:10) ->LogAttribute
>
>
>
> On Execute Sql:
>
> Database Connection Pooling Service : XPTO-CQ
>
> select query : select count(*) as cont from Table
>
>
>
> On pool XPTO-CQ :
>
> Database Connection URL jdbc:xxx:thin://xpto:10800
>
> Database Driver Class Name org.apache.somedriver
>
> Validation query select 1
>
> Minimum Idle Connections 0
>
> Max Idle Connections 8
>
> Max Connection Lifetime -1
>
> Time Between Eviction Runs -1
>
> Minimum Evictable Idle Time 1 mins
>
> Soft Minimum Evictable Idle Time -1
>
>
>
> After run ExecuteSql several times the number of connections ESTABLISHED, using netstat –na | grep 10800 is above 8 and never be destroyed. The unique workaround I founded to avoid a rise in connections was to put ‘Max Idle Connections’ to 0, in practice not use the pool at all.
>
>
>
> Thanks
>
> Carlos
>
>