You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Theo Sweeny <th...@avios.com> on 2016/03/24 18:19:09 UTC

Tomcat 8 Connection Reset Issue

Hello - we are running Tomcat v8.0.21 on RH7 with Java7.


Recently we changed the datasource's to use connection pooling but as a result we are seeing connection timeouts in the logs as seen here -


2016-03-24 16:27:36,638 14321113 [http-nio-20180-exec-3] INFO  org.springframework.security.oauth2.provider.endpoint.TokenEndpoint Correlation-Identifier=77063426-8942-4259-aafd-cd9bc1d03305, Timestamp=2016-03-24T14:54:46.084Z - Handling error: ResourceAccessException, I/O error on POST request for "https://localhost:8080/membership/programmes/ATRP/member-identification":Connection reset; nested exception is java.net.SocketException: Connection reset


The timeouts occur about every 1 in 3 attempts.


Here is the new datasource config -


<Resource name="jdbc/MyDataSource"
        auth="Container"
        factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
        type="javax.sql.DataSource"
        driverClassName="oracle.jdbc.OracleDriver"
        url="jdbc:oracle:thin:@myhost:1521:MYSID"
        username="MYUSER"
        password="xxxxxxxxx"
        removeAbandoned="True"
        removeAbandonedTimeout="30"
        logAbandoned="True"
        abandonWhenPercentageFull="0"
        testOnBorrow="True"
        testOnReturn="False"
        testWhileIdle="True"
        maxActive="32"
        initialSize="8"
        maxIdle="8"
        minIdle="8"
        minEvictableIdleTimeMillis="15000"
        validationQuery="select 1 from dual"
        validationInterval="15000"
        validationQueryTimeout="15"
        timeBetweenEvictionRunsMillis="15000"
        logValidationErrors="True"
        jdbcInterceptors="ConnectionState;StatementFinalizer;SlowQueryReport(threshold=1500);QueryTimeoutInterceptor(queryTimeout=10),ResetAbandonedTimer"
        maxWait="-1"/>

Are there any conflicts within this config that could be resulting in this issue?


Thank you,


Theo

Avios Group (AGL) Ltd is a limited company registered in England (registered number 2260073 and VAT number 512566754) whose registered address is Astral Towers, Betts Way, London Road, Crawley, West Sussex RH10 9XY . Avios Group (AGL) Limited is part of the IAG group of companies This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager.

Re: Tomcat 8 Connection Reset Issue

Posted by Theo Sweeny <th...@avios.com>.
Hi Chris,
________________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: 29 March 2016 16:48
To: Tomcat Users List
Subject: Re: Tomcat 8 Connection Reset Issue

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Theo,

On 3/29/16 2:55 AM, Theo Sweeny wrote:
> Yes - that does look like a spring connect timeout - which then
> passes back to java.net.SocketException: Connection reset. So
> either the client or the server is resetting before the connection
> can complete.
>
> The only system change to coincide with these connection resets is
> the new connection pool config as listed above. This implies that
> there is some conflict within the datasource config.

I don't believe that's the case, unless you are using the same port
numbers for both your JDBC connections and whatever other component is
failing. Have you tried reverting the JNDI configuration only?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlb6o84ACgkQ9CaO5/Lv0PC75QCgiNExQYWamSs9kmnZIoEZEnMP
3FcAoLIBEzB9NpLbvbjwqHH5wdQ/CIe7
=X5ZC
-----END PGP SIGNATURE-----

Yes I rolled back the datasource configs but the problem persisted.

The issue resides with the F5 firewall hanging every other request. When the F5 is removed from the network routing, the connection reset issue resolves itself.

Thanks for the help!

Theo

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Avios Group (AGL) Ltd is a limited company registered in England (registered number 2260073 and VAT number 512566754) whose registered address is Astral Towers, Betts Way, London Road, Crawley, West Sussex RH10 9XY . Avios Group (AGL) Limited is part of the IAG group of companies This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat 8 Connection Reset Issue

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Theo,

On 3/29/16 2:55 AM, Theo Sweeny wrote:
> Yes - that does look like a spring connect timeout - which then 
> passes back to java.net.SocketException: Connection reset. So
> either the client or the server is resetting before the connection
> can complete.
> 
> The only system change to coincide with these connection resets is 
> the new connection pool config as listed above. This implies that 
> there is some conflict within the datasource config.

I don't believe that's the case, unless you are using the same port
numbers for both your JDBC connections and whatever other component is
failing. Have you tried reverting the JNDI configuration only?

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlb6o84ACgkQ9CaO5/Lv0PC75QCgiNExQYWamSs9kmnZIoEZEnMP
3FcAoLIBEzB9NpLbvbjwqHH5wdQ/CIe7
=X5ZC
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat 8 Connection Reset Issue

Posted by Theo Sweeny <th...@avios.com>.
Hi Chris,

________________________________________
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: 24 March 2016 19:42
To: Tomcat Users List
Subject: Re: Tomcat 8 Connection Reset Issue

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Theo,

On 3/24/16 1:19 PM, Theo Sweeny wrote:
> Hello - we are running Tomcat v8.0.21 on RH7 with Java7.
>
> Recently we changed the datasource's to use connection pooling but
> as a result we are seeing connection timeouts in the logs as seen
> here -
>
> 2016-03-24 16:27:36,638 14321113 [http-nio-20180-exec-3] INFO
> org.springframework.security.oauth2.provider.endpoint.TokenEndpoint
>
>
Correlation-Identifier=77063426-8942-4259-aafd-cd9bc1d03305,
> Timestamp=2016-03-24T14:54:46.084Z - Handling error:
> ResourceAccessException, I/O error on POST request for
> "https://localhost:8080/membership/programmes/ATRP/member-identificati
on":Connection
>
>
reset; nested exception is java.net.SocketException: Connection
> reset

That looks like an error with a Spring-related component, not Tomcat.
(Right?). There are plenty of folks here who might be able to answer
this question, but I think this is actually off-topic for this list.
(No problem; carry on. Just pointing it out).

> The timeouts occur about every 1 in 3 attempts.
>
>
> Here is the new datasource config -
>
>
> <Resource name="jdbc/MyDataSource" auth="Container"
> factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
> type="javax.sql.DataSource"
> driverClassName="oracle.jdbc.OracleDriver"
> url="jdbc:oracle:thin:@myhost:1521:MYSID" username="MYUSER"
> password="xxxxxxxxx" removeAbandoned="True"
> removeAbandonedTimeout="30" logAbandoned="True"
> abandonWhenPercentageFull="0" testOnBorrow="True"
> testOnReturn="False" testWhileIdle="True" maxActive="32"
> initialSize="8" maxIdle="8" minIdle="8"
> minEvictableIdleTimeMillis="15000" validationQuery="select 1 from
> dual" validationInterval="15000" validationQueryTimeout="15"
> timeBetweenEvictionRunsMillis="15000" logValidationErrors="True"
> jdbcInterceptors="ConnectionState;StatementFinalizer;SlowQueryReport(t
hreshold=1500);QueryTimeoutInterceptor(queryTimeout=10),ResetAbandonedTi
mer"
>
>
maxWait="-1"/>
>
> Are there any conflicts within this config that could be resulting
> in this issue?

I don't believe your JNDI DataSource has anything to do with making
outgoing connections to OAuth providers...

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlb0Qx0ACgkQ9CaO5/Lv0PA7KACfYVLubbNfIxKrInh9SNwY/JHf
h+kAnRRevkPjZBdCl0RD53QdtsndtXGe
=h3bQ
-----END PGP SIGNATURE-----

Thanks for the replay.

Yes - that does look like a spring connect timeout - which then passes back to java.net.SocketException: Connection reset. So either the client or the server is resetting before the connection can complete.

The only system change to coincide with these connection resets is the new connection pool config as listed above. This implies that there is some conflict within the datasource config.

Theo
Avios Group (AGL) Ltd is a limited company registered in England (registered number 2260073 and VAT number 512566754) whose registered address is Astral Towers, Betts Way, London Road, Crawley, West Sussex RH10 9XY . Avios Group (AGL) Limited is part of the IAG group of companies This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org

Re: Tomcat 8 Connection Reset Issue

Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Theo,

On 3/24/16 1:19 PM, Theo Sweeny wrote:
> Hello - we are running Tomcat v8.0.21 on RH7 with Java7.
> 
> Recently we changed the datasource's to use connection pooling but
> as a result we are seeing connection timeouts in the logs as seen
> here -
> 
> 2016-03-24 16:27:36,638 14321113 [http-nio-20180-exec-3] INFO 
> org.springframework.security.oauth2.provider.endpoint.TokenEndpoint
>
> 
Correlation-Identifier=77063426-8942-4259-aafd-cd9bc1d03305,
> Timestamp=2016-03-24T14:54:46.084Z - Handling error: 
> ResourceAccessException, I/O error on POST request for 
> "https://localhost:8080/membership/programmes/ATRP/member-identificati
on":Connection
>
> 
reset; nested exception is java.net.SocketException: Connection
> reset

That looks like an error with a Spring-related component, not Tomcat.
(Right?). There are plenty of folks here who might be able to answer
this question, but I think this is actually off-topic for this list.
(No problem; carry on. Just pointing it out).

> The timeouts occur about every 1 in 3 attempts.
> 
> 
> Here is the new datasource config -
> 
> 
> <Resource name="jdbc/MyDataSource" auth="Container" 
> factory="org.apache.tomcat.jdbc.pool.DataSourceFactory" 
> type="javax.sql.DataSource" 
> driverClassName="oracle.jdbc.OracleDriver" 
> url="jdbc:oracle:thin:@myhost:1521:MYSID" username="MYUSER" 
> password="xxxxxxxxx" removeAbandoned="True" 
> removeAbandonedTimeout="30" logAbandoned="True" 
> abandonWhenPercentageFull="0" testOnBorrow="True" 
> testOnReturn="False" testWhileIdle="True" maxActive="32" 
> initialSize="8" maxIdle="8" minIdle="8" 
> minEvictableIdleTimeMillis="15000" validationQuery="select 1 from
> dual" validationInterval="15000" validationQueryTimeout="15" 
> timeBetweenEvictionRunsMillis="15000" logValidationErrors="True" 
> jdbcInterceptors="ConnectionState;StatementFinalizer;SlowQueryReport(t
hreshold=1500);QueryTimeoutInterceptor(queryTimeout=10),ResetAbandonedTi
mer"
>
> 
maxWait="-1"/>
> 
> Are there any conflicts within this config that could be resulting
> in this issue?

I don't believe your JNDI DataSource has anything to do with making
outgoing connections to OAuth providers...

- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlb0Qx0ACgkQ9CaO5/Lv0PA7KACfYVLubbNfIxKrInh9SNwY/JHf
h+kAnRRevkPjZBdCl0RD53QdtsndtXGe
=h3bQ
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org