You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Igor T <ig...@gmail.com> on 2019/03/13 20:30:48 UTC
Tomcat 9 Nio2+OpenSSL problem (very likely a bug)
Prerequisites:
OS: Windows Server 2012 R2
Java: checked on both jdk1.8.0_162 jdk1.8.0_181
Tomcat: windows x64 builds checked on 9.0.12, 9.0.16, 9.0.17-dev
Valid SSL certificates
Content of file located at webapp/ROOT/1.txt: []
Tomcat's connector settings:
<Connector port="443"
protocol="org.apache.coyote.http11.Http11Nio2Protocol"
sslImplementationName="org.apache.tomcat.util.net.openssl.OpenSSLImplementation"
connectionTimeout="5000"
SSLEnabled="true"
scheme="https"
secure="true"
>
This configuration leads to 50% of the traffic to be rejected with
Connection resets. First socket connects and receives the service, but
every second is resetted.
Exactly this combination leads to connection resets:
protocol="org.apache.coyote.http11.Http11Nio2Protocol"
sslImplementationName="org.apache.tomcat.util.net.openssl.OpenSSLImplementation"
Configurations that work well without connection resets:
protocol="org.apache.coyote.http11.Http11Nio2Protocol"
sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation"
or
protocol="org.apache.coyote.http11.Http11NioProtocol"
sslImplementationName="org.apache.tomcat.util.net.openssl.OpenSSLImplementation"
Java code to reproduce the connection resets (works well with any
other secure server):
(there is no resets if a variable named FIX__gotoSleepAfterHandshake = true)
public class CheckConnectionResets{
static String host = "your-test-host";
public static void main( String[] args ) throws
InterruptedException, IOException{
SSLSocketFactory factory =
(SSLSocketFactory)SSLSocketFactory.getDefault();
int nRuns = 4;
int success = 0;
int denial = 0;
boolean FIX__gotoSleepAfterHandshake = false;
for( int i = 0; i < nRuns; i++ ){
try ( SSLSocket socket = (SSLSocket)factory.createSocket(
host, 443 ) ){
if( FIX__gotoSleepAfterHandshake ){
socket.startHandshake();
Thread.sleep( 500 );
}
try ( PrintWriter out = new PrintWriter( new
BufferedWriter( new OutputStreamWriter( socket.getOutputStream() ) )
);
InputStream is = socket.getInputStream(); ){
out.println( "GET /1.txt HTTP/1.1" );
out.println( "Host: " + host );
out.println( "Accept: */*" );
out.println();
out.flush();
if( out.checkError() ){
System.out.println( "SSLSocketClient:
java.io.PrintWriter error" );
}
Instant start = Instant.now();
/* read full response */
byte[] buff = new byte[ 1024 ];
int read = is.read( buff );
success++;
System.out.println( "success: " + success + ",
read " + read + " bytes for: " + start.until( Instant.now(),
ChronoUnit.MILLIS ) + "ms" );
} catch ( IOException e ) {
denial++;
System.err.println( "denial: " + denial + ", " +
e.getMessage() );
}}}}}
Sample output:
success: 1, read 73 bytes for: 78ms
denial: 1, Connection reset
success: 2, read 73 bytes for: 78ms
denial: 2, Connection reset
The bug is stable, and always reproducible.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 9 Nio2+OpenSSL problem (very likely a bug)
Posted by Rémy Maucherat <re...@apache.org>.
On Wed, Mar 13, 2019 at 10:21 PM Mark Thomas <ma...@apache.org> wrote:
> On 13/03/2019 20:30, Igor T wrote:
> > Prerequisites:
> > OS: Windows Server 2012 R2
> > Java: checked on both jdk1.8.0_162 jdk1.8.0_181
> > Tomcat: windows x64 builds checked on 9.0.12, 9.0.16, 9.0.17-dev
>
> 9.0.17-dev at which point in time?
>
Since 9.0.12 and 16 do the same, I wouldn't look at that at all. Something
simple like this works in the general case, there must be something
specific here. So it's Windows, which some unspecified OpenSSL version.
Rémy
>
> Have you tested the current 9.0.17 release candidate (see dev@ for
> details)
>
> Mark
>
>
>
> > Valid SSL certificates
> > Content of file located at webapp/ROOT/1.txt: []
> > Tomcat's connector settings:
> > <Connector port="443"
> >
> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
> >
> > sslImplementationName="org.apache.tomcat.util.net
> .openssl.OpenSSLImplementation"
> > connectionTimeout="5000"
> > SSLEnabled="true"
> > scheme="https"
> > secure="true"
> > >
> >
> > This configuration leads to 50% of the traffic to be rejected with
> > Connection resets. First socket connects and receives the service, but
> > every second is resetted.
> >
> > Exactly this combination leads to connection resets:
> > protocol="org.apache.coyote.http11.Http11Nio2Protocol"
> > sslImplementationName="org.apache.tomcat.util.net
> .openssl.OpenSSLImplementation"
> >
> > Configurations that work well without connection resets:
> > protocol="org.apache.coyote.http11.Http11Nio2Protocol"
> >
> sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation"
> > or
> > protocol="org.apache.coyote.http11.Http11NioProtocol"
> > sslImplementationName="org.apache.tomcat.util.net
> .openssl.OpenSSLImplementation"
> >
> > Java code to reproduce the connection resets (works well with any
> > other secure server):
> > (there is no resets if a variable named FIX__gotoSleepAfterHandshake =
> true)
> >
> > public class CheckConnectionResets{
> > static String host = "your-test-host";
> >
> > public static void main( String[] args ) throws
> > InterruptedException, IOException{
> >
> > SSLSocketFactory factory =
> > (SSLSocketFactory)SSLSocketFactory.getDefault();
> > int nRuns = 4;
> > int success = 0;
> > int denial = 0;
> >
> > boolean FIX__gotoSleepAfterHandshake = false;
> >
> > for( int i = 0; i < nRuns; i++ ){
> > try ( SSLSocket socket = (SSLSocket)factory.createSocket(
> > host, 443 ) ){
> >
> > if( FIX__gotoSleepAfterHandshake ){
> > socket.startHandshake();
> > Thread.sleep( 500 );
> > }
> > try ( PrintWriter out = new PrintWriter( new
> > BufferedWriter( new OutputStreamWriter( socket.getOutputStream() ) )
> > );
> > InputStream is = socket.getInputStream(); ){
> >
> > out.println( "GET /1.txt HTTP/1.1" );
> > out.println( "Host: " + host );
> > out.println( "Accept: */*" );
> > out.println();
> > out.flush();
> >
> > if( out.checkError() ){
> > System.out.println( "SSLSocketClient:
> > java.io.PrintWriter error" );
> > }
> >
> > Instant start = Instant.now();
> > /* read full response */
> > byte[] buff = new byte[ 1024 ];
> > int read = is.read( buff );
> > success++;
> > System.out.println( "success: " + success + ",
> > read " + read + " bytes for: " + start.until( Instant.now(),
> > ChronoUnit.MILLIS ) + "ms" );
> >
> > } catch ( IOException e ) {
> > denial++;
> > System.err.println( "denial: " + denial + ", " +
> > e.getMessage() );
> > }}}}}
> >
> > Sample output:
> > success: 1, read 73 bytes for: 78ms
> > denial: 1, Connection reset
> > success: 2, read 73 bytes for: 78ms
> > denial: 2, Connection reset
> >
> > The bug is stable, and always reproducible.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> > For additional commands, e-mail: users-help@tomcat.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat 9 Nio2+OpenSSL problem (very likely a bug)
Posted by Mark Thomas <ma...@apache.org>.
On 13/03/2019 20:30, Igor T wrote:
> Prerequisites:
> OS: Windows Server 2012 R2
> Java: checked on both jdk1.8.0_162 jdk1.8.0_181
> Tomcat: windows x64 builds checked on 9.0.12, 9.0.16, 9.0.17-dev
9.0.17-dev at which point in time?
Have you tested the current 9.0.17 release candidate (see dev@ for details)
Mark
> Valid SSL certificates
> Content of file located at webapp/ROOT/1.txt: []
> Tomcat's connector settings:
> <Connector port="443"
> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
>
> sslImplementationName="org.apache.tomcat.util.net.openssl.OpenSSLImplementation"
> connectionTimeout="5000"
> SSLEnabled="true"
> scheme="https"
> secure="true"
> >
>
> This configuration leads to 50% of the traffic to be rejected with
> Connection resets. First socket connects and receives the service, but
> every second is resetted.
>
> Exactly this combination leads to connection resets:
> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
> sslImplementationName="org.apache.tomcat.util.net.openssl.OpenSSLImplementation"
>
> Configurations that work well without connection resets:
> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
> sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation"
> or
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> sslImplementationName="org.apache.tomcat.util.net.openssl.OpenSSLImplementation"
>
> Java code to reproduce the connection resets (works well with any
> other secure server):
> (there is no resets if a variable named FIX__gotoSleepAfterHandshake = true)
>
> public class CheckConnectionResets{
> static String host = "your-test-host";
>
> public static void main( String[] args ) throws
> InterruptedException, IOException{
>
> SSLSocketFactory factory =
> (SSLSocketFactory)SSLSocketFactory.getDefault();
> int nRuns = 4;
> int success = 0;
> int denial = 0;
>
> boolean FIX__gotoSleepAfterHandshake = false;
>
> for( int i = 0; i < nRuns; i++ ){
> try ( SSLSocket socket = (SSLSocket)factory.createSocket(
> host, 443 ) ){
>
> if( FIX__gotoSleepAfterHandshake ){
> socket.startHandshake();
> Thread.sleep( 500 );
> }
> try ( PrintWriter out = new PrintWriter( new
> BufferedWriter( new OutputStreamWriter( socket.getOutputStream() ) )
> );
> InputStream is = socket.getInputStream(); ){
>
> out.println( "GET /1.txt HTTP/1.1" );
> out.println( "Host: " + host );
> out.println( "Accept: */*" );
> out.println();
> out.flush();
>
> if( out.checkError() ){
> System.out.println( "SSLSocketClient:
> java.io.PrintWriter error" );
> }
>
> Instant start = Instant.now();
> /* read full response */
> byte[] buff = new byte[ 1024 ];
> int read = is.read( buff );
> success++;
> System.out.println( "success: " + success + ",
> read " + read + " bytes for: " + start.until( Instant.now(),
> ChronoUnit.MILLIS ) + "ms" );
>
> } catch ( IOException e ) {
> denial++;
> System.err.println( "denial: " + denial + ", " +
> e.getMessage() );
> }}}}}
>
> Sample output:
> success: 1, read 73 bytes for: 78ms
> denial: 1, Connection reset
> success: 2, read 73 bytes for: 78ms
> denial: 2, Connection reset
>
> The bug is stable, and always reproducible.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 9 Nio2+OpenSSL problem (very likely a bug)
Posted by John Palmer <jo...@gmail.com>.
I'm testing to see if this might be an issue on a new tomcat 8.5.38 upgrade
I'm doing (using NIO2 and OpenSSL) before I promote this to our Production
environment :)
(Windows Server 2008R2, Java (javaC.exe) version is 1.8.0_191)
.. after some missteps (had to add some imports to get it to compile, and
use the -Djavax.net.ssl.trustStore, ... .trustStoreType,
..trustStorePassword args when running...
4 successes. no connection resets.
import javax.net.ss.SSLSocket;
import javax.net.ss.SSLSocketFactory
import java.io.*;
import java.time.*;
import java.time.temporal.ChronoUnit;
On Wed, Mar 13, 2019 at 3:29 PM Igor T <ig...@gmail.com> wrote:
> Prerequisites:
> OS: Windows Server 2012 R2
> Java: checked on both jdk1.8.0_162 jdk1.8.0_181
> Tomcat: windows x64 builds checked on 9.0.12, 9.0.16, 9.0.17-dev
> Valid SSL certificates
> Content of file located at webapp/ROOT/1.txt: []
> Tomcat's connector settings:
> <Connector port="443"
>
> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
>
> sslImplementationName="org.apache.tomcat.util.net
> .openssl.OpenSSLImplementation"
> connectionTimeout="5000"
> SSLEnabled="true"
> scheme="https"
> secure="true"
> >
>
> This configuration leads to 50% of the traffic to be rejected with
> Connection resets. First socket connects and receives the service, but
> every second is resetted.
>
> Exactly this combination leads to connection resets:
> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
> sslImplementationName="org.apache.tomcat.util.net
> .openssl.OpenSSLImplementation"
>
> Configurations that work well without connection resets:
> protocol="org.apache.coyote.http11.Http11Nio2Protocol"
>
> sslImplementationName="org.apache.tomcat.util.net.jsse.JSSEImplementation"
> or
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> sslImplementationName="org.apache.tomcat.util.net
> .openssl.OpenSSLImplementation"
>
> Java code to reproduce the connection resets (works well with any
> other secure server):
> (there is no resets if a variable named FIX__gotoSleepAfterHandshake =
> true)
>
> public class CheckConnectionResets{
> static String host = "your-test-host";
>
> public static void main( String[] args ) throws
> InterruptedException, IOException{
>
> SSLSocketFactory factory =
> (SSLSocketFactory)SSLSocketFactory.getDefault();
> int nRuns = 4;
> int success = 0;
> int denial = 0;
>
> boolean FIX__gotoSleepAfterHandshake = false;
>
> for( int i = 0; i < nRuns; i++ ){
> try ( SSLSocket socket = (SSLSocket)factory.createSocket(
> host, 443 ) ){
>
> if( FIX__gotoSleepAfterHandshake ){
> socket.startHandshake();
> Thread.sleep( 500 );
> }
> try ( PrintWriter out = new PrintWriter( new
> BufferedWriter( new OutputStreamWriter( socket.getOutputStream() ) )
> );
> InputStream is = socket.getInputStream(); ){
>
> out.println( "GET /1.txt HTTP/1.1" );
> out.println( "Host: " + host );
> out.println( "Accept: */*" );
> out.println();
> out.flush();
>
> if( out.checkError() ){
> System.out.println( "SSLSocketClient:
> java.io.PrintWriter error" );
> }
>
> Instant start = Instant.now();
> /* read full response */
> byte[] buff = new byte[ 1024 ];
> int read = is.read( buff );
> success++;
> System.out.println( "success: " + success + ",
> read " + read + " bytes for: " + start.until( Instant.now(),
> ChronoUnit.MILLIS ) + "ms" );
>
> } catch ( IOException e ) {
> denial++;
> System.err.println( "denial: " + denial + ", " +
> e.getMessage() );
> }}}}}
>
> Sample output:
> success: 1, read 73 bytes for: 78ms
> denial: 1, Connection reset
> success: 2, read 73 bytes for: 78ms
> denial: 2, Connection reset
>
> The bug is stable, and always reproducible.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat 9 Nio2+OpenSSL problem (very likely a bug)
Posted by "info@flyingfischer.ch" <in...@flyingfischer.ch>.
Am 18.03.19 um 16:43 schrieb Igor T:
>> Since 9.0.12 and 16 do the same, I wouldn't look at that at all. Something
>> simple like this works in the general case, there must be something
>> specific here. So it's Windows, which some unspecified OpenSSL version.
>>
>> Rémy
> That's not right. After many tests I've found out that 9.0.12 build
> comes with [OpenSSL 1.0.2o 27 Mar 2018], while 9.0.16 comes with
> [OpenSSL 1.1.1a 20 Nov 2018].
> The problem was localized to OpenSSL 1.1.1a on Nio2.
> Also it became clear that establishing of connection takes more time
> with OpenSSL 1.1.1a on Nio.
> So it looks like OpenSSL 1.1.1a build is much less optimized and buggy.
>
> So the question is: how to change OpenSSL version that is shipped with
> the latest tomcat build back to 1.0.2?
> Any feedback appreciated.
>
I did have to reset some installations to Tomcat 8.5.35 to avoid using
TC native latest two versions on Linux. We have seen some bugfixes in
the lastes TC native, which did slightly improve the situation. But TC
still tends to crash on some machines (Linux).
Some of the changes made in native after 8.5.35 are unstable.
Markus
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Tomcat 9 Nio2+OpenSSL problem (very likely a bug)
Posted by Rémy Maucherat <re...@apache.org>.
On Mon, Mar 18, 2019 at 4:44 PM Igor T <ig...@gmail.com> wrote:
> success: 1, read 73 bytes for: 109ms
> success: 2, read 73 bytes for: 218ms
> success: 3, read 73 bytes for: 203ms
> success: 4, read 73 bytes for: 203ms
>
You are right there is something wrong here as well, especially since the
time gets worse on the second attempt. What is the actual roundtrip time
with your server ?
Rémy
Re: Tomcat 9 Nio2+OpenSSL problem (very likely a bug)
Posted by Rémy Maucherat <re...@apache.org>.
On Mon, Mar 18, 2019 at 4:44 PM Igor T <ig...@gmail.com> wrote:
> > Since 9.0.12 and 16 do the same, I wouldn't look at that at all.
> Something
> > simple like this works in the general case, there must be something
> > specific here. So it's Windows, which some unspecified OpenSSL version.
> >
> > Rémy
>
> That's not right. After many tests I've found out that 9.0.12 build
> comes with [OpenSSL 1.0.2o 27 Mar 2018], while 9.0.16 comes with
> [OpenSSL 1.1.1a 20 Nov 2018].
> The problem was localized to OpenSSL 1.1.1a on Nio2.
> Also it became clear that establishing of connection takes more time
> with OpenSSL 1.1.1a on Nio.
> So it looks like OpenSSL 1.1.1a build is much less optimized and buggy.
>
> So the question is: how to change OpenSSL version that is shipped with
> the latest tomcat build back to 1.0.2?
> Any feedback appreciated.
>
Ok, thanks for the information. The code has been updated for TLS 1.3 when
using OpenSSL 1.1.1, so there are significant changes in all components. We
will investigate.
Rémy
>
>
>
> Detailed test results:
>
> The problem exist:
> Apache Tomcat 9.0.16/Http11Nio2Protocol/OpenSSL 1.1.1a
> 18-Mar-2019 14:34:54.103 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
> APR based Apache Tomcat Native library [1.2.21] using APR version
> [1.6.5].
> 18-Mar-2019 14:34:54.103 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
> capabilities: IPv6 [true], sendfile [true], accept filters [false],
> random [true].
> 18-Mar-2019 14:34:54.103 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
> APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
> 18-Mar-2019 14:34:54.103 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
> successfully initialized [OpenSSL 1.1.1a 20 Nov 2018]
> 18-Mar-2019 14:34:54.306 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["http-nio2-80"]
> 18-Mar-2019 14:34:54.353 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["https-openssl-nio2-443"]
> 18-Mar-2019 14:34:54.947 INFO [main]
> org.apache.catalina.startup.Catalina.load Server initialization in
> [1,516] milliseconds
> 18-Mar-2019 14:34:54.994 INFO [main]
> org.apache.catalina.core.StandardService.startInternal Starting
> service [Catalina]
> 18-Mar-2019 14:34:54.994 INFO [main]
> org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
> engine: [Apache Tomcat/9.0.16]
> success: 1, read 73 bytes for: 125ms
> denial: 1, Connection reset
> success: 2, read 73 bytes for: 94ms
> denial: 2, Connection reset
> success: 3, read 73 bytes for: 93ms
> denial: 3, Connection reset
> success: 4, read 73 bytes for: 78ms
> denial: 4, Connection reset
> success: 5, read 73 bytes for: 94ms
> denial: 5, Connection reset
>
> Apache Tomcat 9.0.17/Http11Nio2Protocol/OpenSSL 1.1.1a
> 18-Mar-2019 14:41:46.708 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
> APR based Apache Tomcat Native library [1.2.21] using APR version
> [1.6.5].
> 18-Mar-2019 14:41:46.708 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
> capabilities: IPv6 [true], sendfile [true], accept filters [false],
> random [true].
> 18-Mar-2019 14:41:46.708 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
> APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
> 18-Mar-2019 14:41:46.724 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
> successfully initialized [OpenSSL 1.1.1a 20 Nov 2018]
> 18-Mar-2019 14:41:46.896 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["http-nio2-80"]
> 18-Mar-2019 14:41:46.912 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["https-openssl-nio2-443"]
> 18-Mar-2019 14:41:47.443 INFO [main]
> org.apache.catalina.startup.Catalina.load Server initialization in
> [1,335] milliseconds
> 18-Mar-2019 14:41:47.474 INFO [main]
> org.apache.catalina.core.StandardService.startInternal Starting
> service [Catalina]
> 18-Mar-2019 14:41:47.474 INFO [main]
> org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
> engine: [Apache Tomcat/9.0.17]
> success: 1, read 73 bytes for: 78ms
> denial: 1, Connection reset
> success: 2, read 73 bytes for: 93ms
> denial: 2, Connection reset
> success: 3, read 73 bytes for: 78ms
> denial: 3, Connection reset
> success: 4, read 73 bytes for: 94ms
> denial: 4, Connection reset
> success: 5, read 73 bytes for: 78ms
> denial: 5, Connection reset
>
>
> The problem does not exist:
> Apache Tomcat 9.0.12/Http11Nio2Protocol/OpenSSL 1.0.2o
> 18-Mar-2019 14:30:21.917 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
> APR based Apache Tomcat Native library [1.2.17] using APR version
> [1.6.3].
> 18-Mar-2019 14:30:21.917 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
> capabilities: IPv6 [true], sendfile [true], accept filters [false],
> random [true].
> 18-Mar-2019 14:30:21.917 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
> APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
> 18-Mar-2019 14:30:22.932 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
> successfully initialized [OpenSSL 1.0.2o 27 Mar 2018]
> 18-Mar-2019 14:30:23.135 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["http-nio2-80"]
> 18-Mar-2019 14:30:23.167 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["https-openssl-nio2-443"]
> 18-Mar-2019 14:30:24.059 INFO [main]
> org.apache.catalina.startup.Catalina.load Initialization processed in
> 2763 ms
> 18-Mar-2019 14:30:24.135 INFO [main]
> org.apache.catalina.core.StandardService.startInternal Starting
> service [Catalina]
> 18-Mar-2019 14:30:24.135 INFO [main]
> org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
> Engine: Apache Tomcat/9.0.12
> success: 1, read 73 bytes for: 125ms
> success: 2, read 73 bytes for: 78ms
> success: 3, read 73 bytes for: 78ms
> success: 4, read 73 bytes for: 78ms
> success: 5, read 73 bytes for: 93ms
> success: 6, read 73 bytes for: 78ms
> success: 7, read 73 bytes for: 94ms
> success: 8, read 73 bytes for: 78ms
> success: 9, read 73 bytes for: 94ms
> success: 10, read 73 bytes for: 94ms
>
> Apache Tomcat 9.0.12/Http11NioProtocol/OpenSSL 1.0.2o
> 18-Mar-2019 14:31:42.476 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
> APR based Apache Tomcat Native library [1.2.17] using APR version
> [1.6.3].
> 18-Mar-2019 14:31:42.476 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
> capabilities: IPv6 [true], sendfile [true], accept filters [false],
> random [true].
> 18-Mar-2019 14:31:42.476 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
> APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
> 18-Mar-2019 14:31:43.492 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
> successfully initialized [OpenSSL 1.0.2o 27 Mar 2018]
> 18-Mar-2019 14:31:43.726 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["http-nio2-80"]
> 18-Mar-2019 14:31:43.758 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["https-openssl-nio-443"]
> 18-Mar-2019 14:31:44.336 INFO [main]
> org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a
> shared selector for servlet write/read
> 18-Mar-2019 14:31:44.351 INFO [main]
> org.apache.catalina.startup.Catalina.load Initialization processed in
> 2483 ms
> 18-Mar-2019 14:31:44.383 INFO [main]
> org.apache.catalina.core.StandardService.startInternal Starting
> service [Catalina]
> 18-Mar-2019 14:31:44.383 INFO [main]
> org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
> Engine: Apache Tomcat/9.0.12
> success: 1, read 73 bytes for: 109ms
> success: 2, read 73 bytes for: 78ms
> success: 3, read 73 bytes for: 78ms
> success: 4, read 73 bytes for: 94ms
> success: 5, read 73 bytes for: 78ms
> success: 6, read 73 bytes for: 78ms
> success: 7, read 73 bytes for: 78ms
> success: 8, read 73 bytes for: 78ms
> success: 9, read 73 bytes for: 78ms
> success: 10, read 73 bytes for: 94ms
>
> Apache Tomcat 9.0.16/Http11NioProtocol/OpenSSL 1.1.1a
> 18-Mar-2019 14:37:12.000 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
> APR based Apache Tomcat Native library [1.2.21] using APR version
> [1.6.5].
> 18-Mar-2019 14:37:12.000 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
> capabilities: IPv6 [true], sendfile [true], accept filters [false],
> random [true].
> 18-Mar-2019 14:37:12.000 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
> APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
> 18-Mar-2019 14:37:12.000 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
> successfully initialized [OpenSSL 1.1.1a 20 Nov 2018]
> 18-Mar-2019 14:37:12.203 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["http-nio2-80"]
> 18-Mar-2019 14:37:12.219 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["https-openssl-nio-443"]
> 18-Mar-2019 14:37:12.797 INFO [main]
> org.apache.catalina.startup.Catalina.load Server initialization in
> [1,497] milliseconds
> 18-Mar-2019 14:37:12.844 INFO [main]
> org.apache.catalina.core.StandardService.startInternal Starting
> service [Catalina]
> 18-Mar-2019 14:37:12.844 INFO [main]
> org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
> engine: [Apache Tomcat/9.0.16]
> success: 1, read 73 bytes for: 78ms
> success: 2, read 73 bytes for: 218ms
> success: 3, read 73 bytes for: 203ms
> success: 4, read 73 bytes for: 203ms
> success: 5, read 73 bytes for: 203ms
> success: 6, read 73 bytes for: 203ms
> success: 7, read 73 bytes for: 203ms
> success: 8, read 73 bytes for: 202ms
> success: 9, read 73 bytes for: 187ms
> success: 10, read 73 bytes for: 187ms
>
> Apache Tomcat 9.0.17/Http11NioProtocol/OpenSSL 1.1.1a
> 18-Mar-2019 14:40:17.879 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
> APR based Apache Tomcat Native library [1.2.21] using APR version
> [1.6.5].
> 18-Mar-2019 14:40:17.879 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
> capabilities: IPv6 [true], sendfile [true], accept filters [false],
> random [true].
> 18-Mar-2019 14:40:17.879 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
> APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
> 18-Mar-2019 14:40:17.926 INFO [main]
> org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
> successfully initialized [OpenSSL 1.1.1a 20 Nov 2018]
> 18-Mar-2019 14:40:18.098 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["http-nio2-80"]
> 18-Mar-2019 14:40:18.129 INFO [main]
> org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
> ["https-openssl-nio-443"]
> 18-Mar-2019 14:40:18.645 INFO [main]
> org.apache.catalina.startup.Catalina.load Server initialization in
> [1,449] milliseconds
> 18-Mar-2019 14:40:18.692 INFO [main]
> org.apache.catalina.core.StandardService.startInternal Starting
> service [Catalina]
> 18-Mar-2019 14:40:18.692 INFO [main]
> org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
> engine: [Apache Tomcat/9.0.17]
> success: 1, read 73 bytes for: 109ms
> success: 2, read 73 bytes for: 218ms
> success: 3, read 73 bytes for: 203ms
> success: 4, read 73 bytes for: 203ms
> success: 5, read 73 bytes for: 187ms
> success: 6, read 73 bytes for: 203ms
> success: 7, read 73 bytes for: 187ms
> success: 8, read 73 bytes for: 187ms
> success: 9, read 73 bytes for: 203ms
> success: 10, read 73 bytes for: 203ms
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
Re: Tomcat 9 Nio2+OpenSSL problem (very likely a bug)
Posted by Igor T <ig...@gmail.com>.
> Since 9.0.12 and 16 do the same, I wouldn't look at that at all. Something
> simple like this works in the general case, there must be something
> specific here. So it's Windows, which some unspecified OpenSSL version.
>
> Rémy
That's not right. After many tests I've found out that 9.0.12 build
comes with [OpenSSL 1.0.2o 27 Mar 2018], while 9.0.16 comes with
[OpenSSL 1.1.1a 20 Nov 2018].
The problem was localized to OpenSSL 1.1.1a on Nio2.
Also it became clear that establishing of connection takes more time
with OpenSSL 1.1.1a on Nio.
So it looks like OpenSSL 1.1.1a build is much less optimized and buggy.
So the question is: how to change OpenSSL version that is shipped with
the latest tomcat build back to 1.0.2?
Any feedback appreciated.
Detailed test results:
The problem exist:
Apache Tomcat 9.0.16/Http11Nio2Protocol/OpenSSL 1.1.1a
18-Mar-2019 14:34:54.103 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
APR based Apache Tomcat Native library [1.2.21] using APR version
[1.6.5].
18-Mar-2019 14:34:54.103 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
capabilities: IPv6 [true], sendfile [true], accept filters [false],
random [true].
18-Mar-2019 14:34:54.103 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
18-Mar-2019 14:34:54.103 INFO [main]
org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
successfully initialized [OpenSSL 1.1.1a 20 Nov 2018]
18-Mar-2019 14:34:54.306 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["http-nio2-80"]
18-Mar-2019 14:34:54.353 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["https-openssl-nio2-443"]
18-Mar-2019 14:34:54.947 INFO [main]
org.apache.catalina.startup.Catalina.load Server initialization in
[1,516] milliseconds
18-Mar-2019 14:34:54.994 INFO [main]
org.apache.catalina.core.StandardService.startInternal Starting
service [Catalina]
18-Mar-2019 14:34:54.994 INFO [main]
org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
engine: [Apache Tomcat/9.0.16]
success: 1, read 73 bytes for: 125ms
denial: 1, Connection reset
success: 2, read 73 bytes for: 94ms
denial: 2, Connection reset
success: 3, read 73 bytes for: 93ms
denial: 3, Connection reset
success: 4, read 73 bytes for: 78ms
denial: 4, Connection reset
success: 5, read 73 bytes for: 94ms
denial: 5, Connection reset
Apache Tomcat 9.0.17/Http11Nio2Protocol/OpenSSL 1.1.1a
18-Mar-2019 14:41:46.708 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
APR based Apache Tomcat Native library [1.2.21] using APR version
[1.6.5].
18-Mar-2019 14:41:46.708 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
capabilities: IPv6 [true], sendfile [true], accept filters [false],
random [true].
18-Mar-2019 14:41:46.708 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
18-Mar-2019 14:41:46.724 INFO [main]
org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
successfully initialized [OpenSSL 1.1.1a 20 Nov 2018]
18-Mar-2019 14:41:46.896 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["http-nio2-80"]
18-Mar-2019 14:41:46.912 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["https-openssl-nio2-443"]
18-Mar-2019 14:41:47.443 INFO [main]
org.apache.catalina.startup.Catalina.load Server initialization in
[1,335] milliseconds
18-Mar-2019 14:41:47.474 INFO [main]
org.apache.catalina.core.StandardService.startInternal Starting
service [Catalina]
18-Mar-2019 14:41:47.474 INFO [main]
org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
engine: [Apache Tomcat/9.0.17]
success: 1, read 73 bytes for: 78ms
denial: 1, Connection reset
success: 2, read 73 bytes for: 93ms
denial: 2, Connection reset
success: 3, read 73 bytes for: 78ms
denial: 3, Connection reset
success: 4, read 73 bytes for: 94ms
denial: 4, Connection reset
success: 5, read 73 bytes for: 78ms
denial: 5, Connection reset
The problem does not exist:
Apache Tomcat 9.0.12/Http11Nio2Protocol/OpenSSL 1.0.2o
18-Mar-2019 14:30:21.917 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
APR based Apache Tomcat Native library [1.2.17] using APR version
[1.6.3].
18-Mar-2019 14:30:21.917 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
capabilities: IPv6 [true], sendfile [true], accept filters [false],
random [true].
18-Mar-2019 14:30:21.917 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
18-Mar-2019 14:30:22.932 INFO [main]
org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
successfully initialized [OpenSSL 1.0.2o 27 Mar 2018]
18-Mar-2019 14:30:23.135 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["http-nio2-80"]
18-Mar-2019 14:30:23.167 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["https-openssl-nio2-443"]
18-Mar-2019 14:30:24.059 INFO [main]
org.apache.catalina.startup.Catalina.load Initialization processed in
2763 ms
18-Mar-2019 14:30:24.135 INFO [main]
org.apache.catalina.core.StandardService.startInternal Starting
service [Catalina]
18-Mar-2019 14:30:24.135 INFO [main]
org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
Engine: Apache Tomcat/9.0.12
success: 1, read 73 bytes for: 125ms
success: 2, read 73 bytes for: 78ms
success: 3, read 73 bytes for: 78ms
success: 4, read 73 bytes for: 78ms
success: 5, read 73 bytes for: 93ms
success: 6, read 73 bytes for: 78ms
success: 7, read 73 bytes for: 94ms
success: 8, read 73 bytes for: 78ms
success: 9, read 73 bytes for: 94ms
success: 10, read 73 bytes for: 94ms
Apache Tomcat 9.0.12/Http11NioProtocol/OpenSSL 1.0.2o
18-Mar-2019 14:31:42.476 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
APR based Apache Tomcat Native library [1.2.17] using APR version
[1.6.3].
18-Mar-2019 14:31:42.476 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
capabilities: IPv6 [true], sendfile [true], accept filters [false],
random [true].
18-Mar-2019 14:31:42.476 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
18-Mar-2019 14:31:43.492 INFO [main]
org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
successfully initialized [OpenSSL 1.0.2o 27 Mar 2018]
18-Mar-2019 14:31:43.726 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["http-nio2-80"]
18-Mar-2019 14:31:43.758 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["https-openssl-nio-443"]
18-Mar-2019 14:31:44.336 INFO [main]
org.apache.tomcat.util.net.NioSelectorPool.getSharedSelector Using a
shared selector for servlet write/read
18-Mar-2019 14:31:44.351 INFO [main]
org.apache.catalina.startup.Catalina.load Initialization processed in
2483 ms
18-Mar-2019 14:31:44.383 INFO [main]
org.apache.catalina.core.StandardService.startInternal Starting
service [Catalina]
18-Mar-2019 14:31:44.383 INFO [main]
org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
Engine: Apache Tomcat/9.0.12
success: 1, read 73 bytes for: 109ms
success: 2, read 73 bytes for: 78ms
success: 3, read 73 bytes for: 78ms
success: 4, read 73 bytes for: 94ms
success: 5, read 73 bytes for: 78ms
success: 6, read 73 bytes for: 78ms
success: 7, read 73 bytes for: 78ms
success: 8, read 73 bytes for: 78ms
success: 9, read 73 bytes for: 78ms
success: 10, read 73 bytes for: 94ms
Apache Tomcat 9.0.16/Http11NioProtocol/OpenSSL 1.1.1a
18-Mar-2019 14:37:12.000 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
APR based Apache Tomcat Native library [1.2.21] using APR version
[1.6.5].
18-Mar-2019 14:37:12.000 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
capabilities: IPv6 [true], sendfile [true], accept filters [false],
random [true].
18-Mar-2019 14:37:12.000 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
18-Mar-2019 14:37:12.000 INFO [main]
org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
successfully initialized [OpenSSL 1.1.1a 20 Nov 2018]
18-Mar-2019 14:37:12.203 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["http-nio2-80"]
18-Mar-2019 14:37:12.219 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["https-openssl-nio-443"]
18-Mar-2019 14:37:12.797 INFO [main]
org.apache.catalina.startup.Catalina.load Server initialization in
[1,497] milliseconds
18-Mar-2019 14:37:12.844 INFO [main]
org.apache.catalina.core.StandardService.startInternal Starting
service [Catalina]
18-Mar-2019 14:37:12.844 INFO [main]
org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
engine: [Apache Tomcat/9.0.16]
success: 1, read 73 bytes for: 78ms
success: 2, read 73 bytes for: 218ms
success: 3, read 73 bytes for: 203ms
success: 4, read 73 bytes for: 203ms
success: 5, read 73 bytes for: 203ms
success: 6, read 73 bytes for: 203ms
success: 7, read 73 bytes for: 203ms
success: 8, read 73 bytes for: 202ms
success: 9, read 73 bytes for: 187ms
success: 10, read 73 bytes for: 187ms
Apache Tomcat 9.0.17/Http11NioProtocol/OpenSSL 1.1.1a
18-Mar-2019 14:40:17.879 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent Loaded
APR based Apache Tomcat Native library [1.2.21] using APR version
[1.6.5].
18-Mar-2019 14:40:17.879 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent APR
capabilities: IPv6 [true], sendfile [true], accept filters [false],
random [true].
18-Mar-2019 14:40:17.879 INFO [main]
org.apache.catalina.core.AprLifecycleListener.lifecycleEvent
APR/OpenSSL configuration: useAprConnector [false], useOpenSSL [true]
18-Mar-2019 14:40:17.926 INFO [main]
org.apache.catalina.core.AprLifecycleListener.initializeSSL OpenSSL
successfully initialized [OpenSSL 1.1.1a 20 Nov 2018]
18-Mar-2019 14:40:18.098 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["http-nio2-80"]
18-Mar-2019 14:40:18.129 INFO [main]
org.apache.coyote.AbstractProtocol.init Initializing ProtocolHandler
["https-openssl-nio-443"]
18-Mar-2019 14:40:18.645 INFO [main]
org.apache.catalina.startup.Catalina.load Server initialization in
[1,449] milliseconds
18-Mar-2019 14:40:18.692 INFO [main]
org.apache.catalina.core.StandardService.startInternal Starting
service [Catalina]
18-Mar-2019 14:40:18.692 INFO [main]
org.apache.catalina.core.StandardEngine.startInternal Starting Servlet
engine: [Apache Tomcat/9.0.17]
success: 1, read 73 bytes for: 109ms
success: 2, read 73 bytes for: 218ms
success: 3, read 73 bytes for: 203ms
success: 4, read 73 bytes for: 203ms
success: 5, read 73 bytes for: 187ms
success: 6, read 73 bytes for: 203ms
success: 7, read 73 bytes for: 187ms
success: 8, read 73 bytes for: 187ms
success: 9, read 73 bytes for: 203ms
success: 10, read 73 bytes for: 203ms
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org