You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Richard Tearle <ri...@northgateps.com> on 2018/03/20 07:52:47 UTC
Connection closed error and certificateVerification="required"
Hello
We have 4 applications built on the same architecture with a web UI
and camel based ESB running in separate Tomcat's, using REST/XML to
communicate between the two. This is all deployed within separate
Docker containers but on the same VM (at least for test), either on
Centos Linux or Oracle Linux. This all works on Tomcat 8.0.x. We've
been upgrading to Tomcat 8.5.x since November last year, but this has
been hampered by what looks to be random connection closed errors when
our UI communicates to the ESB. We have a series of Selenium based
autotests which will fail in different places, but with the same
error:
09:30:46.057 [main] ERROR test.ui.Main - Catching
java.util.concurrent.ExecutionException:
org.apache.http.ConnectionClosedException: Connection closed
at org.apache.http.concurrent.BasicFuture.getResult(BasicFuture.java:70)
~[test-ui.jar:?]
at org.apache.http.concurrent.BasicFuture.get(BasicFuture.java:80)
~[test-ui.jar:?]
at org.apache.http.impl.nio.client.FutureWrapper.get(FutureWrapper.java:70)
~[test-ui.jar:?]
at org.springframework.util.concurrent.FutureAdapter.get(FutureAdapter.java:81)
~[test-ui.jar:?]
at org.springframework.util.concurrent.FutureAdapter.get(FutureAdapter.java:81)
~[test-ui.jar:?]
at test.ui.Main.post(Main.java:182) ~[test-ui.jar:?]
at test.ui.Main.callServer(Main.java:148) [test-ui.jar:?]
at test.ui.Main.main(Main.java:109) [test-ui.jar:?]
Caused by: org.apache.http.ConnectionClosedException: Connection closed
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.endOfInput(HttpAsyncRequestExecutor.java:345)
~[test-ui.jar:?]
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:261)
~[test-ui.jar:?]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
~[test-ui.jar:?]
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
~[test-ui.jar:?]
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:121)
~[test-ui.jar:?]
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
~[test-ui.jar:?]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
~[test-ui.jar:?]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
~[test-ui.jar:?]
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
~[test-ui.jar:?]
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
~[test-ui.jar:?]
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
~[test-ui.jar:?]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_162]
We've seen this on all of the versions of Tomcat 8.5 we've tried from
8.5.23 onwards - and I've just tried Tomcat 9.0.6 with the same error.
We're using Oracle Java version 1.8u162.
Development have looked at this off and on over the last couple of
months, but without success, so I was asked to reproduce this with a
simple test case (I'm devops usually, but with a development
background). The client stack uses a
PoolingNHttpClientConnectionManager, CloseableHttpAsyncClient and
AsyncRestTemplate to communicate to the ESB. My simple test case
replaces the web UI with a simple jar based application, but it still
replicates the issue. The ESB is a simple Camel route that adds two
supplied numbers, waits for a time (supplied by the UI), and returns
the result. The UI repeatedly calls the ESB (on a single thread, with
no overlapping calls), with two random numbers to be summed, and a set
amount of time to wait before the ESB returns. Between each call to
the ESB there's a set amount of time to wait before the next call. It
seems that when the ESB wait is >= 5secs the time take to fail is
greatly decreased.
The connection between the UI and the ESB is TLSv1.2 with the ESB
configured with certificateVerification="required". The connector
configuration with the ESB's server.xml is:
<Connector port="${servers.port}"
protocol="org.apache.coyote.http11.Http11NioProtocol"
maxThreads="150" SSLEnabled="true" scheme="https"
secure="true" server="Apache" maxPostSize="100000">
<SSLHostConfig certificateVerification="required" protocols="TLSv1.2"
caCertificateFile="/usr/local/tomcat/ssl/ca-cert.pem" >
<Certificate
certificateFile="/usr/local/tomcat/ssl/server-cert.pem"
certificateKeyFile="/usr/local/tomcat/ssl/server-key.pem"
certificateKeyPassword="${keystore.pass}">
</Certificate>
</SSLHostConfig>
</Connector>
We are using PEM encoded certificates, and using Tomcat Native Library
version 1.2.16. If I take out certificateVerification="required", we
don't get the error (with my simple test case running for an hour)
I've tried using PKCS12 and JKS certificates, but I can't get Tomcat
8.5.x to work with our self signed trust store (we get the
"trustAnchors parameter must be non-empty" errors), although Tomcat
8.0.x seems happy enough with them. So I can't tell if it's a
difference between using OpenSSL and JSSE.
Enabling SSL debug traces on the client shows that in the error case,
after the initial successful handshaking between the client and the
server, the ESB shuts down the connection just after the client has
sent the data:
*** CertificateVerify
Signature Algorithm SHA512withRSA
I/O dispatcher 1, WRITE: TLSv1.2 Handshake, length = 520
I/O dispatcher 1, WRITE: TLSv1.2 Change Cipher Spec, length = 1
*** Finished
verify_data: { 197, 39, 73, 181, 14, 87, 139, 81, 247, 181, 32, 17 }
***
I/O dispatcher 1, WRITE: TLSv1.2 Handshake, length = 40
I/O dispatcher 1, READ: TLSv1.2 Change Cipher Spec, length = 1
I/O dispatcher 1, READ: TLSv1.2 Handshake, length = 40
*** Finished
verify_data: { 84, 164, 177, 160, 235, 23, 31, 20, 252, 149, 214, 245 }
***
%% Cached client session: [Session-101,
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384]
I/O dispatcher 1, WRITE: TLSv1.2 Application Data, length = 391
I/O dispatcher 1, READ: TLSv1.2 Alert, length = 26
I/O dispatcher 1, RECV TLSv1.2 ALERT: warning, close_notify
I/O dispatcher 1, closeInboundInternal()
I/O dispatcher 1, closeOutboundInternal()
I/O dispatcher 1, SEND TLSv1.2 ALERT: warning, description = close_notify
I/O dispatcher 1, WRITE: TLSv1.2 Alert, length = 26
The failed call doesn't make it into the ESB application logs, and I
can't see any errors in any of the ESB logs.
I appreciate that I haven't included all the information required to
help resolve this issue; so please tell me what extra information is
required.
--
Richard
--
This email is sent on behalf of Northgate Public Services (UK) Limited and
its associated companies including Rave Technologies (India) Pvt Limited
(together "Northgate Public Services") and is strictly confidential and
intended solely for the addressee(s).
If you are not the intended recipient of this email you must: (i) not
disclose, copy or distribute its contents to any other person nor use its
contents in any way or you may be acting unlawfully; (ii) contact
Northgate Public Services immediately on +44(0)1442 768445 quoting the name
of the sender and the addressee then delete it from your system.
Northgate Public Services has taken reasonable precautions to ensure that
no viruses are contained in this email, but does not accept any
responsibility once this email has been transmitted. You should scan
attachments (if any) for viruses.
Northgate Public Services (UK) Limited, registered in England and Wales
under number 00968498 with a registered address of Peoplebuilding 2,
Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2
4NW. Rave Technologies (India) Pvt Limited, registered in India under
number 117068 with a registered address of 2nd Floor, Ballard House, Adi
Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 400001.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Richard Tearle <ri...@northgateps.com>.
On 20 March 2018 at 14:49, Richard Tearle
<ri...@northgateps.com> wrote:
> Hello
>
> On 20 March 2018 at 11:29, Mark Thomas <ma...@apache.org> wrote:
>>
>> <snip/>
>>
>> There are rather too many factors at play here. It would be good to try
>> and eliminate some of them.
>>
>> What are the known working 8.0.x versions?
>>
Sorry I missed these in my previous reply:
8.0.50, 8.0.48 to 8.0.46, 8.0.41. 8.0.33, 8.0.26
--
Richard
--
This email is sent on behalf of Northgate Public Services (UK) Limited and
its associated companies including Rave Technologies (India) Pvt Limited
(together "Northgate Public Services") and is strictly confidential and
intended solely for the addressee(s).
If you are not the intended recipient of this email you must: (i) not
disclose, copy or distribute its contents to any other person nor use its
contents in any way or you may be acting unlawfully; (ii) contact
Northgate Public Services immediately on +44(0)1442 768445 quoting the name
of the sender and the addressee then delete it from your system.
Northgate Public Services has taken reasonable precautions to ensure that
no viruses are contained in this email, but does not accept any
responsibility once this email has been transmitted. You should scan
attachments (if any) for viruses.
Northgate Public Services (UK) Limited, registered in England and Wales
under number 00968498 with a registered address of Peoplebuilding 2,
Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2
4NW. Rave Technologies (India) Pvt Limited, registered in India under
number 117068 with a registered address of 2nd Floor, Ballard House, Adi
Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 400001.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by lo...@kreuser.name.
Mark,
> Am 19.04.2018 um 20:58 schrieb Mark Thomas <ma...@apache.org>:
>
> On 19/04/18 16:50, Peter@Kreuser-Online wrote:
>
> <snip/>
>
>> Do you mind to share more about the root cause? I’ve followed this mail communication from the start and am curious.
>
> Sure.
>
> Tomcat was configured to require CLIENT-CERT auth and the main client
> was configured to use this. Occasionally, the main client would see
> connection problems when using 8.5.x or later with NIO and the OpenSSL
> TLS implementation.
>
> There was a second client the performed health checks on the server.
> This client did not use a certificate. The requests it made always
> failed but did so with a predictable error message that the health check
> looked for.
>
> OpenSSL error states are stored per thread.
>
> Each Java thread is mapped 1-to-1 to an OS thread.
>
> The sequence of events that cause the problem was as follows:
>
> - Health check ran
> - TLS connection failed because no client certificate was provided
> - OpenSSL set an error state that - depending on the timing of the
> socket closure - was not always cleaned up
> - Standard request was received and was handled by the thread that
> previously experienced the error
> - Because the error had not been cleaned up, this new connection thought
> the error was meant for it and closed the connection
>
> The fix was to ensure that, whenever the Tomcat code made a call to
> OpenSSL that looked like this:
> - Do something via the OpenSSL API
> - Check the OpenSSL error state
>
> the code was changed so it looked like this:
> - Clear the OpenSSL error state
> - Do something via the OpenSSL API
> - Check the OpenSSL error state
>
> I also added a TODO for the arguably more complete fix which is to check
> the OpenSSL error state after every call to the OpenSSL API.
>
>> Let me tell you that your endurance on all the tricky issues here is admirable!
>>
>> Thank you for that!
>
> Thank you.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Much appreciated. That was a mean one.
Best regards.
Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Mark Thomas <ma...@apache.org>.
On 19/04/18 16:50, Peter@Kreuser-Online wrote:
<snip/>
> Do you mind to share more about the root cause? I’ve followed this mail communication from the start and am curious.
Sure.
Tomcat was configured to require CLIENT-CERT auth and the main client
was configured to use this. Occasionally, the main client would see
connection problems when using 8.5.x or later with NIO and the OpenSSL
TLS implementation.
There was a second client the performed health checks on the server.
This client did not use a certificate. The requests it made always
failed but did so with a predictable error message that the health check
looked for.
OpenSSL error states are stored per thread.
Each Java thread is mapped 1-to-1 to an OS thread.
The sequence of events that cause the problem was as follows:
- Health check ran
- TLS connection failed because no client certificate was provided
- OpenSSL set an error state that - depending on the timing of the
socket closure - was not always cleaned up
- Standard request was received and was handled by the thread that
previously experienced the error
- Because the error had not been cleaned up, this new connection thought
the error was meant for it and closed the connection
The fix was to ensure that, whenever the Tomcat code made a call to
OpenSSL that looked like this:
- Do something via the OpenSSL API
- Check the OpenSSL error state
the code was changed so it looked like this:
- Clear the OpenSSL error state
- Do something via the OpenSSL API
- Check the OpenSSL error state
I also added a TODO for the arguably more complete fix which is to check
the OpenSSL error state after every call to the OpenSSL API.
> Let me tell you that your endurance on all the tricky issues here is admirable!
>
> Thank you for that!
Thank you.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by "Peter@Kreuser-Online" <lo...@kreuser.name>.
Mark,
>> Am 18.04.2018 um 11:55 schrieb Mark Thomas <ma...@apache.org>:
>>
>> On 18/04/18 10:36, Richard Tearle wrote:
>> On 17 April 2018 at 16:45, Richard Tearle
>> <ri...@northgateps.com> wrote:
>>> On 17 April 2018 at 14:54, Mark Thomas <ma...@apache.org> wrote:
>>>>> On 17/04/18 11:36, Mark Thomas wrote:
>>>>> On 17/04/18 10:14, Richard Tearle wrote:
>>>>
>>>> <snip/>
>>>>
>>>>> Now all we need to to do is to figure out how to fix this. With the
>>>>> understanding of what is (probably) going wrong, the problem can be
>>>>> produced with a clean build and the certs we use for unit tests which
>>>>> makes things a lot easier. I hope to make progress on this today.
>>>>
>>>> I believe this is fixed in trunk for both 9.0.x and 8.5.x.
>>>>
>>>> Are you able to build either of those from source and test? If not, I
>>>> can provide a snapshot build for you to test with.
>>>>
>>>> Cheers,
>>>>
>>>> Mark
>>>
>>> I've built from source, re-enabled the health check, and so
>>> far so good - it's been running an hour without an error.
>>>
>>> Once this run has finished, I'll run it overnight, and let you
>>> know tomorrow morning.
>>>
>>> Many thanks for your help on this.
>>>
>>> --
>>> Richard
>>
>> Just a quick follow up - I've run our test for 8 hours, without the
>> connection closed error.
>
> Excellent. That is really good news.
>
>> Again, many thanks.
>
> No problem. Happy to help. Thanks for your assistance with this issue.
> Your test case and debug logs were invaluable. I couldn't have fixed
> this without them.
>
> Mark
>
Do you mind to share more about the root cause? I’ve followed this mail communication from the start and am curious.
Let me tell you that your endurance on all the tricky issues here is admirable!
Thank you for that!
Peter
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Mark Thomas <ma...@apache.org>.
On 18/04/18 10:36, Richard Tearle wrote:
> On 17 April 2018 at 16:45, Richard Tearle
> <ri...@northgateps.com> wrote:
>> On 17 April 2018 at 14:54, Mark Thomas <ma...@apache.org> wrote:
>>> On 17/04/18 11:36, Mark Thomas wrote:
>>>> On 17/04/18 10:14, Richard Tearle wrote:
>>>
>>> <snip/>
>>>
>>>> Now all we need to to do is to figure out how to fix this. With the
>>>> understanding of what is (probably) going wrong, the problem can be
>>>> produced with a clean build and the certs we use for unit tests which
>>>> makes things a lot easier. I hope to make progress on this today.
>>>
>>> I believe this is fixed in trunk for both 9.0.x and 8.5.x.
>>>
>>> Are you able to build either of those from source and test? If not, I
>>> can provide a snapshot build for you to test with.
>>>
>>> Cheers,
>>>
>>> Mark
>>>
>>
>> I've built from source, re-enabled the health check, and so
>> far so good - it's been running an hour without an error.
>>
>> Once this run has finished, I'll run it overnight, and let you
>> know tomorrow morning.
>>
>> Many thanks for your help on this.
>>
>> --
>> Richard
>
> Just a quick follow up - I've run our test for 8 hours, without the
> connection closed error.
Excellent. That is really good news.
> Again, many thanks.
No problem. Happy to help. Thanks for your assistance with this issue.
Your test case and debug logs were invaluable. I couldn't have fixed
this without them.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Richard Tearle <ri...@northgateps.com>.
On 17 April 2018 at 16:45, Richard Tearle
<ri...@northgateps.com> wrote:
> On 17 April 2018 at 14:54, Mark Thomas <ma...@apache.org> wrote:
>> On 17/04/18 11:36, Mark Thomas wrote:
>>> On 17/04/18 10:14, Richard Tearle wrote:
>>
>> <snip/>
>>
>>> Now all we need to to do is to figure out how to fix this. With the
>>> understanding of what is (probably) going wrong, the problem can be
>>> produced with a clean build and the certs we use for unit tests which
>>> makes things a lot easier. I hope to make progress on this today.
>>
>> I believe this is fixed in trunk for both 9.0.x and 8.5.x.
>>
>> Are you able to build either of those from source and test? If not, I
>> can provide a snapshot build for you to test with.
>>
>> Cheers,
>>
>> Mark
>>
>
> I've built from source, re-enabled the health check, and so
> far so good - it's been running an hour without an error.
>
> Once this run has finished, I'll run it overnight, and let you
> know tomorrow morning.
>
> Many thanks for your help on this.
>
> --
> Richard
Just a quick follow up - I've run our test for 8 hours, without the
connection closed error.
Again, many thanks.
--
Richard
--
This email is sent on behalf of Northgate Public Services (UK) Limited and
its associated companies including Rave Technologies (India) Pvt Limited
(together "Northgate Public Services") and is strictly confidential and
intended solely for the addressee(s).
If you are not the intended
recipient of this email you must: (i) not disclose, copy or distribute its
contents to any other person nor use its contents in any way or you may be
acting unlawfully; (ii) contact Northgate Public Services immediately on
+44(0)1442 768445 quoting the name of the sender and the addressee then
delete it from your system.
Northgate Public Services has taken reasonable
precautions to ensure that no viruses are contained in this email, but does
not accept any responsibility once this email has been transmitted. You
should scan attachments (if any) for viruses.
Northgate Public Services
(UK) Limited, registered in England and Wales under number 00968498 with a
registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands
Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India)
Pvt Limited, registered in India under number 117068 with a registered
address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate,
Mumbai, Maharashtra, India, 400001.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Richard Tearle <ri...@northgateps.com>.
On 17 April 2018 at 14:54, Mark Thomas <ma...@apache.org> wrote:
> On 17/04/18 11:36, Mark Thomas wrote:
>> On 17/04/18 10:14, Richard Tearle wrote:
>
> <snip/>
>
>> Now all we need to to do is to figure out how to fix this. With the
>> understanding of what is (probably) going wrong, the problem can be
>> produced with a clean build and the certs we use for unit tests which
>> makes things a lot easier. I hope to make progress on this today.
>
> I believe this is fixed in trunk for both 9.0.x and 8.5.x.
>
> Are you able to build either of those from source and test? If not, I
> can provide a snapshot build for you to test with.
>
> Cheers,
>
> Mark
>
I've built from source, re-enabled the health check, and so
far so good - it's been running an hour without an error.
Once this run has finished, I'll run it overnight, and let you
know tomorrow morning.
Many thanks for your help on this.
--
Richard
--
This email is sent on behalf of Northgate Public Services (UK) Limited and
its associated companies including Rave Technologies (India) Pvt Limited
(together "Northgate Public Services") and is strictly confidential and
intended solely for the addressee(s).
If you are not the intended
recipient of this email you must: (i) not disclose, copy or distribute its
contents to any other person nor use its contents in any way or you may be
acting unlawfully; (ii) contact Northgate Public Services immediately on
+44(0)1442 768445 quoting the name of the sender and the addressee then
delete it from your system.
Northgate Public Services has taken reasonable
precautions to ensure that no viruses are contained in this email, but does
not accept any responsibility once this email has been transmitted. You
should scan attachments (if any) for viruses.
Northgate Public Services
(UK) Limited, registered in England and Wales under number 00968498 with a
registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands
Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India)
Pvt Limited, registered in India under number 117068 with a registered
address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate,
Mumbai, Maharashtra, India, 400001.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Mark Thomas <ma...@apache.org>.
On 17/04/18 11:36, Mark Thomas wrote:
> On 17/04/18 10:14, Richard Tearle wrote:
<snip/>
>> I've also disabled the health check on ESB container, and my tests
>> ran through for an hour, without a connection closed error.
>
> That is good news. That is a strong indicator that we are on the right
> track. It also explains why I could not reproduce the problem with your
> test case. And finally, it is another example of the debug logging added
> to the I/O layer proving worth while.
>
> Now all we need to to do is to figure out how to fix this. With the
> understanding of what is (probably) going wrong, the problem can be
> produced with a clean build and the certs we use for unit tests which
> makes things a lot easier. I hope to make progress on this today.
I believe this is fixed in trunk for both 9.0.x and 8.5.x.
Are you able to build either of those from source and test? If not, I
can provide a snapshot build for you to test with.
Cheers,
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Mark Thomas <ma...@apache.org>.
On 17/04/18 10:14, Richard Tearle wrote:
> On 16 April 2018 at 22:04, Mark Thomas <ma...@apache.org> wrote:
<snip/>
>> I've started to look at them. I don't have any firm conclusions yet. I
>> have noticed that the problem occurs after a connection is made to the
>> service from localhost rather than the remote IP that is making the
>> other requests. The localhost client does not present a certificate.
>>
>> My working theory (so chances are it is completely wrong) is that the
>> missing certificate in the request from localhost puts the OpenSSL
>> engine into an error state that is not correctly handled by Tomcat
>> causing the subsequent request to fail.
>>
>> I've also noticed that the debug level log message consistently report 0
>> bytes being read which looks wrong but is probably a separate (minor) issue.
The above message are correct but misleading. I'm planning to add
additional debug logging which should clarify things.
<snip/>
> Ah that rings a bell.
>
> Our containers have a simple health check, simply does
>
> curl --connect-timeout 5 --max-time 20 -k -s -S --stderr -\
> https://localhost:${TOMCATS_PORT}/ |\
> grep -q "NSS: client certificate not found" || exit 1
>
> just to make sure the ESB is responding, with something we expect.
> These are set to run at an interval of every 2m30s. The full parameters
> in the docker-compose[1] file are:
>
> healthcheck:
> test: ["CMD", "/usr/local/bin/healthcheck.sh"]
> interval: 2m30s
> timeout: 10s
> retries: 3
> start_period: 20s
>
> I've also disabled the health check on ESB container, and my tests
> ran through for an hour, without a connection closed error.
That is good news. That is a strong indicator that we are on the right
track. It also explains why I could not reproduce the problem with your
test case. And finally, it is another example of the debug logging added
to the I/O layer proving worth while.
Now all we need to to do is to figure out how to fix this. With the
understanding of what is (probably) going wrong, the problem can be
produced with a clean build and the certs we use for unit tests which
makes things a lot easier. I hope to make progress on this today.
Mark
>
> [1] https://docs.docker.com/compose/compose-file/#healthcheck
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Richard Tearle <ri...@northgateps.com>.
On 16 April 2018 at 22:04, Mark Thomas <ma...@apache.org> wrote:
> On 11/04/18 09:22, Richard Tearle wrote:
>
> <snip/>
>
>> I've built tomcat from source using the link you provided, and rebuilt the
>> containers with this tomcat, and can still reproduce the issue. I've uploaded
>> the logs (30s before the connection closed error), to dropbox:
>>
>> https://www.dropbox.com/s/qe50jbd196krtyo/logs-10-04-17.zip?dl=0
>
> Thanks for these.
>
> I've started to look at them. I don't have any firm conclusions yet. I
> have noticed that the problem occurs after a connection is made to the
> service from localhost rather than the remote IP that is making the
> other requests. The localhost client does not present a certificate.
>
> My working theory (so chances are it is completely wrong) is that the
> missing certificate in the request from localhost puts the OpenSSL
> engine into an error state that is not correctly handled by Tomcat
> causing the subsequent request to fail.
>
> I've also noticed that the debug level log message consistently report 0
> bytes being read which looks wrong but is probably a separate (minor) issue.
>
> Investigations continue.
>
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
Ah that rings a bell.
Our containers have a simple health check, simply does
curl --connect-timeout 5 --max-time 20 -k -s -S --stderr -\
https://localhost:${TOMCATS_PORT}/ |\
grep -q "NSS: client certificate not found" || exit 1
just to make sure the ESB is responding, with something we expect.
These are set to run at an interval of every 2m30s. The full parameters
in the docker-compose[1] file are:
healthcheck:
test: ["CMD", "/usr/local/bin/healthcheck.sh"]
interval: 2m30s
timeout: 10s
retries: 3
start_period: 20s
I've also disabled the health check on ESB container, and my tests
ran through for an hour, without a connection closed error.
[1] https://docs.docker.com/compose/compose-file/#healthcheck
--
Richard
--
This email is sent on behalf of Northgate Public Services (UK) Limited and
its associated companies including Rave Technologies (India) Pvt Limited
(together "Northgate Public Services") and is strictly confidential and
intended solely for the addressee(s).
If you are not the intended
recipient of this email you must: (i) not disclose, copy or distribute its
contents to any other person nor use its contents in any way or you may be
acting unlawfully; (ii) contact Northgate Public Services immediately on
+44(0)1442 768445 quoting the name of the sender and the addressee then
delete it from your system.
Northgate Public Services has taken reasonable
precautions to ensure that no viruses are contained in this email, but does
not accept any responsibility once this email has been transmitted. You
should scan attachments (if any) for viruses.
Northgate Public Services
(UK) Limited, registered in England and Wales under number 00968498 with a
registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands
Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India)
Pvt Limited, registered in India under number 117068 with a registered
address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate,
Mumbai, Maharashtra, India, 400001.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Mark Thomas <ma...@apache.org>.
On 11/04/18 09:22, Richard Tearle wrote:
<snip/>
> I've built tomcat from source using the link you provided, and rebuilt the
> containers with this tomcat, and can still reproduce the issue. I've uploaded
> the logs (30s before the connection closed error), to dropbox:
>
> https://www.dropbox.com/s/qe50jbd196krtyo/logs-10-04-17.zip?dl=0
Thanks for these.
I've started to look at them. I don't have any firm conclusions yet. I
have noticed that the problem occurs after a connection is made to the
service from localhost rather than the remote IP that is making the
other requests. The localhost client does not present a certificate.
My working theory (so chances are it is completely wrong) is that the
missing certificate in the request from localhost puts the OpenSSL
engine into an error state that is not correctly handled by Tomcat
causing the subsequent request to fail.
I've also noticed that the debug level log message consistently report 0
bytes being read which looks wrong but is probably a separate (minor) issue.
Investigations continue.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Richard Tearle <ri...@northgateps.com>.
On 5 April 2018 at 08:35, Richard Tearle <ri...@northgateps.com> wrote:
>
> On 4 April 2018 at 17:58, Mark Thomas <ma...@apache.org> wrote:
> > On 26/03/18 08:25, Richard Tearle wrote:
> >
> > <snip/>
> >
> > Thanks. I've got the test application and UI running but I haven't yet
> > reproduced the problem. What parameters are you calling run-test.sh with?
>
> This usually get's a failure within 10 minutes of running:
>
> ./run-test.sh 28800 5000 5000 0 0 true
>
> (I've just tried it and it failed after 4m 23s - from the previous rounds
> of testing, it failed at around the 4m 30s mark 7 times out of 10)
>
> > I'll continue to try and reproduce the issue but I think it makes sense
> > to try and generate some debug data on your system as you can reproduce it.
> >
>
> I'll get onto this, but it might not be until next week.
>
> Thanks
>
> Richard.
I've built tomcat from source using the link you provided, and rebuilt the
containers with this tomcat, and can still reproduce the issue. I've uploaded
the logs (30s before the connection closed error), to dropbox:
https://www.dropbox.com/s/qe50jbd196krtyo/logs-10-04-17.zip?dl=0
Regards
--
Richard
--
This email is sent on behalf of Northgate Public Services (UK) Limited and
its associated companies including Rave Technologies (India) Pvt Limited
(together "Northgate Public Services") and is strictly confidential and
intended solely for the addressee(s).
If you are not the intended
recipient of this email you must: (i) not disclose, copy or distribute its
contents to any other person nor use its contents in any way or you may be
acting unlawfully; (ii) contact Northgate Public Services immediately on
+44(0)1442 768445 quoting the name of the sender and the addressee then
delete it from your system.
Northgate Public Services has taken reasonable
precautions to ensure that no viruses are contained in this email, but does
not accept any responsibility once this email has been transmitted. You
should scan attachments (if any) for viruses.
Northgate Public Services
(UK) Limited, registered in England and Wales under number 00968498 with a
registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands
Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India)
Pvt Limited, registered in India under number 117068 with a registered
address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate,
Mumbai, Maharashtra, India, 400001.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Richard Tearle <ri...@northgateps.com>.
On 4 April 2018 at 17:58, Mark Thomas <ma...@apache.org> wrote:
> On 26/03/18 08:25, Richard Tearle wrote:
>
> <snip/>
>
> Thanks. I've got the test application and UI running but I haven't yet
> reproduced the problem. What parameters are you calling run-test.sh with?
This usually get's a failure within 10 minutes of running:
./run-test.sh 28800 5000 5000 0 0 true
(I've just tried it and it failed after 4m 23s - from the previous rounds
of testing, it failed at around the 4m 30s mark 7 times out of 10)
> I'll continue to try and reproduce the issue but I think it makes sense
> to try and generate some debug data on your system as you can reproduce it.
>
I'll get onto this, but it might not be until next week.
Thanks
Richard.
--
This email is sent on behalf of Northgate Public Services (UK) Limited and
its associated companies including Rave Technologies (India) Pvt Limited
(together "Northgate Public Services") and is strictly confidential and
intended solely for the addressee(s).
If you are not the intended
recipient of this email you must: (i) not disclose, copy or distribute its
contents to any other person nor use its contents in any way or you may be
acting unlawfully; (ii) contact Northgate Public Services immediately on
+44(0)1442 768445 quoting the name of the sender and the addressee then
delete it from your system.
Northgate Public Services has taken reasonable
precautions to ensure that no viruses are contained in this email, but does
not accept any responsibility once this email has been transmitted. You
should scan attachments (if any) for viruses.
Northgate Public Services
(UK) Limited, registered in England and Wales under number 00968498 with a
registered address of Peoplebuilding 2, Peoplebuilding Estate, Maylands
Avenue, Hemel Hempstead, Hertfordshire, HP2 4NW. Rave Technologies (India)
Pvt Limited, registered in India under number 117068 with a registered
address of 2nd Floor, Ballard House, Adi Marzban Marg, Ballard Estate,
Mumbai, Maharashtra, India, 400001.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Mark Thomas <ma...@apache.org>.
On 26/03/18 08:25, Richard Tearle wrote:
<snip/>
> I've uploaded a ZIP with my test "UI" code (standalone java program),
> and the "ESB"
> code which goes into tomcat.
>
> https://www.dropbox.com/s/nhfx7va4uzkr728/Source.zip?dl=0
>
> In the support folder within the ZIP are updated scripts to create the
> certificates - which
> now includes generating the client certificate as well. Also in there
> are the server.xml
> and other tomcat configuration files that are changed as part of our
> installation process
> - although these are the same as I'd included in the previous ZIP.
>
> Also included is a very simple shell script I use to call the UI.
> Usually setting the ESB
> delay to 5 seconds causes the connection closed error to occur in
> around 5 minutes of
> running the program.
Thanks. I've got the test application and UI running but I haven't yet
reproduced the problem. What parameters are you calling run-test.sh with?
I'll continue to try and reproduce the issue but I think it makes sense
to try and generate some debug data on your system as you can reproduce it.
The first step is to move to testing on a Tomcat instance built from
source. This will enable me to add additional debug logging as we go to
try and narrow down the root cause. The build process is fairly simple.
You'll need to check out https://svn.apache.org/repos/asf/tomcat/trunk/
and follow the instructions in BUILDING.txt. You won't need a release
build. The default ant deploy should be fine.
Assuming you can still repeat this on a build from trunk, please enable
the following logging:
- SSL debug logging on client and server via -Djavax.net.debug=ssl
- Tomcat I/O layer debug logging by adding the following to
conf/logging.properties:
org.apache.tomcat.util.net.level = FINE
If you can repeat the issue with the above debug logging enabled, if you
can provide the logs for the 30s or so before the error occurs hopefully
there will be some clues in there as to what is going wrong.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Richard Tearle <ri...@northgateps.com>.
Hi
On 24 March 2018 at 23:06, Mark Thomas <ma...@apache.org> wrote:
> On 23/03/18 15:00, Richard Tearle wrote:
>> On 22 March 2018 at 23:06, Mark Thomas <ma...@apache.org> wrote:
>>> On 22/03/18 15:27, Richard Tearle wrote:
>>>> On 22 March 2018 at 14:49, Mark Thomas <ma...@apache.org> wrote:
>>>
>>> <snip/>
>>>
>
> I've taken another look at the configuration options and
> disableSessionTickets="true" is the only one that stands out as a
> possibility.
>
> I think we have reached the point where we need a reproducible test case.
>
> Mark
>
I've uploaded a ZIP with my test "UI" code (standalone java program),
and the "ESB"
code which goes into tomcat.
https://www.dropbox.com/s/nhfx7va4uzkr728/Source.zip?dl=0
In the support folder within the ZIP are updated scripts to create the
certificates - which
now includes generating the client certificate as well. Also in there
are the server.xml
and other tomcat configuration files that are changed as part of our
installation process
- although these are the same as I'd included in the previous ZIP.
Also included is a very simple shell script I use to call the UI.
Usually setting the ESB
delay to 5 seconds causes the connection closed error to occur in
around 5 minutes of
running the program.
Kinds Regards
--
Richard
--
This email is sent on behalf of Northgate Public Services (UK) Limited and
its associated companies including Rave Technologies (India) Pvt Limited
(together "Northgate Public Services") and is strictly confidential and
intended solely for the addressee(s).
If you are not the intended recipient of this email you must: (i) not
disclose, copy or distribute its contents to any other person nor use its
contents in any way or you may be acting unlawfully; (ii) contact
Northgate Public Services immediately on +44(0)1442 768445 quoting the name
of the sender and the addressee then delete it from your system.
Northgate Public Services has taken reasonable precautions to ensure that
no viruses are contained in this email, but does not accept any
responsibility once this email has been transmitted. You should scan
attachments (if any) for viruses.
Northgate Public Services (UK) Limited, registered in England and Wales
under number 00968498 with a registered address of Peoplebuilding 2,
Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2
4NW. Rave Technologies (India) Pvt Limited, registered in India under
number 117068 with a registered address of 2nd Floor, Ballard House, Adi
Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 400001.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Mark Thomas <ma...@apache.org>.
On 23/03/18 15:00, Richard Tearle wrote:
> On 22 March 2018 at 23:06, Mark Thomas <ma...@apache.org> wrote:
>> On 22/03/18 15:27, Richard Tearle wrote:
>>> On 22 March 2018 at 14:49, Mark Thomas <ma...@apache.org> wrote:
>>
>> <snip/>
>>
>> OK. Time to think about this. NIO + JSSE works whereas NIO + OpenSSL
>> doesn't with the same configuration apart from the presence of the
>> native library.
>>
>> That points to something OpenSSL specific.
>>
>> Disabling client verification fixes the problem.
>>
>> So it looks to be something to do with how OpenSSL handles client
>> verification. It feels like configuration at this point rather than a
>> bug but it needs some more thought.
>>
>> There will probably be some configuration combinations to experiment
>> with but if they fail, something we can use to reproduce this is going
>> to be the next step.
>>
>> Mark
>>
>
> That's fine and many thanks for your help - I can get quite a good turn around
> on testing various configuration changes. Anything that looks
> promising, I'll run
> for 8 hours, and we can usually get an inkling after an hour.
I've taken another look at the configuration options and
disableSessionTickets="true" is the only one that stands out as a
possibility.
I think we have reached the point where we need a reproducible test case.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Richard Tearle <ri...@northgateps.com>.
On 22 March 2018 at 23:06, Mark Thomas <ma...@apache.org> wrote:
> On 22/03/18 15:27, Richard Tearle wrote:
>> On 22 March 2018 at 14:49, Mark Thomas <ma...@apache.org> wrote:
>
> <snip/>
>
> OK. Time to think about this. NIO + JSSE works whereas NIO + OpenSSL
> doesn't with the same configuration apart from the presence of the
> native library.
>
> That points to something OpenSSL specific.
>
> Disabling client verification fixes the problem.
>
> So it looks to be something to do with how OpenSSL handles client
> verification. It feels like configuration at this point rather than a
> bug but it needs some more thought.
>
> There will probably be some configuration combinations to experiment
> with but if they fail, something we can use to reproduce this is going
> to be the next step.
>
> Mark
>
That's fine and many thanks for your help - I can get quite a good turn around
on testing various configuration changes. Anything that looks
promising, I'll run
for 8 hours, and we can usually get an inkling after an hour.
--
Richard
--
This email is sent on behalf of Northgate Public Services (UK) Limited and
its associated companies including Rave Technologies (India) Pvt Limited
(together "Northgate Public Services") and is strictly confidential and
intended solely for the addressee(s).
If you are not the intended recipient of this email you must: (i) not
disclose, copy or distribute its contents to any other person nor use its
contents in any way or you may be acting unlawfully; (ii) contact
Northgate Public Services immediately on +44(0)1442 768445 quoting the name
of the sender and the addressee then delete it from your system.
Northgate Public Services has taken reasonable precautions to ensure that
no viruses are contained in this email, but does not accept any
responsibility once this email has been transmitted. You should scan
attachments (if any) for viruses.
Northgate Public Services (UK) Limited, registered in England and Wales
under number 00968498 with a registered address of Peoplebuilding 2,
Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2
4NW. Rave Technologies (India) Pvt Limited, registered in India under
number 117068 with a registered address of 2nd Floor, Ballard House, Adi
Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 400001.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Mark Thomas <ma...@apache.org>.
On 22/03/18 15:27, Richard Tearle wrote:
> On 22 March 2018 at 14:49, Mark Thomas <ma...@apache.org> wrote:
<snip/>
>> What we have so far is:
>>
>> 8.0.x, http-nio-nnnn (this is always JSSE in 8.0.x), clientAuth="true"
>> This works.
>
> Yes this works.
>
>> 8.5.x, http-nio-openssl-nnnn, certificateVerification="required"
>> This fails intermittently
>
> Its https-openssl-nio-nnnn and yes this fails intermittently.
>
>> 8.5.x, http-nio-jsse-nnnn, certificateVerification="required"
>> This works
>
> Its https-jsse-nio-nnnn, and yes this works
>
>> Is this correct?
>>
>> Thanks,
>>
>> Mark
>>
>
> Also working is 8.5.x, https-openssl-nio-nnnn, certificateVerification="none"
OK. Time to think about this. NIO + JSSE works whereas NIO + OpenSSL
doesn't with the same configuration apart from the presence of the
native library.
That points to something OpenSSL specific.
Disabling client verification fixes the problem.
So it looks to be something to do with how OpenSSL handles client
verification. It feels like configuration at this point rather than a
bug but it needs some more thought.
There will probably be some configuration combinations to experiment
with but if they fail, something we can use to reproduce this is going
to be the next step.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Richard Tearle <ri...@northgateps.com>.
On 22 March 2018 at 14:49, Mark Thomas <ma...@apache.org> wrote:
> On 22/03/18 07:46, Richard Tearle wrote:
>> On 21 March 2018 at 14:54, Mark Thomas <ma...@apache.org> wrote:
[snip]
> Excellent.
>
> There have been a few moving parts here so I'd like to get some
> clarification on exactly where we are. I know from bitter personal
> experience that it is all too easy to end up using a slightly different
> TLS configuration to the one you think you are using so please could you
> confirm the following.
>
> The connector name can be obtained from the logs. You'll see lines that
> look like this:
>
> 22-Mar-2018 14:39:30.156 INFO [main]
> org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler
> ["http-nio-8080"]
> 22-Mar-2018 14:39:30.161 INFO [main]
> org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler
> ["https-jsse-nio-8443"]
> 22-Mar-2018 14:39:30.163 INFO [main]
> org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler
> ["ajp-nio-8009"]
>
> The part I am using below is the bit in the square brackes. The format
> is <protocol>-<tls-implementation-if-any>-<connector>-<port>.
>
> What we have so far is:
>
> 8.0.x, http-nio-nnnn (this is always JSSE in 8.0.x), clientAuth="true"
> This works.
Yes this works.
> 8.5.x, http-nio-openssl-nnnn, certificateVerification="required"
> This fails intermittently
Its https-openssl-nio-nnnn and yes this fails intermittently.
> 8.5.x, http-nio-jsse-nnnn, certificateVerification="required"
> This works
Its https-jsse-nio-nnnn, and yes this works
> Is this correct?
>
> Thanks,
>
> Mark
>
Also working is 8.5.x, https-openssl-nio-nnnn, certificateVerification="none"
Thanks
--
Richard
--
This email is sent on behalf of Northgate Public Services (UK) Limited and
its associated companies including Rave Technologies (India) Pvt Limited
(together "Northgate Public Services") and is strictly confidential and
intended solely for the addressee(s).
If you are not the intended recipient of this email you must: (i) not
disclose, copy or distribute its contents to any other person nor use its
contents in any way or you may be acting unlawfully; (ii) contact
Northgate Public Services immediately on +44(0)1442 768445 quoting the name
of the sender and the addressee then delete it from your system.
Northgate Public Services has taken reasonable precautions to ensure that
no viruses are contained in this email, but does not accept any
responsibility once this email has been transmitted. You should scan
attachments (if any) for viruses.
Northgate Public Services (UK) Limited, registered in England and Wales
under number 00968498 with a registered address of Peoplebuilding 2,
Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2
4NW. Rave Technologies (India) Pvt Limited, registered in India under
number 117068 with a registered address of 2nd Floor, Ballard House, Adi
Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 400001.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Mark Thomas <ma...@apache.org>.
On 22/03/18 07:46, Richard Tearle wrote:
> On 21 March 2018 at 14:54, Mark Thomas <ma...@apache.org> wrote:
<snip/>
>> Please can you test your set-up with 8.5.x, the modified trust store and
>> the same configuration as 8.0.x (NIO, JSSE). That should help us track
>> down where the problem may lie.
>>
>> Thanks,
>>
>> Mark
>>
>
> I created the PKCS12 as you showed above used my 8.0.x configuration and
> ran my test application for 8 hours without a single connection closed error .
Excellent.
There have been a few moving parts here so I'd like to get some
clarification on exactly where we are. I know from bitter personal
experience that it is all too easy to end up using a slightly different
TLS configuration to the one you think you are using so please could you
confirm the following.
The connector name can be obtained from the logs. You'll see lines that
look like this:
22-Mar-2018 14:39:30.156 INFO [main]
org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler
["http-nio-8080"]
22-Mar-2018 14:39:30.161 INFO [main]
org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler
["https-jsse-nio-8443"]
22-Mar-2018 14:39:30.163 INFO [main]
org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler
["ajp-nio-8009"]
The part I am using below is the bit in the square brackes. The format
is <protocol>-<tls-implementation-if-any>-<connector>-<port>.
What we have so far is:
8.0.x, http-nio-nnnn (this is always JSSE in 8.0.x), clientAuth="true"
This works.
8.5.x, http-nio-openssl-nnnn, certificateVerification="required"
This fails intermittently
8.5.x, http-nio-jsse-nnnn, certificateVerification="required"
This works
Is this correct?
Thanks,
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Richard Tearle <ri...@northgateps.com>.
On 21 March 2018 at 14:54, Mark Thomas <ma...@apache.org> wrote:
>
>
> Progress.
>
> Tomcat 8.0.x is more relaxed about the content of PKCS12 trust stores
> then 8.5.x because of a change[1] made so that the effectiveness of the
> certificateVerificationDepth configuration attribute did not depend on
> the presence of a certificate revocation list.
>
> The PKCS12 store the scripts you provided creates includes the private
> key of the trusted certificate. This is ... unusual. 8.5.x skips this
> cert as it does not expect a trusted cert to include the private key.
>
> I've tried various ways to get openssl to create a PKCS12 file without
> the private key but with the certificate without success. In the end I
> used keytool to do this and that worked. Something along these lines:
>
> keytool -storetype pkcs12 -importcert -file ca-cert.pem \
> -keystore ca-truststore.p12
>
> With the modified trust store 8.5.x started with the same configuration
> as 8.0.x.
>
> Please can you test your set-up with 8.5.x, the modified trust store and
> the same configuration as 8.0.x (NIO, JSSE). That should help us track
> down where the problem may lie.
>
> Thanks,
>
> Mark
>
I created the PKCS12 as you showed above used my 8.0.x configuration and
ran my test application for 8 hours without a single connection closed error .
--
Richard
--
This email is sent on behalf of Northgate Public Services (UK) Limited and
its associated companies including Rave Technologies (India) Pvt Limited
(together "Northgate Public Services") and is strictly confidential and
intended solely for the addressee(s).
If you are not the intended recipient of this email you must: (i) not
disclose, copy or distribute its contents to any other person nor use its
contents in any way or you may be acting unlawfully; (ii) contact
Northgate Public Services immediately on +44(0)1442 768445 quoting the name
of the sender and the addressee then delete it from your system.
Northgate Public Services has taken reasonable precautions to ensure that
no viruses are contained in this email, but does not accept any
responsibility once this email has been transmitted. You should scan
attachments (if any) for viruses.
Northgate Public Services (UK) Limited, registered in England and Wales
under number 00968498 with a registered address of Peoplebuilding 2,
Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2
4NW. Rave Technologies (India) Pvt Limited, registered in India under
number 117068 with a registered address of 2nd Floor, Ballard House, Adi
Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 400001.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Mark Thomas <ma...@apache.org>.
On 21/03/18 12:14, Mark Thomas wrote:
> On 21/03/18 08:35, Richard Tearle wrote:
>> On 20 March 2018 at 19:58, Mark Thomas <ma...@apache.org> wrote:
>>
>>> On 20/03/18 14:49, Richard Tearle wrote:
>>> OK. Can you share you configuration and the steps you used to create the
>>> self-signed certificate. I'd like to see if I can reproduce this.
>>>
>>>
>>> Mark
>>>
>>
>> I thought it might be easier to drop the configuration and certificate
>> generating scripts into a ZIP on dropbox:
>>
>> https://www.dropbox.com/s/ib98y6ti2bem53v/TomcatCertsIssue.zip?dl=0
>>
>> In the root of the ZIP contains two scripts, run the create-cert.sh,
>> to generate them.
>>
>> Our Java installation has the Java Cryptography Extension (JCE)
>> installed, and generally we run with the java security manager
>> enabled, but I've tested running without it doesn't seem to affect the
>> error we get.
>
> Perfect. I tweaked the scripts slightly but that was a lot easier than
> starting from scratch.
>
> I can reproduce the problem. I need to do some more research to figure
> out what is going wrong and if we can fix it.
Progress.
Tomcat 8.0.x is more relaxed about the content of PKCS12 trust stores
then 8.5.x because of a change[1] made so that the effectiveness of the
certificateVerificationDepth configuration attribute did not depend on
the presence of a certificate revocation list.
The PKCS12 store the scripts you provided creates includes the private
key of the trusted certificate. This is ... unusual. 8.5.x skips this
cert as it does not expect a trusted cert to include the private key.
I've tried various ways to get openssl to create a PKCS12 file without
the private key but with the certificate without success. In the end I
used keytool to do this and that worked. Something along these lines:
keytool -storetype pkcs12 -importcert -file ca-cert.pem \
-keystore ca-truststore.p12
With the modified trust store 8.5.x started with the same configuration
as 8.0.x.
Please can you test your set-up with 8.5.x, the modified trust store and
the same configuration as 8.0.x (NIO, JSSE). That should help us track
down where the problem may lie.
Thanks,
Mark
[1] https://svn.apache.org/viewvc?view=revision&revision=1757578
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Mark Thomas <ma...@apache.org>.
On 21/03/18 08:35, Richard Tearle wrote:
> On 20 March 2018 at 19:58, Mark Thomas <ma...@apache.org> wrote:
>
>> On 20/03/18 14:49, Richard Tearle wrote:
>> OK. Can you share you configuration and the steps you used to create the
>> self-signed certificate. I'd like to see if I can reproduce this.
>>
>>
>> Mark
>>
>
> I thought it might be easier to drop the configuration and certificate
> generating scripts into a ZIP on dropbox:
>
> https://www.dropbox.com/s/ib98y6ti2bem53v/TomcatCertsIssue.zip?dl=0
>
> In the root of the ZIP contains two scripts, run the create-cert.sh,
> to generate them.
>
> Our Java installation has the Java Cryptography Extension (JCE)
> installed, and generally we run with the java security manager
> enabled, but I've tested running without it doesn't seem to affect the
> error we get.
Perfect. I tweaked the scripts slightly but that was a lot easier than
starting from scratch.
I can reproduce the problem. I need to do some more research to figure
out what is going wrong and if we can fix it.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Richard Tearle <ri...@northgateps.com>.
On 20 March 2018 at 19:58, Mark Thomas <ma...@apache.org> wrote:
> On 20/03/18 14:49, Richard Tearle wrote:
> OK. Can you share you configuration and the steps you used to create the
> self-signed certificate. I'd like to see if I can reproduce this.
>
>
> Mark
>
I thought it might be easier to drop the configuration and certificate
generating scripts into a ZIP on dropbox:
https://www.dropbox.com/s/ib98y6ti2bem53v/TomcatCertsIssue.zip?dl=0
In the root of the ZIP contains two scripts, run the create-cert.sh,
to generate them.
Our Java installation has the Java Cryptography Extension (JCE)
installed, and generally we run with the java security manager
enabled, but I've tested running without it doesn't seem to affect the
error we get.
--
Richard
--
This email is sent on behalf of Northgate Public Services (UK) Limited and
its associated companies including Rave Technologies (India) Pvt Limited
(together "Northgate Public Services") and is strictly confidential and
intended solely for the addressee(s).
If you are not the intended recipient of this email you must: (i) not
disclose, copy or distribute its contents to any other person nor use its
contents in any way or you may be acting unlawfully; (ii) contact
Northgate Public Services immediately on +44(0)1442 768445 quoting the name
of the sender and the addressee then delete it from your system.
Northgate Public Services has taken reasonable precautions to ensure that
no viruses are contained in this email, but does not accept any
responsibility once this email has been transmitted. You should scan
attachments (if any) for viruses.
Northgate Public Services (UK) Limited, registered in England and Wales
under number 00968498 with a registered address of Peoplebuilding 2,
Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2
4NW. Rave Technologies (India) Pvt Limited, registered in India under
number 117068 with a registered address of 2nd Floor, Ballard House, Adi
Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 400001.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Mark Thomas <ma...@apache.org>.
On 20/03/18 14:49, Richard Tearle wrote:
> Hello
>
> On 20 March 2018 at 11:29, Mark Thomas <ma...@apache.org> wrote:
>>
>> On 20/03/18 07:52, Richard Tearle wrote:
>>> Hello
>>>
>>> We have 4 applications built on the same architecture with a web UI
>>> and camel based ESB running in separate Tomcat's, using REST/XML to
>>> communicate between the two. This is all deployed within separate
>>> Docker containers but on the same VM (at least for test), either on
>>> Centos Linux or Oracle Linux. This all works on Tomcat 8.0.x. We've
>>> been upgrading to Tomcat 8.5.x since November last year, but this has
>>> been hampered by what looks to be random connection closed errors when
>>> our UI communicates to the ESB. We have a series of Selenium based
>>> autotests which will fail in different places, but with the same
>>> error:
>>
>> <snip/>
>>
>> There are rather too many factors at play here. It would be good to try
>> and eliminate some of them.
>>
>> What are the known working 8.0.x versions?
>>
>> I looks like you are using JSSE with 8.0.x. It should be possible to use
>> the exact same configuration with 8.5.x. No need to use the native
>> library and no need to switch to the new configuration style.
>>
>> Lets try and get 8.5.x working with JSSE. That should help narrow down
>> the root cause. What happens when you transfer the working 8.0.x config
>> to 8.5.x?
>
> On startup we get:
>
> 20-Mar-2018 14:43:18.908 SEVERE [main]
> org.apache.catalina.util.LifecycleBase.handleSubClassException Failed
> to initialize component [Connector[HTTP/1.1-4001]]
> org.apache.catalina.LifecycleException: Protocol handler initialization failed
> at org.apache.catalina.connector.Connector.initInternal(Connector.java:935)
> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at org.apache.catalina.core.StandardService.initInternal(StandardService.java:530)
> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:852)
> at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
> at org.apache.catalina.startup.Catalina.load(Catalina.java:633)
> at org.apache.catalina.startup.Catalina.load(Catalina.java:656)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:309)
> at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)
> Caused by: java.lang.IllegalArgumentException: the trustAnchors
> parameter must be non-empty
> at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:114)
> at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:85)
> at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:216)
> at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1043)
> at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:540)
> at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:74)
> at org.apache.catalina.connector.Connector.initInternal(Connector.java:932)
> ... 13 more
> Caused by: java.security.InvalidAlgorithmParameterException: the
> trustAnchors parameter must be non-empty
> at java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
> at java.security.cert.PKIXParameters.<init>(PKIXParameters.java:157)
> at java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:130)
> at org.apache.tomcat.util.net.jsse.JSSEUtil.getParameters(JSSEUtil.java:389)
> at org.apache.tomcat.util.net.jsse.JSSEUtil.getTrustManagers(JSSEUtil.java:313)
> at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:112)
> ... 19 more
OK. Can you share you configuration and the steps you used to create the
self-signed certificate. I'd like to see if I can reproduce this.
>> Also, anything you can do to reduce the complexity of the test
>> application (ideally reducing it to simple Servlets/JSPs) would make it
>> easier for others to reproduce the issue.
>
> I can ZIP my code and drop it somewhere if that will help.
Yes. But lets get the JSSE issue fixed first.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Richard Tearle <ri...@northgateps.com>.
Hello
On 20 March 2018 at 11:29, Mark Thomas <ma...@apache.org> wrote:
>
> On 20/03/18 07:52, Richard Tearle wrote:
> > Hello
> >
> > We have 4 applications built on the same architecture with a web UI
> > and camel based ESB running in separate Tomcat's, using REST/XML to
> > communicate between the two. This is all deployed within separate
> > Docker containers but on the same VM (at least for test), either on
> > Centos Linux or Oracle Linux. This all works on Tomcat 8.0.x. We've
> > been upgrading to Tomcat 8.5.x since November last year, but this has
> > been hampered by what looks to be random connection closed errors when
> > our UI communicates to the ESB. We have a series of Selenium based
> > autotests which will fail in different places, but with the same
> > error:
>
> <snip/>
>
> There are rather too many factors at play here. It would be good to try
> and eliminate some of them.
>
> What are the known working 8.0.x versions?
>
> I looks like you are using JSSE with 8.0.x. It should be possible to use
> the exact same configuration with 8.5.x. No need to use the native
> library and no need to switch to the new configuration style.
>
> Lets try and get 8.5.x working with JSSE. That should help narrow down
> the root cause. What happens when you transfer the working 8.0.x config
> to 8.5.x?
On startup we get:
20-Mar-2018 14:43:18.908 SEVERE [main]
org.apache.catalina.util.LifecycleBase.handleSubClassException Failed
to initialize component [Connector[HTTP/1.1-4001]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:935)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:530)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:852)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.startup.Catalina.load(Catalina.java:633)
at org.apache.catalina.startup.Catalina.load(Catalina.java:656)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.catalina.startup.Bootstrap.load(Bootstrap.java:309)
at org.apache.catalina.startup.Bootstrap.main(Bootstrap.java:492)
Caused by: java.lang.IllegalArgumentException: the trustAnchors
parameter must be non-empty
at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:114)
at org.apache.tomcat.util.net.AbstractJsseEndpoint.initialiseSsl(AbstractJsseEndpoint.java:85)
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:216)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1043)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:540)
at org.apache.coyote.http11.AbstractHttp11Protocol.init(AbstractHttp11Protocol.java:74)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:932)
... 13 more
Caused by: java.security.InvalidAlgorithmParameterException: the
trustAnchors parameter must be non-empty
at java.security.cert.PKIXParameters.setTrustAnchors(PKIXParameters.java:200)
at java.security.cert.PKIXParameters.<init>(PKIXParameters.java:157)
at java.security.cert.PKIXBuilderParameters.<init>(PKIXBuilderParameters.java:130)
at org.apache.tomcat.util.net.jsse.JSSEUtil.getParameters(JSSEUtil.java:389)
at org.apache.tomcat.util.net.jsse.JSSEUtil.getTrustManagers(JSSEUtil.java:313)
at org.apache.tomcat.util.net.AbstractJsseEndpoint.createSSLContext(AbstractJsseEndpoint.java:112)
... 19 more
> Also, anything you can do to reduce the complexity of the test
> application (ideally reducing it to simple Servlets/JSPs) would make it
> easier for others to reproduce the issue.
I can ZIP my code and drop it somewhere if that will help.
> Hmm. That looks like a controlled shutdown. Random thought, does setting
> disableSessionTickets="true" help at all when using OpenSSL?
>
I'm afraid that didn't work, but thanks for the suggestion.
> Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
--
Richard
--
This email is sent on behalf of Northgate Public Services (UK) Limited and
its associated companies including Rave Technologies (India) Pvt Limited
(together "Northgate Public Services") and is strictly confidential and
intended solely for the addressee(s).
If you are not the intended recipient of this email you must: (i) not
disclose, copy or distribute its contents to any other person nor use its
contents in any way or you may be acting unlawfully; (ii) contact
Northgate Public Services immediately on +44(0)1442 768445 quoting the name
of the sender and the addressee then delete it from your system.
Northgate Public Services has taken reasonable precautions to ensure that
no viruses are contained in this email, but does not accept any
responsibility once this email has been transmitted. You should scan
attachments (if any) for viruses.
Northgate Public Services (UK) Limited, registered in England and Wales
under number 00968498 with a registered address of Peoplebuilding 2,
Peoplebuilding Estate, Maylands Avenue, Hemel Hempstead, Hertfordshire, HP2
4NW. Rave Technologies (India) Pvt Limited, registered in India under
number 117068 with a registered address of 2nd Floor, Ballard House, Adi
Marzban Marg, Ballard Estate, Mumbai, Maharashtra, India, 400001.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connection closed error and certificateVerification="required"
Posted by Mark Thomas <ma...@apache.org>.
On 20/03/18 07:52, Richard Tearle wrote:
> Hello
>
> We have 4 applications built on the same architecture with a web UI
> and camel based ESB running in separate Tomcat's, using REST/XML to
> communicate between the two. This is all deployed within separate
> Docker containers but on the same VM (at least for test), either on
> Centos Linux or Oracle Linux. This all works on Tomcat 8.0.x. We've
> been upgrading to Tomcat 8.5.x since November last year, but this has
> been hampered by what looks to be random connection closed errors when
> our UI communicates to the ESB. We have a series of Selenium based
> autotests which will fail in different places, but with the same
> error:
<snip/>
> We've seen this on all of the versions of Tomcat 8.5 we've tried from
> 8.5.23 onwards - and I've just tried Tomcat 9.0.6 with the same error.
> We're using Oracle Java version 1.8u162.
>
> Development have looked at this off and on over the last couple of
> months, but without success, so I was asked to reproduce this with a
> simple test case (I'm devops usually, but with a development
> background). The client stack uses a
> PoolingNHttpClientConnectionManager, CloseableHttpAsyncClient and
> AsyncRestTemplate to communicate to the ESB. My simple test case
> replaces the web UI with a simple jar based application, but it still
> replicates the issue. The ESB is a simple Camel route that adds two
> supplied numbers, waits for a time (supplied by the UI), and returns
> the result. The UI repeatedly calls the ESB (on a single thread, with
> no overlapping calls), with two random numbers to be summed, and a set
> amount of time to wait before the ESB returns. Between each call to
> the ESB there's a set amount of time to wait before the next call. It
> seems that when the ESB wait is >= 5secs the time take to fail is
> greatly decreased.
>
> The connection between the UI and the ESB is TLSv1.2 with the ESB
> configured with certificateVerification="required". The connector
> configuration with the ESB's server.xml is:
>
> <Connector port="${servers.port}"
> protocol="org.apache.coyote.http11.Http11NioProtocol"
> maxThreads="150" SSLEnabled="true" scheme="https"
> secure="true" server="Apache" maxPostSize="100000">
> <SSLHostConfig certificateVerification="required" protocols="TLSv1.2"
> caCertificateFile="/usr/local/tomcat/ssl/ca-cert.pem" >
> <Certificate
> certificateFile="/usr/local/tomcat/ssl/server-cert.pem"
>
> certificateKeyFile="/usr/local/tomcat/ssl/server-key.pem"
> certificateKeyPassword="${keystore.pass}">
> </Certificate>
> </SSLHostConfig>
> </Connector>
>
> We are using PEM encoded certificates, and using Tomcat Native Library
> version 1.2.16. If I take out certificateVerification="required", we
> don't get the error (with my simple test case running for an hour)
> I've tried using PKCS12 and JKS certificates, but I can't get Tomcat
> 8.5.x to work with our self signed trust store (we get the
> "trustAnchors parameter must be non-empty" errors), although Tomcat
> 8.0.x seems happy enough with them. So I can't tell if it's a
> difference between using OpenSSL and JSSE.
There are rather too many factors at play here. It would be good to try
and eliminate some of them.
What are the known working 8.0.x versions?
I looks like you are using JSSE with 8.0.x. It should be possible to use
the exact same configuration with 8.5.x. No need to use the native
library and no need to switch to the new configuration style.
Lets try and get 8.5.x working with JSSE. That should help narrow down
the root cause. What happens when you transfer the working 8.0.x config
to 8.5.x?
Also, anything you can do to reduce the complexity of the test
application (ideally reducing it to simple Servlets/JSPs) would make it
easier for others to reproduce the issue.
> Enabling SSL debug traces on the client shows that in the error case,
> after the initial successful handshaking between the client and the
> server, the ESB shuts down the connection just after the client has
> sent the data:
>
> *** CertificateVerify
> Signature Algorithm SHA512withRSA
> I/O dispatcher 1, WRITE: TLSv1.2 Handshake, length = 520
> I/O dispatcher 1, WRITE: TLSv1.2 Change Cipher Spec, length = 1
> *** Finished
> verify_data: { 197, 39, 73, 181, 14, 87, 139, 81, 247, 181, 32, 17 }
> ***
> I/O dispatcher 1, WRITE: TLSv1.2 Handshake, length = 40
> I/O dispatcher 1, READ: TLSv1.2 Change Cipher Spec, length = 1
> I/O dispatcher 1, READ: TLSv1.2 Handshake, length = 40
> *** Finished
> verify_data: { 84, 164, 177, 160, 235, 23, 31, 20, 252, 149, 214, 245 }
> ***
> %% Cached client session: [Session-101,
> TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384]
> I/O dispatcher 1, WRITE: TLSv1.2 Application Data, length = 391
> I/O dispatcher 1, READ: TLSv1.2 Alert, length = 26
> I/O dispatcher 1, RECV TLSv1.2 ALERT: warning, close_notify
> I/O dispatcher 1, closeInboundInternal()
> I/O dispatcher 1, closeOutboundInternal()
> I/O dispatcher 1, SEND TLSv1.2 ALERT: warning, description = close_notify
> I/O dispatcher 1, WRITE: TLSv1.2 Alert, length = 26
>
> The failed call doesn't make it into the ESB application logs, and I
> can't see any errors in any of the ESB logs.
Hmm. That looks like a controlled shutdown. Random thought, does setting
disableSessionTickets="true" help at all when using OpenSSL?
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org