You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Christopher Schultz <ch...@christopherschultz.net> on 2016/12/01 22:59:16 UTC
Connector bindOnInit=false not behaving as expected
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
All,
I'm trying to use JMX to do things with Tomcat Connectors and
ProtocolHandlers. Specifically, I'd like to re-load the keystore
(really certificate) used for an HTTPs connection.
I'm currently using Tomcat 8.0.30 for my testing.
It looks like the ProtocolHandler is really the place where the TLS
configuration is taking effect, and not the Connector, so I'm largely
ignoring the Connector for now. Is that the right choice to make, here?
It seems that calling the pause()/resume() or stop()/start() on the
ProtocolHandler have no effect on resetting the
SSLServerSockeyFactory, which is what would be required to achieve my
goals (update a certificate for a running Tomcat instance).
I suspect I'll have to call init(). When I do this without specifying
bindOnInit=false awful things happen. First, calling init() gets me an
error on stdout that the address is already in use, and then it's
basically not possible to restart the ProtocolHandler after that
point: it's dead as far as I can tell, because you can't call start()
or resume() without getting a whole bunch of errors.
Does that sound like a problem to anyone? I would think that failure
to call init() would leave the ProtocolHandler in an uninitialized
state, but I'm wondering if trying to RE-initialize the
ProtocolHandler should be something that won't damage a
previously-initialized component. When trying to script these types of
connections, having a non-destructive init() might be useful.
So, I set bindOnInit="false" which is documented[1] to unbind on
"stop". When calling stop(), the port continues to be bound by Tomcat.
Calling stop() and then start() throws a BindException. :( Destroying
the ProtocolHandler also leaves the port still bound, and also
(unsurprisingly) destroys the ProtocolHandler.
Stopping the Connector also does not release the port. :( Calling
stop() and then start() also throws a BindException.
At this point, I think I'm stuck. Is there a bug here?
I'm going to upgrade to 8.0.latest and repeat my tests, just in case.
Thanks,
- -chris
[1] http://tomcat.apache.org/tomcat-8.0-doc/config/http.html
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJYQKtEAAoJEBzwKT+lPKRYuIgQAJ9UP58XujGXn8K1ucxNkm4H
2Y+1x+mTHTH1Rkkm6KTRPOPOjY98yIrl+oO91m/hsHrTVQJ/QdQ8y9hIpKCv8S3i
KWyi54627RiefiMc9l1p6/iYsXe5Sntysln02ADUEG4sSSnqwmL1mvsoZlIOXvLd
A+50PKIAtUyG46tqLugfV1YmzUOqkAMEXzF1son6Z/RcH9Eoct8Ez5eb4U1VMoLu
pIlsvrIrEdgrlbvcL0nHHaeR0FjBJprvg80mPe7YaSZiRedAohWizPfJ7kcnVyeZ
8dj9H0MO2GvJOtZCnVcy+6Mlg1WR3DYawNMRL+RKRl32prwdR3A3LkEgF4JVJc+c
+KEhLFlDUhgKhv/axTz69dR7fIk7Haswed9IV5t6ZAZP1CY3fkDhMtMqb3k+NhMh
UVxd8Wi2uxbRI4HtbxBmwv0Nry5khrQZuFaobOmw5ZCOuOWmdyMQbKRHa0Y0YCLZ
TUiQVwP7plXOcOqnk4U8akvjyzV+I3OoFB2SZmeA/sHOj18aA2KMr+vx6phRWMTY
MsJ7D4zqsTS43UqQSYEVu0jpcqD1RigaVCpSUldUdd16VvzqethXxtKYQ/OnR5Kl
0nL/MHfTw+HqGEdBXQQXasSoPNBE1fQuyZKGXOA5rzOhlygFMwmzFUzURKSDCvqz
DjUn9zF29Fol4gTOKKX0
=xoqM
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connector bindOnInit=false not behaving as expected
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mark,
On 12/4/16 3:24 PM, Mark Thomas wrote:
> On 04/12/2016 20:09, Christopher Schultz wrote:
>> All,
>>
>> On 12/1/16 6:07 PM, Christopher Schultz wrote:
>>> All,
>>
>>> On 12/1/16 5:59 PM, Christopher Schultz wrote:
>>>> All,
>>
>>>> I'm trying to use JMX to do things with Tomcat Connectors and
>>>> ProtocolHandlers. Specifically, I'd like to re-load the
>>>> keystore (really certificate) used for an HTTPs connection.
>>
>>>> I'm currently using Tomcat 8.0.30 for my testing.
>>
>>>> It looks like the ProtocolHandler is really the place where
>>>> the TLS configuration is taking effect, and not the
>>>> Connector, so I'm largely ignoring the Connector for now. Is
>>>> that the right choice to make, here?
>
> It depends. What you actually need to do is swap out the
> SSLContext. This is in the SSLHostConfigCertificate.
Yeah, I knew I really needed to swap-out the SSLContext. I wasn't sure
how easy that would be, so I started out with what was possible with JMX
.
>>>> It seems that calling the pause()/resume() or stop()/start()
>>>> on the ProtocolHandler have no effect on resetting the
>>>> SSLServerSockeyFactory, which is what would be required to
>>>> achieve my goals (update a certificate for a running Tomcat
>>>> instance).
>
> You need new code to do this if you want to handle it smoothly.
> Failing that, you need to trigger bind(). Which means:
> bindOnInit=false start() stop()
>
> change TLS config
>
> stop() * drop existing connections, new connections fail here *
> this is generally a Bad Thing (tm) start()
>
> new TLS config available
I'll have to check again, but above I was talking about the protocol
handler, not the Connector. I believe that bouncing the protocol
handler does not reload the certificate, etc.
>>>> I suspect I'll have to call init(). When I do this without
>>>> specifying bindOnInit=false awful things happen. First,
>>>> calling init() gets me an error on stdout that the address is
>>>> already in use, and then it's basically not possible to
>>>> restart the ProtocolHandler after that point: it's dead as
>>>> far as I can tell, because you can't call start() or resume()
>>>> without getting a whole bunch of errors.
>
> You can't just call any method you like. You have to respect the
> standard Lifecycle state transitions even though Endpoint doesn't
> enforce them (the Interface and supporting base classes are not
> visible to the package.
:)
Yeah. I was just flailing about a bit, here.
>>>> Does that sound like a problem to anyone? I would think that
>>>> failure to call init() would leave the ProtocolHandler in an
>>>> uninitialized state, but I'm wondering if trying to
>>>> RE-initialize the ProtocolHandler should be something that
>>>> won't damage a previously-initialized component. When trying
>>>> to script these types of connections, having a
>>>> non-destructive init() might be useful.
>
> This is not a problem.
Ok.
>>>> So, I set bindOnInit="false" which is documented[1] to unbind
>>>> on "stop". When calling stop(), the port continues to be
>>>> bound by Tomcat. Calling stop() and then start() throws a
>>>> BindException. :( Destroying the ProtocolHandler also leaves
>>>> the port still bound, and also (unsurprisingly) destroys the
>>>> ProtocolHandler.
>
> That sounds like a possible problem but we have unit tests that
> explicitly test this behaves as expected (TestXxxEndpoint) so a
> Tomcat bug looks unlikely.
I think this ended up being an error with the keystore itself which
caused the connector to get itself into a stat from which it could not
be resurrected. I'll repeat my observations with a bit more care and
report back. I think there is in fact something here that could be
improved.
>>>> Stopping the Connector also does not release the port. :(
>>>> Calling stop() and then start() also throws a BindException.
>>
>>>> At this point, I think I'm stuck. Is there a bug here?
>>
>>>> I'm going to upgrade to 8.0.latest and repeat my tests, just
>>>> in case.
>>
>>> I updated to 8.0.39 and noticed that I had moved my keystore
>>> out of the way temporarily and so the connector was failing at
>>> some point looking for that. I'll be repeating my tests with
>>> more attention to detail, but what I think I've noticed is that
>>> there are certain errors which can occur that cause the
>>> Connector to get itself into a bad state.
>>
>>> Specifically, I think that problems with the crypto setup
>>> cause the connector to bind to the port, then fail and not
>>> unbind. Any later attempt to re-start the Connector fails
>>> because the port is still bound.
>>
>>> I think the connector should catch (some?) exceptions and
>>> unbind the port in those cases when bindOnInit=false.
>>
>> Any comments on whether or not these items should be considered
>> bugs? Specifically, uncaught exceptions which can prevent the
>> connector from coming back up again?
>
> If you find a sequence where start-up fails due to bad config and
> puts the connector in a start where it can't be started once the
> config has been fixed then that is probably a bug but likely only a
> minor one.
Okay. I'm fairly sure I can come up with a minimal set of
steps-to-reproduce. I'll report back.
Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJYRey+AAoJEBzwKT+lPKRYMAgP/11tR0gZsd6E2zhf9HcVsl37
xQzXLZqLELBXBj0Ih+aK9fk598M8k4KMKBsdMlMSKGvcM0MCSgANHvoGSjYW7znn
9w7Y3+oVmjpR3NNcL4IoHyy15n7zSC8f3E8bDbf6RsWix2YHm6ZE/zf672yzCPJW
1F5ADICbBy0nQt6PPU2gMEnd4T3DE2bG4K/s+EhRKUsY174ohOwmjK+90zhEd0Yi
UFbbSHgv3B3KBFh+lY7S3Cskc3UF/lu4bc/npVSr6AVXEtODO4RHdvifpgEFyG23
okj0jEFKOlKDAoC1UTsbENLXF2OHFamtDbgsj5ynNDKCAzwHbw9/b26mCmdS6ba5
EILBxKHQS1MyMIoSRUzGxmfq9WiprWSCxugcx96q7Wj6se74GT5A8anDWFgl/rMk
4wLeyunHv590CitoAxDusAoG/w+UdNad4MxQ41Y4T61jblbMIOLIk8hFWGj9vNvG
x14KI9YpR4+YPebCGWYQX4N+x/lsLI5Vy8nZVPHgAfMtNkTU5Ru8S1CQHXA1TzIW
B8Eltr3LLaL2uGfZ7b1by0/DKCsrCmGzEM40T8QqM4WCpImIX4q7xd/+HC7ml7Ct
/CUrROSAzhfwHdL2SWgoCYXg5clbOM0UXBnOWBfGHo0sBYdSxeGWU9B6g6KpUhM6
YwlshmQ5UkbcqzUHKDzm
=BFbw
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connector bindOnInit=false not behaving as expected
Posted by Mark Thomas <ma...@apache.org>.
On 06/12/2016 16:25, Christopher Schultz wrote:
> On 12/6/16 4:17 AM, Mark Thomas wrote:
<snip/>
>> It should be as simple as:
>
>> 1. Construct a new SSLContext
>
>> 2. Replace the old SSLContext with the new one.
>
> So if there were a reloadTLSConfiguration method on the Connector (or
> similar component), it could simply call setSSLContext() on the
> SSLHostConfigCertificate and wait for the next request to come in?
In theory, yes. I haven't tested that though.
>> I think what we need is a refreshTLSConfig() method.
>
> Perfect.
>
> If such a method were to be on the Connector (at least for the JMX
> bean representing the Connector), it looks like a fairly simple call
> (or calls.. for each SSLHostConfig) to
> AbstractEndpoint.createSSLContext would do the trick. That method
> seems to not only create the SSLContext but also set that new context
> on the certificate (SSLHostConfigCertificate) object.
>
> I think there might be some thread-safety issues though with the
> changing configuration of e.g. sslHostConfig.enabledProtocols and
> sslHostConfig.enabledCiphers before the certificate is updated. As it
> stands, the code seems to expect that, once initialized, no later
> re-initializations occur. We'd need to make sure that we swap-out that
> configuration in an atomic way. The SSLContext is built separately and
> dropped-onto the certificate object.
sslHostConfigs is a ConcurrentHashMap. It might be possible to construct
the whole new SSLHostConfig (and associated SSLHostConfigCertificate and
SSLContext objects) and then do an atomic update of the mapping in
sslHostConfigs.
> Finally, even for JSSE, there is a releaseSSLContext() that looks like
> it should probably be called[1]. So I think for both OpenSSL and JSSE,
> we'd want to use the same technique. Something like a list of
> currently in-use SSLContext objects (it should be fairly limited) and
> when one of them drops-out of the "in-use" list, we properly-dispose
> of it.
>
> That essentially requires reference-counting around
> connection-management: check-out an SSLContext and then release it
> afterward. The last release for a shutting-down SSLContext will
> release that context.
>
> Does that sound ... overly complicated or foolish? I'm sensitive to
> the fact that there will be a performance impact for something like this
> .
I don't see any option other than reference counting. That could also
get very tricky. It is worth considering keeping a list of all
SSLContexts used and shutting them all down when the Connector stops.
i.e. trade memory for complexity.
> WDYT?
Certainly worth exploring.
> [1] After a little more reading, I see that the JSSESSLContext has a
> no-op implementation of destroy(). So we could be a bit lazy for a
> first implementation.
The destroy() method is only there because APR/native needs it. It is
certainly easier to try this out with JSSE but any solution is going to
have to cover APR as well in the end.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connector bindOnInit=false not behaving as expected
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mark,
On 12/6/16 4:17 AM, Mark Thomas wrote:
> On 05/12/2016 23:03, Christopher Schultz wrote:
>> Mark,
>>
>> On 12/4/16 3:24 PM, Mark Thomas wrote:
>>>>>> It looks like the ProtocolHandler is really the place
>>>>>> where the TLS configuration is taking effect, and not
>>>>>> the Connector, so I'm largely ignoring the Connector for
>>>>>> now. Is that the right choice to make, here?
>>
>>> It depends. What you actually need to do is swap out the
>>> SSLContext. This is in the SSLHostConfigCertificate.
>>
>> It looks like the only place the SSLContext is actually set is
>> from Nio(2?)Endpoint.bind(). So that means that, with the current
>> Tomcat implementation, changing the SSLContext means an unbind()
>> followed by a bind(). Do I have that correct?
>
> Yes.
With changes to the Tomcat code, that answer can obviously change.
>> This would be for JSSE only... for OpenSSL/APR, I suspect it will
>> be the same thing, but any change would have to take both into
>> account.
>>
>> What I'd really like to be able to do is replace the SSLContext
>> without dropping any in-flight requests, while new requests wait
>> to be serviced until the new configuration was available.
>> Something like this:
>>
>> 1. Stop processing incoming connections (e.g. still accept(),
>> but don't handshake, yet... or at least stay bound to the port,
>> but don't yet accept() and allow the TCP stack backlog to queue
>> incoming connections)
>>
>> 2. Load the new TLS configuration
>>
>> 3. Resume accepting connections with the updated TLS
>> configuration
>>
>> 4. Requests accepted before step #1 continue to use the
>> configuration effective at the time
>>
>> I'm not sure how all that squares with that JSSE is willing to
>> do.
>
> It should be as simple as:
>
> 1. Construct a new SSLContext
>
> 2. Replace the old SSLContext with the new one.
So if there were a reloadTLSConfiguration method on the Connector (or
similar component), it could simply call setSSLContext() on the
SSLHostConfigCertificate and wait for the next request to come in?
> It gets slightly trickier with APR/native as you need to
> explicitly clean up the old one and you can't do that until all the
> connections using it have closed. JSSE doesn't have that
> requirement.
Understood. I'm focusing on JSSE for the moment.
I also understand that e.g. websocket makes this even more fun, since
the connections are expected to be relatively long-lived.
>> Once the TLS handshake occurs, presumably it means that the
>> connection will continue to be valid until it's closed. If a
>> connection as mentioned in #4 above is long-running, the old
>> SSLContext etc. will be GC'd after the connection finally closes.
>> If that assumption is not correct, I don't think any of the above
>> is even possible.
>>
>> What probably is possible is pausing all incoming connections,
>> waiting X ms for them to complete, then unbind(), then bind().
>> Client observation would be that the service seems to stall, then
>> fail. Presumably, immediately re-trying would connect and get the
>> new configuration.
>>
>> How difficult do you think it would be to implement such a
>> "graceful" shutdown of the connector? It would be the same as
>> "stop" except that it would first allow all in-flight requests to
>> complete (with some reasonable timeout... say, as specified by a
>> parameter to a gracefulShutdown(int) call?) and then, in the case
>> of the bindOnInit="false", it would unbind().
>
> I think that is the wrong approach. Just leave everything running
> and just replace the SSLContext on the fly. With the caveat that
> APR/native is a little trickier. We might need to add usage
> counting or similar.
Cool.
>> That would allow a JMX client to say "shut down gracefully",
>> waiting for an OK response (which would indicate that the
>> connector had in fact shut down), then immediately request a
>> "start" and the connector would come back up. At this point, I'd
>> probably argue that we should add a "restart gracefully" to the
>> list of JMX-callable actions which just does both of those calls
>> on the server without a second HTTP roundtrip from the JMX
>> client.
>
> I think what we need is a refreshTLSConfig() method.
Perfect.
If such a method were to be on the Connector (at least for the JMX
bean representing the Connector), it looks like a fairly simple call
(or calls.. for each SSLHostConfig) to
AbstractEndpoint.createSSLContext would do the trick. That method
seems to not only create the SSLContext but also set that new context
on the certificate (SSLHostConfigCertificate) object.
I think there might be some thread-safety issues though with the
changing configuration of e.g. sslHostConfig.enabledProtocols and
sslHostConfig.enabledCiphers before the certificate is updated. As it
stands, the code seems to expect that, once initialized, no later
re-initializations occur. We'd need to make sure that we swap-out that
configuration in an atomic way. The SSLContext is built separately and
dropped-onto the certificate object.
Finally, even for JSSE, there is a releaseSSLContext() that looks like
it should probably be called[1]. So I think for both OpenSSL and JSSE,
we'd want to use the same technique. Something like a list of
currently in-use SSLContext objects (it should be fairly limited) and
when one of them drops-out of the "in-use" list, we properly-dispose
of it.
That essentially requires reference-counting around
connection-management: check-out an SSLContext and then release it
afterward. The last release for a shutting-down SSLContext will
release that context.
Does that sound ... overly complicated or foolish? I'm sensitive to
the fact that there will be a performance impact for something like this
.
WDYT?
- -chris
[1] After a little more reading, I see that the JSSESSLContext has a
no-op implementation of destroy(). So we could be a bit lazy for a
first implementation.
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJYRuZ3AAoJEBzwKT+lPKRYejAQALVcIPIXbGOQFxantDBoLgzS
+BVZ6ltmRgCihLrQWUunryKr0M0fI5YhhuW72mj2oMidddXjdZubNora2YdMZzAa
RppkKTBmSB6sTRGs+DnI5KkbKRFeoidOHW/fWal8A7+vhWghs/wQA+eyxXKTnxBs
bEAs8fFoOe83bAU9+NsVv0ROPniDJqajxWDFx/oISUlleucQyD0YKKUWlMpJztvQ
tSY3C276Blm7Owbf9fNjEVxdo55+eQB+M06wfZIYmO3dtn80Euv0fbBJIdCZJSOc
eKq346MOAR4xoLut6eFwAIYwmHNHL/CURLYtiq906VM/x10PBvFKgoypxr81BMIZ
Em2LZ4q3F73kmF/BCxe0XDy1M1WJsiBdFed++m2nnVCqD8m4EQLeq83whA2JgePD
XBqHHipeWaJ/omoV25+nAeTprZgpgILSrmjBss1zVGNaLpxw1lo4qqAC0ykEPTcD
y/LsNpxWwegiOfjFO5SC7/Pjl6xwLlDlZfEaqQJj1POqs2wCub54IZgjJ5J0oDdE
XBbH7lx4Pa4zFndRXEHuyho5rki+rlhcP/q5EcXJqMo/6t6X5uFBoUM1uTjt7Gzc
XoolCSgMaJdrzzfGf/yUm5hZ5KtawgTPi6CFMw5f3DLpghXYByUUP22fkR28yCTk
BwjVSIHnNBPQZ0qpuzHI
=WG0O
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connector bindOnInit=false not behaving as expected
Posted by Mark Thomas <ma...@apache.org>.
On 05/12/2016 23:03, Christopher Schultz wrote:
> Mark,
>
> On 12/4/16 3:24 PM, Mark Thomas wrote:
>>>>> It looks like the ProtocolHandler is really the place where
>>>>> the TLS configuration is taking effect, and not the
>>>>> Connector, so I'm largely ignoring the Connector for now. Is
>>>>> that the right choice to make, here?
>
>> It depends. What you actually need to do is swap out the
>> SSLContext. This is in the SSLHostConfigCertificate.
>
> It looks like the only place the SSLContext is actually set is from
> Nio(2?)Endpoint.bind(). So that means that, with the current Tomcat
> implementation, changing the SSLContext means an unbind() followed by
> a bind(). Do I have that correct?
Yes.
> This would be for JSSE only... for OpenSSL/APR, I suspect it will be
> the same thing, but any change would have to take both into account.
>
> What I'd really like to be able to do is replace the SSLContext
> without dropping any in-flight requests, while new requests wait to be
> serviced until the new configuration was available. Something like this:
>
> 1. Stop processing incoming connections (e.g. still accept(), but
> don't handshake, yet... or at least stay bound to the port, but don't
> yet accept() and allow the TCP stack backlog to queue incoming
> connections)
>
> 2. Load the new TLS configuration
>
> 3. Resume accepting connections with the updated TLS configuration
>
> 4. Requests accepted before step #1 continue to use the configuration
> effective at the time
>
> I'm not sure how all that squares with that JSSE is willing to do.
It should be as simple as:
1. Construct a new SSLContext
2. Replace the old SSLContext with the new one.
It gets slightly trickier with APR/native as you need to explicitly
clean up the old one and you can't do that until all the connections
using it have closed. JSSE doesn't have that requirement.
> Once the TLS handshake occurs, presumably it means that the connection
> will continue to be valid until it's closed. If a connection as
> mentioned in #4 above is long-running, the old SSLContext etc. will be
> GC'd after the connection finally closes. If that assumption is not
> correct, I don't think any of the above is even possible.
>
> What probably is possible is pausing all incoming connections, waiting
> X ms for them to complete, then unbind(), then bind(). Client
> observation would be that the service seems to stall, then fail.
> Presumably, immediately re-trying would connect and get the new
> configuration.
>
> How difficult do you think it would be to implement such a "graceful"
> shutdown of the connector? It would be the same as "stop" except that
> it would first allow all in-flight requests to complete (with some
> reasonable timeout... say, as specified by a parameter to a
> gracefulShutdown(int) call?) and then, in the case of the
> bindOnInit="false", it would unbind().
I think that is the wrong approach. Just leave everything running and
just replace the SSLContext on the fly. With the caveat that APR/native
is a little trickier. We might need to add usage counting or similar.
> That would allow a JMX client to say "shut down gracefully", waiting
> for an OK response (which would indicate that the connector had in
> fact shut down), then immediately request a "start" and the connector
> would come back up. At this point, I'd probably argue that we should
> add a "restart gracefully" to the list of JMX-callable actions which
> just does both of those calls on the server without a second HTTP
> roundtrip from the JMX client.
I think what we need is a refreshTLSConfig() method.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connector bindOnInit=false not behaving as expected
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
Mark,
On 12/4/16 3:24 PM, Mark Thomas wrote:
>>>> It looks like the ProtocolHandler is really the place where
>>>> the TLS configuration is taking effect, and not the
>>>> Connector, so I'm largely ignoring the Connector for now. Is
>>>> that the right choice to make, here?
>
> It depends. What you actually need to do is swap out the
> SSLContext. This is in the SSLHostConfigCertificate.
It looks like the only place the SSLContext is actually set is from
Nio(2?)Endpoint.bind(). So that means that, with the current Tomcat
implementation, changing the SSLContext means an unbind() followed by
a bind(). Do I have that correct?
This would be for JSSE only... for OpenSSL/APR, I suspect it will be
the same thing, but any change would have to take both into account.
What I'd really like to be able to do is replace the SSLContext
without dropping any in-flight requests, while new requests wait to be
serviced until the new configuration was available. Something like this:
1. Stop processing incoming connections (e.g. still accept(), but
don't handshake, yet... or at least stay bound to the port, but don't
yet accept() and allow the TCP stack backlog to queue incoming
connections)
2. Load the new TLS configuration
3. Resume accepting connections with the updated TLS configuration
4. Requests accepted before step #1 continue to use the configuration
effective at the time
I'm not sure how all that squares with that JSSE is willing to do.
Once the TLS handshake occurs, presumably it means that the connection
will continue to be valid until it's closed. If a connection as
mentioned in #4 above is long-running, the old SSLContext etc. will be
GC'd after the connection finally closes. If that assumption is not
correct, I don't think any of the above is even possible.
What probably is possible is pausing all incoming connections, waiting
X ms for them to complete, then unbind(), then bind(). Client
observation would be that the service seems to stall, then fail.
Presumably, immediately re-trying would connect and get the new
configuration.
How difficult do you think it would be to implement such a "graceful"
shutdown of the connector? It would be the same as "stop" except that
it would first allow all in-flight requests to complete (with some
reasonable timeout... say, as specified by a parameter to a
gracefulShutdown(int) call?) and then, in the case of the
bindOnInit="false", it would unbind().
That would allow a JMX client to say "shut down gracefully", waiting
for an OK response (which would indicate that the connector had in
fact shut down), then immediately request a "start" and the connector
would come back up. At this point, I'd probably argue that we should
add a "restart gracefully" to the list of JMX-callable actions which
just does both of those calls on the server without a second HTTP
roundtrip from the JMX client.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJYRfJcAAoJEBzwKT+lPKRYYz4P/RiXs43l3pCXS5RmivlhaIg2
QSTq2Og50I8+qK6wuhpABp0VvwOI29JQ0M+Bg2jSMM7sFRpnH+Z6brPOKArMwrYz
fBRgg0HfwFJQqAFk+wdXfoSNkbvHtsvZIhrrPcJGth9Tr/BHwDXHcmCKmScTCg93
ynyw6AWDzZ8SxU1YpKHYhBXb1Zbz4KDGcxBLDcRs6S4qmnNeu2JQfn7xQNImPZg0
yToctnzzH/SRG2rEoySyKMY+jpynreMw9jUOmdygJaGh1r3dv00yivu08Dv9/8MI
yiQCq678sa0U5pUKTZtvhb00ZszmCZ56dJTC0u51nSTWOkrj+AmaBjmWIR1CUdMO
qO2PhRsL/EnYDeEYORcLuJ1oa+J3GcxplgsU0PX09vVN+7TF7F3h0oUH34PvMyIJ
hrLQv958nMevodkeDEgW5/nVV8WctXXUbSgKG4IFufwPh3paMmqsO+9ea8/vnXWN
iAnijwgRun2Cwl2CrjPDu9isN5zoLQTlyYJnw+tOE4JYFjCpLVJ8ykTeRQs/fSDD
qEuMIAT3+MrNTIMN8c5zBoGxTIouWRuXDetpSlaVm2Ln1R4TghdkBf+D4RdBiMhV
ED08VnzghwbuG6VpdbFTjTThj5qEb+SLiQmM7rf9WF36s7Gatl4z1FyTJn7gtj2+
Uvwqqd0kXA/XFnpmttHA
=qvTT
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connector bindOnInit=false not behaving as expected
Posted by Mark Thomas <ma...@apache.org>.
On 04/12/2016 20:09, Christopher Schultz wrote:
> All,
>
> On 12/1/16 6:07 PM, Christopher Schultz wrote:
>> All,
>
>> On 12/1/16 5:59 PM, Christopher Schultz wrote:
>>> All,
>
>>> I'm trying to use JMX to do things with Tomcat Connectors and
>>> ProtocolHandlers. Specifically, I'd like to re-load the keystore
>>> (really certificate) used for an HTTPs connection.
>
>>> I'm currently using Tomcat 8.0.30 for my testing.
>
>>> It looks like the ProtocolHandler is really the place where the
>>> TLS configuration is taking effect, and not the Connector, so
>>> I'm largely ignoring the Connector for now. Is that the right
>>> choice to make, here?
It depends. What you actually need to do is swap out the SSLContext.
This is in the SSLHostConfigCertificate.
>>> It seems that calling the pause()/resume() or stop()/start() on
>>> the ProtocolHandler have no effect on resetting the
>>> SSLServerSockeyFactory, which is what would be required to
>>> achieve my goals (update a certificate for a running Tomcat
>>> instance).
You need new code to do this if you want to handle it smoothly. Failing
that, you need to trigger bind(). Which means:
bindOnInit=false
start()
stop()
change TLS config
stop()
* drop existing connections, new connections fail here
* this is generally a Bad Thing (tm)
start()
new TLS config available
>>> I suspect I'll have to call init(). When I do this without
>>> specifying bindOnInit=false awful things happen. First, calling
>>> init() gets me an error on stdout that the address is already in
>>> use, and then it's basically not possible to restart the
>>> ProtocolHandler after that point: it's dead as far as I can
>>> tell, because you can't call start() or resume() without getting
>>> a whole bunch of errors.
You can't just call nay method you like. You have to respect the
standard Lifecycle state transitions even though Endpoint doesn't
enforce them (the Interface and supporting base classes are not visible
to the package.
>>> Does that sound like a problem to anyone? I would think that
>>> failure to call init() would leave the ProtocolHandler in an
>>> uninitialized state, but I'm wondering if trying to
>>> RE-initialize the ProtocolHandler should be something that won't
>>> damage a previously-initialized component. When trying to script
>>> these types of connections, having a non-destructive init() might
>>> be useful.
This is not a problem.
>>> So, I set bindOnInit="false" which is documented[1] to unbind on
>>> "stop". When calling stop(), the port continues to be bound by
>>> Tomcat. Calling stop() and then start() throws a BindException.
>>> :( Destroying the ProtocolHandler also leaves the port still
>>> bound, and also (unsurprisingly) destroys the ProtocolHandler.
That sounds like a possible problem but we have unit tests that
explicitly test this behaves as expected (TestXxxEndpoint) so a Tomcat
bug looks unlikely.
>>> Stopping the Connector also does not release the port. :( Calling
>>> stop() and then start() also throws a BindException.
>
>>> At this point, I think I'm stuck. Is there a bug here?
>
>>> I'm going to upgrade to 8.0.latest and repeat my tests, just in
>>> case.
>
>> I updated to 8.0.39 and noticed that I had moved my keystore out
>> of the way temporarily and so the connector was failing at some
>> point looking for that. I'll be repeating my tests with more
>> attention to detail, but what I think I've noticed is that there
>> are certain errors which can occur that cause the Connector to get
>> itself into a bad state.
>
>> Specifically, I think that problems with the crypto setup cause
>> the connector to bind to the port, then fail and not unbind. Any
>> later attempt to re-start the Connector fails because the port is
>> still bound.
>
>> I think the connector should catch (some?) exceptions and unbind
>> the port in those cases when bindOnInit=false.
>
> Any comments on whether or not these items should be considered bugs?
> Specifically, uncaught exceptions which can prevent the connector from
> coming back up again?
If you find a sequence where start-up fails due to bad config and puts
the connector in a start where it can't be started once the config has
been fixed then that is probably a bug but likely only a minor one.
Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connector bindOnInit=false not behaving as expected
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
All,
On 12/1/16 6:07 PM, Christopher Schultz wrote:
> All,
>
> On 12/1/16 5:59 PM, Christopher Schultz wrote:
>> All,
>
>> I'm trying to use JMX to do things with Tomcat Connectors and
>> ProtocolHandlers. Specifically, I'd like to re-load the keystore
>> (really certificate) used for an HTTPs connection.
>
>> I'm currently using Tomcat 8.0.30 for my testing.
>
>> It looks like the ProtocolHandler is really the place where the
>> TLS configuration is taking effect, and not the Connector, so
>> I'm largely ignoring the Connector for now. Is that the right
>> choice to make, here?
>
>> It seems that calling the pause()/resume() or stop()/start() on
>> the ProtocolHandler have no effect on resetting the
>> SSLServerSockeyFactory, which is what would be required to
>> achieve my goals (update a certificate for a running Tomcat
>> instance).
>
>> I suspect I'll have to call init(). When I do this without
>> specifying bindOnInit=false awful things happen. First, calling
>> init() gets me an error on stdout that the address is already in
>> use, and then it's basically not possible to restart the
>> ProtocolHandler after that point: it's dead as far as I can
>> tell, because you can't call start() or resume() without getting
>> a whole bunch of errors.
>
>> Does that sound like a problem to anyone? I would think that
>> failure to call init() would leave the ProtocolHandler in an
>> uninitialized state, but I'm wondering if trying to
>> RE-initialize the ProtocolHandler should be something that won't
>> damage a previously-initialized component. When trying to script
>> these types of connections, having a non-destructive init() might
>> be useful.
>
>> So, I set bindOnInit="false" which is documented[1] to unbind on
>> "stop". When calling stop(), the port continues to be bound by
>> Tomcat. Calling stop() and then start() throws a BindException.
>> :( Destroying the ProtocolHandler also leaves the port still
>> bound, and also (unsurprisingly) destroys the ProtocolHandler.
>
>> Stopping the Connector also does not release the port. :( Calling
>> stop() and then start() also throws a BindException.
>
>> At this point, I think I'm stuck. Is there a bug here?
>
>> I'm going to upgrade to 8.0.latest and repeat my tests, just in
>> case.
>
> I updated to 8.0.39 and noticed that I had moved my keystore out
> of the way temporarily and so the connector was failing at some
> point looking for that. I'll be repeating my tests with more
> attention to detail, but what I think I've noticed is that there
> are certain errors which can occur that cause the Connector to get
> itself into a bad state.
>
> Specifically, I think that problems with the crypto setup cause
> the connector to bind to the port, then fail and not unbind. Any
> later attempt to re-start the Connector fails because the port is
> still bound.
>
> I think the connector should catch (some?) exceptions and unbind
> the port in those cases when bindOnInit=false.
Any comments on whether or not these items should be considered bugs?
Specifically, uncaught exceptions which can prevent the connector from
coming back up again?
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJYRHgQAAoJEBzwKT+lPKRYwzgP/3BlD2DnNa5ZJwpX1Wcl9JXe
3Oj2zgUbv0z9qExmAYfuIYYpyUiIiGoO0oJIo9r4sGjCSeQg/GWv/w9q03YlCKOE
U/+t4FH2UZ0W15x27PF1j1TSMna8Heq2Wysqp3ccgyX4E3BGcOxTE6RwE9MQI3+e
aduTjtw4Wy34jh426pX4ilSxgagHPDXeqsuvUsbt5nS8FGizK0fMqYvcIfpfpfCx
F+BYGKi5ZkafhyrkXyCGzdL8XzpAFDC8Vx2bqQGYAGrsDmP4S7xQwHhMPoG4T0xN
bB98jV7MMJTxb+P4Yog+xN1t3MnpsgCtztW6TxVflSc+fm6mPy+9NGgN7D3vLSHw
EWZSR9AzuLQwNRI8hqubNoLIXT0ddoG3zZ4CP6YtMbtCN9qIGmby9idCM2mEoarp
oBpQL+rvcvQs3qJU9KsVOV3YXjNPsHciXquI8HrvaqwMPaSf3NRHR897OZOwapjX
IHV6/8K2yqL2t3UXoOS87O/94cCZb5g/XPA/QZO8ljF6M7rDKE8omR7XcbMoyW9p
4Dm2RSiUm7nqn2Eto7p2fKlQ10iK4bdeQ/6q4nLuMMEQ/Nmh6pRPt9Jgz8A8goTZ
k/xSQJg+5Pij/fg76StfkLWULdVSm6tUkW9CPPzqEvdPWZeEOn6KBQ9m4d0S80Ro
9MMGOsmaWRf6TfEkwmkB
=Arwu
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Connector bindOnInit=false not behaving as expected
Posted by Christopher Schultz <ch...@christopherschultz.net>.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256
All,
On 12/1/16 5:59 PM, Christopher Schultz wrote:
> All,
>
> I'm trying to use JMX to do things with Tomcat Connectors and
> ProtocolHandlers. Specifically, I'd like to re-load the keystore
> (really certificate) used for an HTTPs connection.
>
> I'm currently using Tomcat 8.0.30 for my testing.
>
> It looks like the ProtocolHandler is really the place where the
> TLS configuration is taking effect, and not the Connector, so I'm
> largely ignoring the Connector for now. Is that the right choice to
> make, here?
>
> It seems that calling the pause()/resume() or stop()/start() on
> the ProtocolHandler have no effect on resetting the
> SSLServerSockeyFactory, which is what would be required to achieve
> my goals (update a certificate for a running Tomcat instance).
>
> I suspect I'll have to call init(). When I do this without
> specifying bindOnInit=false awful things happen. First, calling
> init() gets me an error on stdout that the address is already in
> use, and then it's basically not possible to restart the
> ProtocolHandler after that point: it's dead as far as I can tell,
> because you can't call start() or resume() without getting a whole
> bunch of errors.
>
> Does that sound like a problem to anyone? I would think that
> failure to call init() would leave the ProtocolHandler in an
> uninitialized state, but I'm wondering if trying to RE-initialize
> the ProtocolHandler should be something that won't damage a
> previously-initialized component. When trying to script these types
> of connections, having a non-destructive init() might be useful.
>
> So, I set bindOnInit="false" which is documented[1] to unbind on
> "stop". When calling stop(), the port continues to be bound by
> Tomcat. Calling stop() and then start() throws a BindException. :(
> Destroying the ProtocolHandler also leaves the port still bound,
> and also (unsurprisingly) destroys the ProtocolHandler.
>
> Stopping the Connector also does not release the port. :( Calling
> stop() and then start() also throws a BindException.
>
> At this point, I think I'm stuck. Is there a bug here?
>
> I'm going to upgrade to 8.0.latest and repeat my tests, just in
> case.
I updated to 8.0.39 and noticed that I had moved my keystore out of
the way temporarily and so the connector was failing at some point
looking for that. I'll be repeating my tests with more attention to
detail, but what I think I've noticed is that there are certain errors
which can occur that cause the Connector to get itself into a bad state.
Specifically, I think that problems with the crypto setup cause the
connector to bind to the port, then fail and not unbind. Any later
attempt to re-start the Connector fails because the port is still bound.
I think the connector should catch (some?) exceptions and unbind the
port in those cases when bindOnInit=false.
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQIcBAEBCAAGBQJYQK0sAAoJEBzwKT+lPKRYxEcQAKOnPrSGT12lsaO+Qlk1f0iJ
aUjcEcseYN74R91rd0n/vZdrWlx1pigH4bUkzJsqgxt6En14dQiw1AIgh8SY4QDe
UZxq4HIunDmcVoOpuOl+GOJRlHoA4LvYIVDM8vDBXqBYt3yBgpeC0DiIB+WfQ2hS
bkooGIC1Vkb1BMXv0KOI5NWZ30PoUfm1cfHuoALh1yrPJO2PPYwmaKKka7c5IBKw
t2mnT++3k9qGROw8i9koiEXLgdwKlSpKFz/lPZ8Jd+G3Sar3l0rRPxTFAtp5h2nz
kJOrKqGNnr0N2cihadbFpOncYCowq12bgx3xAu2ezdE6AdhpbpD4dVntrYICV7Sb
gBc4wmNBA3msTzw6dVhf5aDPLOIwHnJJ9oKk/PnVYfvZKjup1ftteYaK//Nty1is
TPywDMsOMFZow9I4ywvibLe+3Orf9wv8HuhZB5bl2MU09OwY3V7+lI5Ehk0iNM5v
SBfZ0dXeeWg8wRxdyaQyC4NvVueRLm7X1B5Jd3+8AUcI9E4sCwVaMLEZHqlANfjX
EGv+Tj7cYvrB8whP0rcZ78emeWDE0P1oR2bbRH5pFBC/MU7ILFeX2g3hcpwQksC3
mODFYFcTBsUdfBrcum5SkpKQL/EjNspztalpLLi712ESd28K4IOSA1VqQAL/MMrl
fjXmgcBr+xfTnZl62eUu
=1MzW
-----END PGP SIGNATURE-----
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org