You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Jerry Malcolm <te...@malcolms.com> on 2023/06/15 04:41:46 UTC
Conclusion - Re: Crypto Randomly Not Getting Initialized
On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>
> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
>> Rob,
>>
>> On 6/13/2023 11:34 AM, Rob Sargent wrote:
>>> In /etc/rc.local I have:
>>>>
>>>> ----------------------
>>>> sleep 120s
>>>> systemctl start tomcat9
>>>>
>>>> ---------------------
>>>>
>>>> I put the sleep in back a couple of years ago that for some reason
>>>> 'fixed' this same random, intermittent crypto file exception.
>>>>
>>>>
>>>
> One observation.... even though the error doesn't show up in the log
> until my first db call, I think the real error has to be occurring
> during tomcat boot. If the failure occurs, I continue to get the same
> "Can't read cryptographic policy directory: unlimited" even hours
> later after boot every time I send another request to TC. If I then
> reboot TC and this time I don't get the failure, that unlimited folder
> magically becomes available. There is something happening in TC boot
> with some sort of crypto initialization that either succeeds or
> fails. And the exception when trying to get connections long after TC
> boot is just hitting whatever problem occurred during boot. In other
> words, delaying my calls, even for hours, has no effect on accessing
> that folder if the problem is present. Rebooting TC gives it another
> chance to succeed or fail. Any ideas what TC could be doing on boot
> that could lock up that folder?
>
>
I never really found a definitive fix for this. But after all of the
research and logging and everything, I pretty much concluded that
there's some kind of timing/race condition between Tomcat boot and Java
crypto initialization. Given that, I figured it was highly unlikely
that two different Java JVM implementations would have the same precise
implementation code to trigger that race condition. So as a last ditch
effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
JVM. It's circumstantial evidence at best, and running a production
environment on fixes that just went away goes against my grain. But if
the problem goes away, maybe it won't come back. At this time, when
using Corretto JVM, I have not encountered the Crypto directory error.
It's been running on all server instances now for almost 24 hours with
no problems. So I guess my suggestion to anyone else that might
encounter something like this, trying using a different JVM from a
different provider. That appears at this time to have worked for me.
Thanks for all the support.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Simon,
On 6/21/23 03:19, Simon Matter wrote:
>> Jerry,
>>
>> On 6/15/23 00:41, Jerry Malcolm wrote:
>>>
>>> On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>>>>
>>>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
>>>>> Rob,
>>>>>
>>>>> On 6/13/2023 11:34 AM, Rob Sargent wrote:
>>>>>> In /etc/rc.local I have:
>>>>>>>
>>>>>>> ----------------------
>>>>>>> sleep 120s
>>>>>>> systemctl start tomcat9
>>>>>>>
>>>>>>> ---------------------
>>>>>>>
>>>>>>> I put the sleep in back a couple of years ago that for some reason
>>>>>>> 'fixed' this same random, intermittent crypto file exception.
>>>>>>>
>>>>>>>
>>>>>>
>>>> One observation.... even though the error doesn't show up in the log
>>>> until my first db call, I think the real error has to be occurring
>>>> during tomcat boot. If the failure occurs, I continue to get the same
>>>> "Can't read cryptographic policy directory: unlimited" even hours
>>>> later after boot every time I send another request to TC. If I then
>>>> reboot TC and this time I don't get the failure, that unlimited folder
>>>> magically becomes available. There is something happening in TC boot
>>>> with some sort of crypto initialization that either succeeds or
>>>> fails. And the exception when trying to get connections long after TC
>>>> boot is just hitting whatever problem occurred during boot. In other
>>>> words, delaying my calls, even for hours, has no effect on accessing
>>>> that folder if the problem is present. Rebooting TC gives it another
>>>> chance to succeed or fail. Any ideas what TC could be doing on boot
>>>> that could lock up that folder?
>>>>
>>>>
>>> I never really found a definitive fix for this. But after all of the
>>> research and logging and everything, I pretty much concluded that
>>> there's some kind of timing/race condition between Tomcat boot and Java
>>> crypto initialization. Given that, I figured it was highly unlikely
>>> that two different Java JVM implementations would have the same precise
>>> implementation code to trigger that race condition. So as a last ditch
>>> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
>>> JVM. It's circumstantial evidence at best, and running a production
>>> environment on fixes that just went away goes against my grain. But if
>>> the problem goes away, maybe it won't come back. At this time, when
>>> using Corretto JVM, I have not encountered the Crypto directory error.
>>> It's been running on all server instances now for almost 24 hours with
>>> no problems. So I guess my suggestion to anyone else that might
>>> encounter something like this, trying using a different JVM from a
>>> different provider. That appears at this time to have worked for me.
>>
>> Yeah, that's certainly a very unsatisfying result, but ... you can't
>> argue with results. I'd love to hear if you end up with any further
>> information.
>>
>> -chris
>
> I was asking this before and ask again, how do we know it's not a missing
> RNG entropy while the crypto stuff of the Java VM is initialised?
Because the symptoms of *that* problem (very long lags when requesting
random numbers) are not being experienced. And also because the error
message is crystal clear:
[...]
Caused by: java.lang.SecurityException: Can not initialize cryptographic
mechanism
at
java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120) ... 86 mo
Caused by: java.lang.SecurityException: Can't read cryptographic policy
directory: unlimited
at
java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:326)
[...]
The cryptographic provider itself cannot even initialize. It's *way*
before any randomness is needed. The "unlimited" directory is presumably
the directory which contains the policy files which do not impose the
archaic US-imposed "export restrictions" on cryptographic primitives.
The Provider cannot find the configuration directory, and fails.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
Posted by Simon Matter <si...@invoca.ch>.
> Jerry,
>
> On 6/15/23 00:41, Jerry Malcolm wrote:
>>
>> On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>>>
>>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
>>>> Rob,
>>>>
>>>> On 6/13/2023 11:34 AM, Rob Sargent wrote:
>>>>> In /etc/rc.local I have:
>>>>>>
>>>>>> ----------------------
>>>>>> sleep 120s
>>>>>> systemctl start tomcat9
>>>>>>
>>>>>> ---------------------
>>>>>>
>>>>>> I put the sleep in back a couple of years ago that for some reason
>>>>>> 'fixed' this same random, intermittent crypto file exception.
>>>>>>
>>>>>>
>>>>>
>>> One observation.... even though the error doesn't show up in the log
>>> until my first db call, I think the real error has to be occurring
>>> during tomcat boot. If the failure occurs, I continue to get the same
>>> "Can't read cryptographic policy directory: unlimited" even hours
>>> later after boot every time I send another request to TC. If I then
>>> reboot TC and this time I don't get the failure, that unlimited folder
>>> magically becomes available. There is something happening in TC boot
>>> with some sort of crypto initialization that either succeeds or
>>> fails. And the exception when trying to get connections long after TC
>>> boot is just hitting whatever problem occurred during boot. In other
>>> words, delaying my calls, even for hours, has no effect on accessing
>>> that folder if the problem is present. Rebooting TC gives it another
>>> chance to succeed or fail. Any ideas what TC could be doing on boot
>>> that could lock up that folder?
>>>
>>>
>> I never really found a definitive fix for this. But after all of the
>> research and logging and everything, I pretty much concluded that
>> there's some kind of timing/race condition between Tomcat boot and Java
>> crypto initialization. Given that, I figured it was highly unlikely
>> that two different Java JVM implementations would have the same precise
>> implementation code to trigger that race condition. So as a last ditch
>> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
>> JVM. It's circumstantial evidence at best, and running a production
>> environment on fixes that just went away goes against my grain. But if
>> the problem goes away, maybe it won't come back. At this time, when
>> using Corretto JVM, I have not encountered the Crypto directory error.
>> It's been running on all server instances now for almost 24 hours with
>> no problems. So I guess my suggestion to anyone else that might
>> encounter something like this, trying using a different JVM from a
>> different provider. That appears at this time to have worked for me.
>
> Yeah, that's certainly a very unsatisfying result, but ... you can't
> argue with results. I'd love to hear if you end up with any further
> information.
>
> -chris
I was asking this before and ask again, how do we know it's not a missing
RNG entropy while the crypto stuff of the Java VM is initialised?
Simon
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Jerry,
On 6/15/23 00:41, Jerry Malcolm wrote:
>
> On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>>
>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
>>> Rob,
>>>
>>> On 6/13/2023 11:34 AM, Rob Sargent wrote:
>>>> In /etc/rc.local I have:
>>>>>
>>>>> ----------------------
>>>>> sleep 120s
>>>>> systemctl start tomcat9
>>>>>
>>>>> ---------------------
>>>>>
>>>>> I put the sleep in back a couple of years ago that for some reason
>>>>> 'fixed' this same random, intermittent crypto file exception.
>>>>>
>>>>>
>>>>
>> One observation.... even though the error doesn't show up in the log
>> until my first db call, I think the real error has to be occurring
>> during tomcat boot. If the failure occurs, I continue to get the same
>> "Can't read cryptographic policy directory: unlimited" even hours
>> later after boot every time I send another request to TC. If I then
>> reboot TC and this time I don't get the failure, that unlimited folder
>> magically becomes available. There is something happening in TC boot
>> with some sort of crypto initialization that either succeeds or
>> fails. And the exception when trying to get connections long after TC
>> boot is just hitting whatever problem occurred during boot. In other
>> words, delaying my calls, even for hours, has no effect on accessing
>> that folder if the problem is present. Rebooting TC gives it another
>> chance to succeed or fail. Any ideas what TC could be doing on boot
>> that could lock up that folder?
>>
>>
> I never really found a definitive fix for this. But after all of the
> research and logging and everything, I pretty much concluded that
> there's some kind of timing/race condition between Tomcat boot and Java
> crypto initialization. Given that, I figured it was highly unlikely
> that two different Java JVM implementations would have the same precise
> implementation code to trigger that race condition. So as a last ditch
> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
> JVM. It's circumstantial evidence at best, and running a production
> environment on fixes that just went away goes against my grain. But if
> the problem goes away, maybe it won't come back. At this time, when
> using Corretto JVM, I have not encountered the Crypto directory error.
> It's been running on all server instances now for almost 24 hours with
> no problems. So I guess my suggestion to anyone else that might
> encounter something like this, trying using a different JVM from a
> different provider. That appears at this time to have worked for me.
Yeah, that's certainly a very unsatisfying result, but ... you can't
argue with results. I'd love to hear if you end up with any further
information.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
Posted by Simon Matter <si...@invoca.ch>.
>
> On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>>
>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
>>> Rob,
>>>
>>> On 6/13/2023 11:34 AM, Rob Sargent wrote:
>>>> In /etc/rc.local I have:
>>>>>
>>>>> ----------------------
>>>>> sleep 120s
>>>>> systemctl start tomcat9
>>>>>
>>>>> ---------------------
>>>>>
>>>>> I put the sleep in back a couple of years ago that for some reason
>>>>> 'fixed' this same random, intermittent crypto file exception.
>>>>>
>>>>>
>>>>
>> One observation.... even though the error doesn't show up in the log
>> until my first db call, I think the real error has to be occurring
>> during tomcat boot. If the failure occurs, I continue to get the same
>> "Can't read cryptographic policy directory: unlimited" even hours
>> later after boot every time I send another request to TC. If I then
>> reboot TC and this time I don't get the failure, that unlimited folder
>> magically becomes available. There is something happening in TC boot
>> with some sort of crypto initialization that either succeeds or
>> fails. And the exception when trying to get connections long after TC
>> boot is just hitting whatever problem occurred during boot. In other
>> words, delaying my calls, even for hours, has no effect on accessing
>> that folder if the problem is present. Rebooting TC gives it another
>> chance to succeed or fail. Any ideas what TC could be doing on boot
>> that could lock up that folder?
>>
>>
> I never really found a definitive fix for this. But after all of the
> research and logging and everything, I pretty much concluded that
> there's some kind of timing/race condition between Tomcat boot and Java
> crypto initialization. Given that, I figured it was highly unlikely
> that two different Java JVM implementations would have the same precise
> implementation code to trigger that race condition. So as a last ditch
> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
> JVM. It's circumstantial evidence at best, and running a production
> environment on fixes that just went away goes against my grain. But if
> the problem goes away, maybe it won't come back. At this time, when
> using Corretto JVM, I have not encountered the Crypto directory error.
> It's been running on all server instances now for almost 24 hours with
> no problems. So I guess my suggestion to anyone else that might
> encounter something like this, trying using a different JVM from a
> different provider. That appears at this time to have worked for me.
>
Maybe you could report this as a bug in the affected JVM? If it's really
coming from the JVM then it would be nice to have it fixed.
Regards,
Simon
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org