You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Jerry Malcolm <te...@malcolms.com> on 2023/06/13 06:49:54 UTC
Crypto Randomly Not Getting Initialized
I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
Linux2 in a production environment. A couple of years ago, we started
getting weird errors that the "Crypto Mechanism" failed to initialize.
Through a lot of trial and error, and reasons I don't quite remember, we
put a 2-min delay in rc.local before starting Tomcat, and the problem
went away. I'm not a Linux nor a crypto guru. But we traced it to some
crypto file that we assumed was not available until later in the Linux
boot sequence. Anyway, the 2 minute delay made it go away, for over two
years. Then all of a sudden in the last day or so, it's back with a
vengeance. It fails with the same crypto error from 2 years ago in
about 50% of the EC2 boot ups. I tried bumping the wait to 3 min, and
no change.
I need help. Our whole production environment is unstable now since
every time an ASG brings a new instance online, I've got a 50-50 chance
that tomcat is going to die (and the health check doesn't catch it, but
that's a different issue).
There are no errors in the Tomcat boot sequence logs. But the first
time and every subsequent time I try to get a connection from the
DataSource pool, I get the stack dump shown below.
I figure it has to be a timing/race condition. But I have no clue what
to do to fix it. I'm baffled that it worked for two years, and now
fails every other time I start an instance. And every instance is
running copies of the exact same Amazon Machine Image. The same EC2
will come up clean 50% of the time the next time it boots.
Can somebody with Tomcat/Crypto/Linux knowledge unravel what's going on
here? Thx
java.lang.ExceptionInInitializerError
at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540)
at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190)
at
java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCipher.java:509)
at java.base/sun.security.ssl.SSLCipher.<init>(SSLCipher.java:498)
at java.base/sun.security.ssl.SSLCipher.<clinit>(SSLCipher.java:81)
at
java.base/sun.security.ssl.CipherSuite.<clinit>(CipherSuite.java:65)
at
java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCipherSuites(SSLContextImpl.java:348)
at
java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.<clinit>(SSLContextImpl.java:580)
at java.base/java.lang.Class.forName0(Native Method)
at java.base/java.lang.Class.forName(Class.java:315)
...
at
com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:948)
at
com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818)
at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:448)
at
com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
at
com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198)
at
org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52)
at
org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:415)
at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:111)
at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:649)
at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:532)
at
org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:731)
at jwm.db.DBData.getConnection(DBData.java:506) //// my call
to get a db connection from connection pool ////
...
Caused by: java.lang.SecurityException: Can not initialize cryptographic
mechanism
at
java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120) ... 86 mo
Caused by: java.lang.SecurityException: Can't read cryptographic policy
directory: unlimited
at
java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:326)
at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111)
at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108)
at java.base/java.security.AccessController.doPrivileged(Native
Method)
at
java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:107)
... 86 more
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: AW: AW: Crypto Randomly Not Getting Initialized
Posted by Jerry Malcolm <te...@malcolms.com>.
Thomas,
On 6/13/2023 10:45 AM, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> Hello Jerry,
>
>
>> -----Ursprüngliche Nachricht-----
>> Von: Jerry Malcolm <te...@malcolms.com>
>> Gesendet: Dienstag, 13. Juni 2023 17:35
>> An: users@tomcat.apache.org
>> Betreff: Re: AW: Crypto Randomly Not Getting Initialized
>>
>> Hi Thomas,
>>
>> On 6/13/2023 2:08 AM, Thomas Hoffmann (Speed4Trade GmbH) wrote:
>>> Hello Jerry,
>>>
>>>> -----Ursprüngliche Nachricht-----
>>>> Von: Jerry Malcolm <te...@malcolms.com>
>>>> Gesendet: Dienstag, 13. Juni 2023 08:50
>>>> An: users@tomcat.apache.org
>>>> Betreff: Crypto Randomly Not Getting Initialized
>>>>
>>>> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
>>>> Linux2 in a production environment. A couple of years ago, we
>>>> started getting weird errors that the "Crypto Mechanism" failed to
>>>> initialize. Through a lot of trial and error, and reasons I don't
>>>> quite remember, we put a 2-min delay in rc.local before starting
>>>> Tomcat, and the problem went away. I'm not a Linux nor a crypto
>>>> guru. But we traced it to some crypto file that we assumed was not
>>>> available until later in the Linux boot sequence. Anyway, the 2
>>>> minute delay made it go away, for over two years. Then all of a
>>>> sudden in the last day or so, it's back with a vengeance. It fails
>>>> with the same crypto error from 2 years ago in about 50% of the EC2 boot
>> ups. I tried bumping the wait to 3 min, and no change.
>>>> I need help. Our whole production environment is unstable now since
>>>> every time an ASG brings a new instance online, I've got a 50-50
>>>> chance that tomcat is going to die (and the health check doesn't
>>>> catch it, but that's a different issue).
>>>>
>>>> There are no errors in the Tomcat boot sequence logs. But the first
>>>> time and every subsequent time I try to get a connection from the
>>>> DataSource pool, I get the stack dump shown below.
>>>>
>>>> I figure it has to be a timing/race condition. But I have no clue
>>>> what to do to fix it. I'm baffled that it worked for two years, and
>>>> now fails every other time I start an instance. And every instance
>>>> is running copies of the exact same Amazon Machine Image. The same
>>>> EC2 will come up clean 50% of the time the next time it boots.
>>>>
>>>> Can somebody with Tomcat/Crypto/Linux knowledge unravel what's
>> going
>>>> on here? Thx
>>>>
>>>> java.lang.ExceptionInInitializerError
>>>> at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540)
>>>> at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190)
>>>> at
>>>> java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCip
>>>> her.jav
>>>> a:509)
>>>> at java.base/sun.security.ssl.SSLCipher.<init>(SSLCipher.java:498)
>>>> at java.base/sun.security.ssl.SSLCipher.<clinit>(SSLCipher.java:81)
>>>> at
>>>> java.base/sun.security.ssl.CipherSuite.<clinit>(CipherSuite.java:65)
>>>> at
>>>> java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCiphe
>>>> rSuit
>>>> es(SSLContextImpl.java:348)
>>>> at
>>>> java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.<clinit>
>>>> (SSLC
>>>> ontextImpl.java:580)
>>>> at java.base/java.lang.Class.forName0(Native Method)
>>>> at java.base/java.lang.Class.forName(Class.java:315)
>>>> ...
>>>>
>>>> at
>>>>
>> com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:
>>>> 948)
>>>> at
>>>>
>> com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818)
>>>> at
>> com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:448)
>>>> at
>>>> com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
>>>> at
>>>>
>> com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:
>>>> 198)
>>>> at
>>>>
>> org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection
>>>> (
>>>> DriverConnectionFactory.java:52)
>>>> at
>>>>
>> org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Po
>>>> olableConnectionFactory.java:415)
>>>> at
>>>>
>> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactor
>>>> y
>>>> (BasicDataSource.java:111)
>>>> at
>>>>
>> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnection
>>>> Factory(BasicDataSource.java:649)
>>>> at
>>>>
>> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDa
>>>> taSource.java:532)
>>>> at
>>>>
>> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataS
>>>> ource.java:731)
>>>> at jwm.db.DBData.getConnection(DBData.java:506) //// my
>>>> call to get a db connection from connection pool ////
>>>>
>>>> ...
>>>>
>>>> Caused by: java.lang.SecurityException: Can not initialize
>>>> cryptographic mechanism
>>>> at
>>>> java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120) ...
>>>> 86 mo Caused by: java.lang.SecurityException: Can't read
>>>> cryptographic policy
>>>> directory: unlimited
>>>> at
>>>>
>> java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:
>>>> 326)
>>>> at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111)
>>>> at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108)
>>>> at
>>>> java.base/java.security.AccessController.doPrivileged(Native
>>>> Method)
>>>> at
>>>> java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:107)
>>>> ... 86 more
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>>>> For additional commands, e-mail: users-help@tomcat.apache.org
>>> Could it be this issue?
>>> https://github.com/docker-library/openjdk/issues/101
>>>
>>> Maybe you can add information about the used jdk and whether you are
>> using containers.
>>> Greetings,
>>> Thomas
>> I'm running Java 11.0.16. No Docker or other containers. Just straight
>> Tomcat running standalone. The github link refers to a Java 9 / Docker
>> issue. I guess it could be related. But I'm not sure due to different
>> environments, and my situation only fails half the time. Thx
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> Maybe you can compare the available ciphers during runtime:
> https://stackoverflow.com/questions/9333504/how-can-i-list-the-available-cipher-algorithms
> Print all ciphers in both environments and check if there are differences.
>
> System properties are also worth comparing:
> https://mkyong.com/java/how-to-list-all-system-properties-key-and-value-in-java/
>
> Greetings! Thomas
There is only one environment. I have the same image copied to all
servers. All servers are identical EC2 types. On EC2 boot, about half
of them throw the exception and the other half boot fine. Then reboot
everything, some of those that failed boot fine and some of the ones
that worked failed. Basically it's one environment that has a 50-50
chance of failing on a boot. And this just started failing after two
years of working fine with no (obvious) changes to TC, Liinux, or
anything else. My TC webapp code is changing. But it doesn't seem like
there would be anything a webapp could do to start this random failing (??)
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: AW: Crypto Randomly Not Getting Initialized
Posted by "Thomas Hoffmann (Speed4Trade GmbH)" <Th...@speed4trade.com.INVALID>.
Hello Jerry,
> -----Ursprüngliche Nachricht-----
> Von: Jerry Malcolm <te...@malcolms.com>
> Gesendet: Dienstag, 13. Juni 2023 17:35
> An: users@tomcat.apache.org
> Betreff: Re: AW: Crypto Randomly Not Getting Initialized
>
> Hi Thomas,
>
> On 6/13/2023 2:08 AM, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> > Hello Jerry,
> >
> >> -----Ursprüngliche Nachricht-----
> >> Von: Jerry Malcolm <te...@malcolms.com>
> >> Gesendet: Dienstag, 13. Juni 2023 08:50
> >> An: users@tomcat.apache.org
> >> Betreff: Crypto Randomly Not Getting Initialized
> >>
> >> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
> >> Linux2 in a production environment. A couple of years ago, we
> >> started getting weird errors that the "Crypto Mechanism" failed to
> >> initialize. Through a lot of trial and error, and reasons I don't
> >> quite remember, we put a 2-min delay in rc.local before starting
> >> Tomcat, and the problem went away. I'm not a Linux nor a crypto
> >> guru. But we traced it to some crypto file that we assumed was not
> >> available until later in the Linux boot sequence. Anyway, the 2
> >> minute delay made it go away, for over two years. Then all of a
> >> sudden in the last day or so, it's back with a vengeance. It fails
> >> with the same crypto error from 2 years ago in about 50% of the EC2 boot
> ups. I tried bumping the wait to 3 min, and no change.
> >>
> >> I need help. Our whole production environment is unstable now since
> >> every time an ASG brings a new instance online, I've got a 50-50
> >> chance that tomcat is going to die (and the health check doesn't
> >> catch it, but that's a different issue).
> >>
> >> There are no errors in the Tomcat boot sequence logs. But the first
> >> time and every subsequent time I try to get a connection from the
> >> DataSource pool, I get the stack dump shown below.
> >>
> >> I figure it has to be a timing/race condition. But I have no clue
> >> what to do to fix it. I'm baffled that it worked for two years, and
> >> now fails every other time I start an instance. And every instance
> >> is running copies of the exact same Amazon Machine Image. The same
> >> EC2 will come up clean 50% of the time the next time it boots.
> >>
> >> Can somebody with Tomcat/Crypto/Linux knowledge unravel what's
> going
> >> on here? Thx
> >>
> >> java.lang.ExceptionInInitializerError
> >> at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540)
> >> at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190)
> >> at
> >> java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCip
> >> her.jav
> >> a:509)
> >> at java.base/sun.security.ssl.SSLCipher.<init>(SSLCipher.java:498)
> >> at java.base/sun.security.ssl.SSLCipher.<clinit>(SSLCipher.java:81)
> >> at
> >> java.base/sun.security.ssl.CipherSuite.<clinit>(CipherSuite.java:65)
> >> at
> >> java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCiphe
> >> rSuit
> >> es(SSLContextImpl.java:348)
> >> at
> >> java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.<clinit>
> >> (SSLC
> >> ontextImpl.java:580)
> >> at java.base/java.lang.Class.forName0(Native Method)
> >> at java.base/java.lang.Class.forName(Class.java:315)
> >> ...
> >>
> >> at
> >>
> com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:
> >> 948)
> >> at
> >>
> com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818)
> >> at
> com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:448)
> >> at
> >> com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
> >> at
> >>
> com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:
> >> 198)
> >> at
> >>
> org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection
> >> (
> >> DriverConnectionFactory.java:52)
> >> at
> >>
> org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Po
> >> olableConnectionFactory.java:415)
> >> at
> >>
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactor
> >> y
> >> (BasicDataSource.java:111)
> >> at
> >>
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnection
> >> Factory(BasicDataSource.java:649)
> >> at
> >>
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDa
> >> taSource.java:532)
> >> at
> >>
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataS
> >> ource.java:731)
> >> at jwm.db.DBData.getConnection(DBData.java:506) //// my
> >> call to get a db connection from connection pool ////
> >>
> >> ...
> >>
> >> Caused by: java.lang.SecurityException: Can not initialize
> >> cryptographic mechanism
> >> at
> >> java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120) ...
> >> 86 mo Caused by: java.lang.SecurityException: Can't read
> >> cryptographic policy
> >> directory: unlimited
> >> at
> >>
> java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:
> >> 326)
> >> at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111)
> >> at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108)
> >> at
> >> java.base/java.security.AccessController.doPrivileged(Native
> >> Method)
> >> at
> >> java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:107)
> >> ... 86 more
> >>
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> >> For additional commands, e-mail: users-help@tomcat.apache.org
> > Could it be this issue?
> > https://github.com/docker-library/openjdk/issues/101
> >
> > Maybe you can add information about the used jdk and whether you are
> using containers.
> >
> > Greetings,
> > Thomas
> I'm running Java 11.0.16. No Docker or other containers. Just straight
> Tomcat running standalone. The github link refers to a Java 9 / Docker
> issue. I guess it could be related. But I'm not sure due to different
> environments, and my situation only fails half the time. Thx
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
Maybe you can compare the available ciphers during runtime:
https://stackoverflow.com/questions/9333504/how-can-i-list-the-available-cipher-algorithms
Print all ciphers in both environments and check if there are differences.
System properties are also worth comparing:
https://mkyong.com/java/how-to-list-all-system-properties-key-and-value-in-java/
Greetings! Thomas
Re: AW: Crypto Randomly Not Getting Initialized
Posted by Jerry Malcolm <te...@malcolms.com>.
Hi Thomas,
On 6/13/2023 2:08 AM, Thomas Hoffmann (Speed4Trade GmbH) wrote:
> Hello Jerry,
>
>> -----Ursprüngliche Nachricht-----
>> Von: Jerry Malcolm <te...@malcolms.com>
>> Gesendet: Dienstag, 13. Juni 2023 08:50
>> An: users@tomcat.apache.org
>> Betreff: Crypto Randomly Not Getting Initialized
>>
>> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
>> Linux2 in a production environment. A couple of years ago, we started
>> getting weird errors that the "Crypto Mechanism" failed to initialize. Through
>> a lot of trial and error, and reasons I don't quite remember, we put a 2-min
>> delay in rc.local before starting Tomcat, and the problem went away. I'm
>> not a Linux nor a crypto guru. But we traced it to some crypto file that we
>> assumed was not available until later in the Linux boot sequence. Anyway,
>> the 2 minute delay made it go away, for over two years. Then all of a sudden
>> in the last day or so, it's back with a vengeance. It fails with the same crypto
>> error from 2 years ago in about 50% of the EC2 boot ups. I tried bumping the
>> wait to 3 min, and no change.
>>
>> I need help. Our whole production environment is unstable now since every
>> time an ASG brings a new instance online, I've got a 50-50 chance that
>> tomcat is going to die (and the health check doesn't catch it, but that's a
>> different issue).
>>
>> There are no errors in the Tomcat boot sequence logs. But the first time and
>> every subsequent time I try to get a connection from the DataSource pool, I
>> get the stack dump shown below.
>>
>> I figure it has to be a timing/race condition. But I have no clue what to do to
>> fix it. I'm baffled that it worked for two years, and now fails every other
>> time I start an instance. And every instance is running copies of the exact
>> same Amazon Machine Image. The same EC2 will come up clean 50% of the
>> time the next time it boots.
>>
>> Can somebody with Tomcat/Crypto/Linux knowledge unravel what's going
>> on here? Thx
>>
>> java.lang.ExceptionInInitializerError
>> at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540)
>> at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190)
>> at
>> java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCipher.jav
>> a:509)
>> at java.base/sun.security.ssl.SSLCipher.<init>(SSLCipher.java:498)
>> at java.base/sun.security.ssl.SSLCipher.<clinit>(SSLCipher.java:81)
>> at
>> java.base/sun.security.ssl.CipherSuite.<clinit>(CipherSuite.java:65)
>> at
>> java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCipherSuit
>> es(SSLContextImpl.java:348)
>> at
>> java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.<clinit>(SSLC
>> ontextImpl.java:580)
>> at java.base/java.lang.Class.forName0(Native Method)
>> at java.base/java.lang.Class.forName(Class.java:315)
>> ...
>>
>> at
>> com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:
>> 948)
>> at
>> com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818)
>> at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:448)
>> at
>> com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
>> at
>> com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:
>> 198)
>> at
>> org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(
>> DriverConnectionFactory.java:52)
>> at
>> org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Po
>> olableConnectionFactory.java:415)
>> at
>> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactory
>> (BasicDataSource.java:111)
>> at
>> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnection
>> Factory(BasicDataSource.java:649)
>> at
>> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDa
>> taSource.java:532)
>> at
>> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataS
>> ource.java:731)
>> at jwm.db.DBData.getConnection(DBData.java:506) //// my call to get
>> a db connection from connection pool ////
>>
>> ...
>>
>> Caused by: java.lang.SecurityException: Can not initialize cryptographic
>> mechanism
>> at
>> java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120) ... 86 mo
>> Caused by: java.lang.SecurityException: Can't read cryptographic policy
>> directory: unlimited
>> at
>> java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:
>> 326)
>> at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111)
>> at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108)
>> at java.base/java.security.AccessController.doPrivileged(Native
>> Method)
>> at
>> java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:107)
>> ... 86 more
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
>> For additional commands, e-mail: users-help@tomcat.apache.org
> Could it be this issue?
> https://github.com/docker-library/openjdk/issues/101
>
> Maybe you can add information about the used jdk and whether you are using containers.
>
> Greetings,
> Thomas
I'm running Java 11.0.16. No Docker or other containers. Just straight
Tomcat running standalone. The github link refers to a Java 9 / Docker
issue. I guess it could be related. But I'm not sure due to different
environments, and my situation only fails half the time. Thx
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
AW: Crypto Randomly Not Getting Initialized
Posted by "Thomas Hoffmann (Speed4Trade GmbH)" <Th...@speed4trade.com.INVALID>.
Hello Jerry,
> -----Ursprüngliche Nachricht-----
> Von: Jerry Malcolm <te...@malcolms.com>
> Gesendet: Dienstag, 13. Juni 2023 08:50
> An: users@tomcat.apache.org
> Betreff: Crypto Randomly Not Getting Initialized
>
> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
> Linux2 in a production environment. A couple of years ago, we started
> getting weird errors that the "Crypto Mechanism" failed to initialize. Through
> a lot of trial and error, and reasons I don't quite remember, we put a 2-min
> delay in rc.local before starting Tomcat, and the problem went away. I'm
> not a Linux nor a crypto guru. But we traced it to some crypto file that we
> assumed was not available until later in the Linux boot sequence. Anyway,
> the 2 minute delay made it go away, for over two years. Then all of a sudden
> in the last day or so, it's back with a vengeance. It fails with the same crypto
> error from 2 years ago in about 50% of the EC2 boot ups. I tried bumping the
> wait to 3 min, and no change.
>
> I need help. Our whole production environment is unstable now since every
> time an ASG brings a new instance online, I've got a 50-50 chance that
> tomcat is going to die (and the health check doesn't catch it, but that's a
> different issue).
>
> There are no errors in the Tomcat boot sequence logs. But the first time and
> every subsequent time I try to get a connection from the DataSource pool, I
> get the stack dump shown below.
>
> I figure it has to be a timing/race condition. But I have no clue what to do to
> fix it. I'm baffled that it worked for two years, and now fails every other
> time I start an instance. And every instance is running copies of the exact
> same Amazon Machine Image. The same EC2 will come up clean 50% of the
> time the next time it boots.
>
> Can somebody with Tomcat/Crypto/Linux knowledge unravel what's going
> on here? Thx
>
> java.lang.ExceptionInInitializerError
> at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540)
> at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:190)
> at
> java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCipher.jav
> a:509)
> at java.base/sun.security.ssl.SSLCipher.<init>(SSLCipher.java:498)
> at java.base/sun.security.ssl.SSLCipher.<clinit>(SSLCipher.java:81)
> at
> java.base/sun.security.ssl.CipherSuite.<clinit>(CipherSuite.java:65)
> at
> java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCipherSuit
> es(SSLContextImpl.java:348)
> at
> java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.<clinit>(SSLC
> ontextImpl.java:580)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Class.java:315)
> ...
>
> at
> com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:
> 948)
> at
> com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:818)
> at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:448)
> at
> com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:241)
> at
> com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:
> 198)
> at
> org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(
> DriverConnectionFactory.java:52)
> at
> org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(Po
> olableConnectionFactory.java:415)
> at
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactory
> (BasicDataSource.java:111)
> at
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnection
> Factory(BasicDataSource.java:649)
> at
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDa
> taSource.java:532)
> at
> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataS
> ource.java:731)
> at jwm.db.DBData.getConnection(DBData.java:506) //// my call to get
> a db connection from connection pool ////
>
> ...
>
> Caused by: java.lang.SecurityException: Can not initialize cryptographic
> mechanism
> at
> java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120) ... 86 mo
> Caused by: java.lang.SecurityException: Can't read cryptographic policy
> directory: unlimited
> at
> java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:
> 326)
> at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111)
> at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108)
> at java.base/java.security.AccessController.doPrivileged(Native
> Method)
> at
> java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:107)
> ... 86 more
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
Could it be this issue?
https://github.com/docker-library/openjdk/issues/101
Maybe you can add information about the used jdk and whether you are using containers.
Greetings,
Thomas
Re: Crypto Randomly Not Getting Initialized
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Jerry,
On 6/13/23 12:25, Jerry Malcolm wrote:
> Chris,
>
> On 6/13/2023 11:03 AM, Christopher Schultz wrote:
>> Jerry,
>>
>> On 6/13/23 11:51, Jerry Malcolm wrote:
>>> Reg,
>>>
>>> On 6/13/2023 2:57 AM, r.barclay@habmalnefrage.de wrote:
>>>> Hey Jerry,
>>>>
>>>>> Caused by: java.lang.SecurityException: Can not initialize
>>>>> cryptographic
>>>>> mechanism
>>>>> at
>>>>> java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120)
>>>>> ... 86 mo
>>>>> Caused by: java.lang.SecurityException: Can't read cryptographic
>>>>> policy
>>>>> directory: unlimited
>>>> Does this help?
>>>> https://stackoverflow.com/a/58183460/
>>>>
>>>> Yours,
>>>> Reg
>>> The stackoverflow link provides a lot of useful info about the error.
>>> But unless something in the boot process deletes those files and
>>> recreates them later, they are all there and correct. One boot works
>>> fine. The next boot throws this error. The next boot is fine. Is
>>> there a possibility that permissions get altered on this 'unlimited'
>>> folder during boot? I can't think of many reasons it would say a
>>> folder doesn't exist that clearly does exist.
>>
>> How are you launching Tomcat on boot?
>>
>> Are you able to add some instrumentation/logging to that?
>>
>> -chris
>>
> In /etc/rc.local I have:
>
> ----------------------
> sleep 120s
> systemctl start tomcat9
That's an odd way to do things (in rc.local).
If you have systemctl, you are using systemd. If you are using systemd,
you isn't it auto-starting that service?
$ sudo systemctl enable tomcat9
That ought to do it, and the tomcat9 service should start on startup,
resolving any dependencies it needs.
Are you able to do an "ls [path to policy files] > /tmp/some_log_file"
in there to see the situation when the service is trying to start?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Simon,
On 6/21/23 03:19, Simon Matter wrote:
>> Jerry,
>>
>> On 6/15/23 00:41, Jerry Malcolm wrote:
>>>
>>> On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>>>>
>>>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
>>>>> Rob,
>>>>>
>>>>> On 6/13/2023 11:34 AM, Rob Sargent wrote:
>>>>>> In /etc/rc.local I have:
>>>>>>>
>>>>>>> ----------------------
>>>>>>> sleep 120s
>>>>>>> systemctl start tomcat9
>>>>>>>
>>>>>>> ---------------------
>>>>>>>
>>>>>>> I put the sleep in back a couple of years ago that for some reason
>>>>>>> 'fixed' this same random, intermittent crypto file exception.
>>>>>>>
>>>>>>>
>>>>>>
>>>> One observation.... even though the error doesn't show up in the log
>>>> until my first db call, I think the real error has to be occurring
>>>> during tomcat boot. If the failure occurs, I continue to get the same
>>>> "Can't read cryptographic policy directory: unlimited" even hours
>>>> later after boot every time I send another request to TC. If I then
>>>> reboot TC and this time I don't get the failure, that unlimited folder
>>>> magically becomes available. There is something happening in TC boot
>>>> with some sort of crypto initialization that either succeeds or
>>>> fails. And the exception when trying to get connections long after TC
>>>> boot is just hitting whatever problem occurred during boot. In other
>>>> words, delaying my calls, even for hours, has no effect on accessing
>>>> that folder if the problem is present. Rebooting TC gives it another
>>>> chance to succeed or fail. Any ideas what TC could be doing on boot
>>>> that could lock up that folder?
>>>>
>>>>
>>> I never really found a definitive fix for this. But after all of the
>>> research and logging and everything, I pretty much concluded that
>>> there's some kind of timing/race condition between Tomcat boot and Java
>>> crypto initialization. Given that, I figured it was highly unlikely
>>> that two different Java JVM implementations would have the same precise
>>> implementation code to trigger that race condition. So as a last ditch
>>> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
>>> JVM. It's circumstantial evidence at best, and running a production
>>> environment on fixes that just went away goes against my grain. But if
>>> the problem goes away, maybe it won't come back. At this time, when
>>> using Corretto JVM, I have not encountered the Crypto directory error.
>>> It's been running on all server instances now for almost 24 hours with
>>> no problems. So I guess my suggestion to anyone else that might
>>> encounter something like this, trying using a different JVM from a
>>> different provider. That appears at this time to have worked for me.
>>
>> Yeah, that's certainly a very unsatisfying result, but ... you can't
>> argue with results. I'd love to hear if you end up with any further
>> information.
>>
>> -chris
>
> I was asking this before and ask again, how do we know it's not a missing
> RNG entropy while the crypto stuff of the Java VM is initialised?
Because the symptoms of *that* problem (very long lags when requesting
random numbers) are not being experienced. And also because the error
message is crystal clear:
[...]
Caused by: java.lang.SecurityException: Can not initialize cryptographic
mechanism
at
java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120) ... 86 mo
Caused by: java.lang.SecurityException: Can't read cryptographic policy
directory: unlimited
at
java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:326)
[...]
The cryptographic provider itself cannot even initialize. It's *way*
before any randomness is needed. The "unlimited" directory is presumably
the directory which contains the policy files which do not impose the
archaic US-imposed "export restrictions" on cryptographic primitives.
The Provider cannot find the configuration directory, and fails.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
Posted by Simon Matter <si...@invoca.ch>.
> Jerry,
>
> On 6/15/23 00:41, Jerry Malcolm wrote:
>>
>> On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>>>
>>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
>>>> Rob,
>>>>
>>>> On 6/13/2023 11:34 AM, Rob Sargent wrote:
>>>>> In /etc/rc.local I have:
>>>>>>
>>>>>> ----------------------
>>>>>> sleep 120s
>>>>>> systemctl start tomcat9
>>>>>>
>>>>>> ---------------------
>>>>>>
>>>>>> I put the sleep in back a couple of years ago that for some reason
>>>>>> 'fixed' this same random, intermittent crypto file exception.
>>>>>>
>>>>>>
>>>>>
>>> One observation.... even though the error doesn't show up in the log
>>> until my first db call, I think the real error has to be occurring
>>> during tomcat boot. If the failure occurs, I continue to get the same
>>> "Can't read cryptographic policy directory: unlimited" even hours
>>> later after boot every time I send another request to TC. If I then
>>> reboot TC and this time I don't get the failure, that unlimited folder
>>> magically becomes available. There is something happening in TC boot
>>> with some sort of crypto initialization that either succeeds or
>>> fails. And the exception when trying to get connections long after TC
>>> boot is just hitting whatever problem occurred during boot. In other
>>> words, delaying my calls, even for hours, has no effect on accessing
>>> that folder if the problem is present. Rebooting TC gives it another
>>> chance to succeed or fail. Any ideas what TC could be doing on boot
>>> that could lock up that folder?
>>>
>>>
>> I never really found a definitive fix for this. But after all of the
>> research and logging and everything, I pretty much concluded that
>> there's some kind of timing/race condition between Tomcat boot and Java
>> crypto initialization. Given that, I figured it was highly unlikely
>> that two different Java JVM implementations would have the same precise
>> implementation code to trigger that race condition. So as a last ditch
>> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
>> JVM. It's circumstantial evidence at best, and running a production
>> environment on fixes that just went away goes against my grain. But if
>> the problem goes away, maybe it won't come back. At this time, when
>> using Corretto JVM, I have not encountered the Crypto directory error.
>> It's been running on all server instances now for almost 24 hours with
>> no problems. So I guess my suggestion to anyone else that might
>> encounter something like this, trying using a different JVM from a
>> different provider. That appears at this time to have worked for me.
>
> Yeah, that's certainly a very unsatisfying result, but ... you can't
> argue with results. I'd love to hear if you end up with any further
> information.
>
> -chris
I was asking this before and ask again, how do we know it's not a missing
RNG entropy while the crypto stuff of the Java VM is initialised?
Simon
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Jerry,
On 6/15/23 00:41, Jerry Malcolm wrote:
>
> On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>>
>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
>>> Rob,
>>>
>>> On 6/13/2023 11:34 AM, Rob Sargent wrote:
>>>> In /etc/rc.local I have:
>>>>>
>>>>> ----------------------
>>>>> sleep 120s
>>>>> systemctl start tomcat9
>>>>>
>>>>> ---------------------
>>>>>
>>>>> I put the sleep in back a couple of years ago that for some reason
>>>>> 'fixed' this same random, intermittent crypto file exception.
>>>>>
>>>>>
>>>>
>> One observation.... even though the error doesn't show up in the log
>> until my first db call, I think the real error has to be occurring
>> during tomcat boot. If the failure occurs, I continue to get the same
>> "Can't read cryptographic policy directory: unlimited" even hours
>> later after boot every time I send another request to TC. If I then
>> reboot TC and this time I don't get the failure, that unlimited folder
>> magically becomes available. There is something happening in TC boot
>> with some sort of crypto initialization that either succeeds or
>> fails. And the exception when trying to get connections long after TC
>> boot is just hitting whatever problem occurred during boot. In other
>> words, delaying my calls, even for hours, has no effect on accessing
>> that folder if the problem is present. Rebooting TC gives it another
>> chance to succeed or fail. Any ideas what TC could be doing on boot
>> that could lock up that folder?
>>
>>
> I never really found a definitive fix for this. But after all of the
> research and logging and everything, I pretty much concluded that
> there's some kind of timing/race condition between Tomcat boot and Java
> crypto initialization. Given that, I figured it was highly unlikely
> that two different Java JVM implementations would have the same precise
> implementation code to trigger that race condition. So as a last ditch
> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
> JVM. It's circumstantial evidence at best, and running a production
> environment on fixes that just went away goes against my grain. But if
> the problem goes away, maybe it won't come back. At this time, when
> using Corretto JVM, I have not encountered the Crypto directory error.
> It's been running on all server instances now for almost 24 hours with
> no problems. So I guess my suggestion to anyone else that might
> encounter something like this, trying using a different JVM from a
> different provider. That appears at this time to have worked for me.
Yeah, that's certainly a very unsatisfying result, but ... you can't
argue with results. I'd love to hear if you end up with any further
information.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Conclusion - Re: Crypto Randomly Not Getting Initialized
Posted by Simon Matter <si...@invoca.ch>.
>
> On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>>
>> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
>>> Rob,
>>>
>>> On 6/13/2023 11:34 AM, Rob Sargent wrote:
>>>> In /etc/rc.local I have:
>>>>>
>>>>> ----------------------
>>>>> sleep 120s
>>>>> systemctl start tomcat9
>>>>>
>>>>> ---------------------
>>>>>
>>>>> I put the sleep in back a couple of years ago that for some reason
>>>>> 'fixed' this same random, intermittent crypto file exception.
>>>>>
>>>>>
>>>>
>> One observation.... even though the error doesn't show up in the log
>> until my first db call, I think the real error has to be occurring
>> during tomcat boot. If the failure occurs, I continue to get the same
>> "Can't read cryptographic policy directory: unlimited" even hours
>> later after boot every time I send another request to TC. If I then
>> reboot TC and this time I don't get the failure, that unlimited folder
>> magically becomes available. There is something happening in TC boot
>> with some sort of crypto initialization that either succeeds or
>> fails. And the exception when trying to get connections long after TC
>> boot is just hitting whatever problem occurred during boot. In other
>> words, delaying my calls, even for hours, has no effect on accessing
>> that folder if the problem is present. Rebooting TC gives it another
>> chance to succeed or fail. Any ideas what TC could be doing on boot
>> that could lock up that folder?
>>
>>
> I never really found a definitive fix for this. But after all of the
> research and logging and everything, I pretty much concluded that
> there's some kind of timing/race condition between Tomcat boot and Java
> crypto initialization. Given that, I figured it was highly unlikely
> that two different Java JVM implementations would have the same precise
> implementation code to trigger that race condition. So as a last ditch
> effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
> JVM. It's circumstantial evidence at best, and running a production
> environment on fixes that just went away goes against my grain. But if
> the problem goes away, maybe it won't come back. At this time, when
> using Corretto JVM, I have not encountered the Crypto directory error.
> It's been running on all server instances now for almost 24 hours with
> no problems. So I guess my suggestion to anyone else that might
> encounter something like this, trying using a different JVM from a
> different provider. That appears at this time to have worked for me.
>
Maybe you could report this as a bug in the affected JVM? If it's really
coming from the JVM then it would be nice to have it fixed.
Regards,
Simon
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Conclusion - Re: Crypto Randomly Not Getting Initialized
Posted by Jerry Malcolm <te...@malcolms.com>.
On 6/13/2023 3:46 PM, Jerry Malcolm wrote:
>
> On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
>> Rob,
>>
>> On 6/13/2023 11:34 AM, Rob Sargent wrote:
>>> In /etc/rc.local I have:
>>>>
>>>> ----------------------
>>>> sleep 120s
>>>> systemctl start tomcat9
>>>>
>>>> ---------------------
>>>>
>>>> I put the sleep in back a couple of years ago that for some reason
>>>> 'fixed' this same random, intermittent crypto file exception.
>>>>
>>>>
>>>
> One observation.... even though the error doesn't show up in the log
> until my first db call, I think the real error has to be occurring
> during tomcat boot. If the failure occurs, I continue to get the same
> "Can't read cryptographic policy directory: unlimited" even hours
> later after boot every time I send another request to TC. If I then
> reboot TC and this time I don't get the failure, that unlimited folder
> magically becomes available. There is something happening in TC boot
> with some sort of crypto initialization that either succeeds or
> fails. And the exception when trying to get connections long after TC
> boot is just hitting whatever problem occurred during boot. In other
> words, delaying my calls, even for hours, has no effect on accessing
> that folder if the problem is present. Rebooting TC gives it another
> chance to succeed or fail. Any ideas what TC could be doing on boot
> that could lock up that folder?
>
>
I never really found a definitive fix for this. But after all of the
research and logging and everything, I pretty much concluded that
there's some kind of timing/race condition between Tomcat boot and Java
crypto initialization. Given that, I figured it was highly unlikely
that two different Java JVM implementations would have the same precise
implementation code to trigger that race condition. So as a last ditch
effort, I replaced my OpenJDK JVM for Tomcat with Amazon's Corretto Java
JVM. It's circumstantial evidence at best, and running a production
environment on fixes that just went away goes against my grain. But if
the problem goes away, maybe it won't come back. At this time, when
using Corretto JVM, I have not encountered the Crypto directory error.
It's been running on all server instances now for almost 24 hours with
no problems. So I guess my suggestion to anyone else that might
encounter something like this, trying using a different JVM from a
different provider. That appears at this time to have worked for me.
Thanks for all the support.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Posted by Jerry Malcolm <te...@malcolms.com>.
On 6/13/2023 12:39 PM, Jerry Malcolm wrote:
> Rob,
>
> On 6/13/2023 11:34 AM, Rob Sargent wrote:
>> In /etc/rc.local I have:
>>>
>>> ----------------------
>>> sleep 120s
>>> systemctl start tomcat9
>>>
>>> ---------------------
>>>
>>> I put the sleep in back a couple of years ago that for some reason
>>> 'fixed' this same random, intermittent crypto file exception.
>>>
>>>
>>
One observation.... even though the error doesn't show up in the log
until my first db call, I think the real error has to be occurring
during tomcat boot. If the failure occurs, I continue to get the same
"Can't read cryptographic policy directory: unlimited" even hours later
after boot every time I send another request to TC. If I then reboot TC
and this time I don't get the failure, that unlimited folder magically
becomes available. There is something happening in TC boot with some
sort of crypto initialization that either succeeds or fails. And the
exception when trying to get connections long after TC boot is just
hitting whatever problem occurred during boot. In other words, delaying
my calls, even for hours, has no effect on accessing that folder if the
problem is present. Rebooting TC gives it another chance to succeed or
fail. Any ideas what TC could be doing on boot that could lock up that
folder?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Posted by Jerry Malcolm <te...@malcolms.com>.
Rob,
On 6/13/2023 11:34 AM, Rob Sargent wrote:
> In /etc/rc.local I have:
>>
>> ----------------------
>> sleep 120s
>> systemctl start tomcat9
>>
>> ---------------------
>>
>> I put the sleep in back a couple of years ago that for some reason
>> 'fixed' this same random, intermittent crypto file exception.
>>
>>
>
> You mentioned earlier that you had recently tried a 3 minute delay, to
> no effect. Did you try any delay longer than 3 minutes? Maybe
> something ridiculous like 15m? It could be that whatever was taking
> two minutes before is now taking much longer.
>
> No change in file permissions at or above the "unlimited" dir?
>
>
I put the time back to 2 min after the 3 min had no effect. But after
the failure occurred on an instance, and I got to all of the logs, I
just did a tomcat reboot from the command line and it worked. So in
that case 2-3 min failed. But several hours later it worked. Still not
what could cause a "missing" folder to finally become available after
several minutes (or hours). Definitely not doing anything with
permissions on that file. Didn't even know it existed until this
problem surfaced.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Posted by Rob Sargent <rs...@xmission.com>.
In /etc/rc.local I have:
>
> ----------------------
> sleep 120s
> systemctl start tomcat9
>
> ---------------------
>
> I put the sleep in back a couple of years ago that for some reason
> 'fixed' this same random, intermittent crypto file exception.
>
>
You mentioned earlier that you had recently tried a 3 minute delay, to
no effect. Did you try any delay longer than 3 minutes? Maybe something
ridiculous like 15m? It could be that whatever was taking two minutes
before is now taking much longer.
No change in file permissions at or above the "unlimited" dir?
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Posted by Jerry Malcolm <te...@malcolms.com>.
Chris,
On 6/13/2023 11:03 AM, Christopher Schultz wrote:
> Jerry,
>
> On 6/13/23 11:51, Jerry Malcolm wrote:
>> Reg,
>>
>> On 6/13/2023 2:57 AM, r.barclay@habmalnefrage.de wrote:
>>> Hey Jerry,
>>>
>>>> Caused by: java.lang.SecurityException: Can not initialize
>>>> cryptographic
>>>> mechanism
>>>> at
>>>> java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120)
>>>> ... 86 mo
>>>> Caused by: java.lang.SecurityException: Can't read cryptographic
>>>> policy
>>>> directory: unlimited
>>> Does this help?
>>> https://stackoverflow.com/a/58183460/
>>>
>>> Yours,
>>> Reg
>> The stackoverflow link provides a lot of useful info about the error.
>> But unless something in the boot process deletes those files and
>> recreates them later, they are all there and correct. One boot works
>> fine. The next boot throws this error. The next boot is fine. Is
>> there a possibility that permissions get altered on this 'unlimited'
>> folder during boot? I can't think of many reasons it would say a
>> folder doesn't exist that clearly does exist.
>
> How are you launching Tomcat on boot?
>
> Are you able to add some instrumentation/logging to that?
>
> -chris
>
In /etc/rc.local I have:
----------------------
sleep 120s
systemctl start tomcat9
---------------------
I put the sleep in back a couple of years ago that for some reason
'fixed' this same random, intermittent crypto file exception.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Posted by Jerry Malcolm <te...@malcolms.com>.
chris
On 6/13/2023 11:03 AM, Christopher Schultz wrote:
> Jerry,
>
> On 6/13/23 11:51, Jerry Malcolm wrote:
>> Reg,
>>
>> On 6/13/2023 2:57 AM, r.barclay@habmalnefrage.de wrote:
>>> Hey Jerry,
>>>
>>>> Caused by: java.lang.SecurityException: Can not initialize
>>>> cryptographic
>>>> mechanism
>>>> at
>>>> java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120)
>>>> ... 86 mo
>>>> Caused by: java.lang.SecurityException: Can't read cryptographic
>>>> policy
>>>> directory: unlimited
>>> Does this help?
>>> https://stackoverflow.com/a/58183460/
>>>
>>> Yours,
>>> Reg
>> The stackoverflow link provides a lot of useful info about the error.
>> But unless something in the boot process deletes those files and
>> recreates them later, they are all there and correct. One boot works
>> fine. The next boot throws this error. The next boot is fine. Is
>> there a possibility that permissions get altered on this 'unlimited'
>> folder during boot? I can't think of many reasons it would say a
>> folder doesn't exist that clearly does exist.
>
> How are you launching Tomcat on boot?
>
> Are you able to add some instrumentation/logging to that?
>
> -chris
The Tomcat boot log in catalina.out runs fine. The exception first
occurs when a request comes in and I have to go to the database. It
happens on getting a connection from the datasource.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Jerry,
On 6/13/23 11:51, Jerry Malcolm wrote:
> Reg,
>
> On 6/13/2023 2:57 AM, r.barclay@habmalnefrage.de wrote:
>> Hey Jerry,
>>
>>> Caused by: java.lang.SecurityException: Can not initialize cryptographic
>>> mechanism
>>> at
>>> java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120) ...
>>> 86 mo
>>> Caused by: java.lang.SecurityException: Can't read cryptographic policy
>>> directory: unlimited
>> Does this help?
>> https://stackoverflow.com/a/58183460/
>>
>> Yours,
>> Reg
> The stackoverflow link provides a lot of useful info about the error.
> But unless something in the boot process deletes those files and
> recreates them later, they are all there and correct. One boot works
> fine. The next boot throws this error. The next boot is fine. Is
> there a possibility that permissions get altered on this 'unlimited'
> folder during boot? I can't think of many reasons it would say a folder
> doesn't exist that clearly does exist.
How are you launching Tomcat on boot?
Are you able to add some instrumentation/logging to that?
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Posted by Jerry Malcolm <te...@malcolms.com>.
Reg,
On 6/13/2023 2:57 AM, r.barclay@habmalnefrage.de wrote:
> Hey Jerry,
>
>> Caused by: java.lang.SecurityException: Can not initialize cryptographic
>> mechanism
>> at
>> java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120) ... 86 mo
>> Caused by: java.lang.SecurityException: Can't read cryptographic policy
>> directory: unlimited
> Does this help?
> https://stackoverflow.com/a/58183460/
>
> Yours,
> Reg
The stackoverflow link provides a lot of useful info about the error.
But unless something in the boot process deletes those files and
recreates them later, they are all there and correct. One boot works
fine. The next boot throws this error. The next boot is fine. Is
there a possibility that permissions get altered on this 'unlimited'
folder during boot? I can't think of many reasons it would say a folder
doesn't exist that clearly does exist.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Posted by r....@habmalnefrage.de.
Hey Jerry,
> Caused by: java.lang.SecurityException: Can not initialize cryptographic
> mechanism
> at
> java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120) ... 86 mo
> Caused by: java.lang.SecurityException: Can't read cryptographic policy
> directory: unlimited
Does this help?
https://stackoverflow.com/a/58183460/
Yours,
Reg
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Posted by Simon Matter <si...@invoca.ch>.
Hi,
> Jerry,
>
> On 6/13/23 11:42, Jerry Malcolm wrote:
>> Simon,
>>
>> On 6/13/2023 2:20 AM, Simon Matter wrote:
>>> Hi,
>>>
>>>> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
>>>> Linux2 in a production environment. A couple of years ago, we started
>>>> getting weird errors that the "Crypto Mechanism" failed to initialize.
>>>> Through a lot of trial and error, and reasons I don't quite remember,
>>>> we
>>>> put a 2-min delay in rc.local before starting Tomcat, and the problem
>>>> went away. I'm not a Linux nor a crypto guru. But we traced it to
>>>> some
>>>> crypto file that we assumed was not available until later in the Linux
>>>> boot sequence. Anyway, the 2 minute delay made it go away, for over
>>>> two
>>>> years. Then all of a sudden in the last day or so, it's back with a
>>>> vengeance. It fails with the same crypto error from 2 years ago in
>>>> about 50% of the EC2 boot ups. I tried bumping the wait to 3 min, and
>>>> no change.
>>>>
>>>> I need help. Our whole production environment is unstable now since
>>>> every time an ASG brings a new instance online, I've got a 50-50
>>>> chance
>>>> that tomcat is going to die (and the health check doesn't catch it,
>>>> but
>>>> that's a different issue).
>>>>
>>>> There are no errors in the Tomcat boot sequence logs. But the first
>>>> time and every subsequent time I try to get a connection from the
>>>> DataSource pool, I get the stack dump shown below.
>>>>
>>>> I figure it has to be a timing/race condition. But I have no clue
>>>> what
>>>> to do to fix it. I'm baffled that it worked for two years, and now
>>>> fails every other time I start an instance. And every instance is
>>>> running copies of the exact same Amazon Machine Image. The same EC2
>>>> will come up clean 50% of the time the next time it boots.
>>> Could it be that your hosts are running out of entropy on boot?
>>>
>>> Maybe there were RNG related changes in the kernel or rng-tools?
>>>
>>> Maybe monitor available entropy in
>>> /proc/sys/kernel/random/entropy_avail,
>>> it should not go below 100 or so.
>>>
>>> Regards,
>>> Simon
>>>
>> I haven't done any Linux or other system updates in several weeks. I'll
>> look into the entropy possibility. Would running out of entropy cause
>> an exception stating that a crypto directory doesn't exist? I don't
>> know much about Java entropy. Any ideas what would cause entropy to be
>> good on one boot and bad on the next boot? Thx.
>
> This isn't about entropy. Focus on the actual error message that the
> system can't find the unlimited policy file for some reason.
>
> -chris
Sorry, I didn't read the errors too carefully :)
I don't know Amazon Linux2 but one question, does it have the directory
/etc/crypto-policies? Apart from this, does systemd somehow fiddling with
crypto stuff while booting? Since systemd parallelizes the whole boot
process many things can (and sometimes do) break and it's difficult to
find it out.
Regards,
Simon
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Jerry,
On 6/13/23 11:42, Jerry Malcolm wrote:
> Simon,
>
> On 6/13/2023 2:20 AM, Simon Matter wrote:
>> Hi,
>>
>>> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
>>> Linux2 in a production environment. A couple of years ago, we started
>>> getting weird errors that the "Crypto Mechanism" failed to initialize.
>>> Through a lot of trial and error, and reasons I don't quite remember, we
>>> put a 2-min delay in rc.local before starting Tomcat, and the problem
>>> went away. I'm not a Linux nor a crypto guru. But we traced it to some
>>> crypto file that we assumed was not available until later in the Linux
>>> boot sequence. Anyway, the 2 minute delay made it go away, for over two
>>> years. Then all of a sudden in the last day or so, it's back with a
>>> vengeance. It fails with the same crypto error from 2 years ago in
>>> about 50% of the EC2 boot ups. I tried bumping the wait to 3 min, and
>>> no change.
>>>
>>> I need help. Our whole production environment is unstable now since
>>> every time an ASG brings a new instance online, I've got a 50-50 chance
>>> that tomcat is going to die (and the health check doesn't catch it, but
>>> that's a different issue).
>>>
>>> There are no errors in the Tomcat boot sequence logs. But the first
>>> time and every subsequent time I try to get a connection from the
>>> DataSource pool, I get the stack dump shown below.
>>>
>>> I figure it has to be a timing/race condition. But I have no clue what
>>> to do to fix it. I'm baffled that it worked for two years, and now
>>> fails every other time I start an instance. And every instance is
>>> running copies of the exact same Amazon Machine Image. The same EC2
>>> will come up clean 50% of the time the next time it boots.
>> Could it be that your hosts are running out of entropy on boot?
>>
>> Maybe there were RNG related changes in the kernel or rng-tools?
>>
>> Maybe monitor available entropy in /proc/sys/kernel/random/entropy_avail,
>> it should not go below 100 or so.
>>
>> Regards,
>> Simon
>>
> I haven't done any Linux or other system updates in several weeks. I'll
> look into the entropy possibility. Would running out of entropy cause
> an exception stating that a crypto directory doesn't exist? I don't
> know much about Java entropy. Any ideas what would cause entropy to be
> good on one boot and bad on the next boot? Thx.
This isn't about entropy. Focus on the actual error message that the
system can't find the unlimited policy file for some reason.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Posted by Jerry Malcolm <te...@malcolms.com>.
Simon,
On 6/13/2023 2:20 AM, Simon Matter wrote:
> Hi,
>
>> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
>> Linux2 in a production environment. A couple of years ago, we started
>> getting weird errors that the "Crypto Mechanism" failed to initialize.
>> Through a lot of trial and error, and reasons I don't quite remember, we
>> put a 2-min delay in rc.local before starting Tomcat, and the problem
>> went away. I'm not a Linux nor a crypto guru. But we traced it to some
>> crypto file that we assumed was not available until later in the Linux
>> boot sequence. Anyway, the 2 minute delay made it go away, for over two
>> years. Then all of a sudden in the last day or so, it's back with a
>> vengeance. It fails with the same crypto error from 2 years ago in
>> about 50% of the EC2 boot ups. I tried bumping the wait to 3 min, and
>> no change.
>>
>> I need help. Our whole production environment is unstable now since
>> every time an ASG brings a new instance online, I've got a 50-50 chance
>> that tomcat is going to die (and the health check doesn't catch it, but
>> that's a different issue).
>>
>> There are no errors in the Tomcat boot sequence logs. But the first
>> time and every subsequent time I try to get a connection from the
>> DataSource pool, I get the stack dump shown below.
>>
>> I figure it has to be a timing/race condition. But I have no clue what
>> to do to fix it. I'm baffled that it worked for two years, and now
>> fails every other time I start an instance. And every instance is
>> running copies of the exact same Amazon Machine Image. The same EC2
>> will come up clean 50% of the time the next time it boots.
> Could it be that your hosts are running out of entropy on boot?
>
> Maybe there were RNG related changes in the kernel or rng-tools?
>
> Maybe monitor available entropy in /proc/sys/kernel/random/entropy_avail,
> it should not go below 100 or so.
>
> Regards,
> Simon
>
I haven't done any Linux or other system updates in several weeks. I'll
look into the entropy possibility. Would running out of entropy cause
an exception stating that a crypto directory doesn't exist? I don't
know much about Java entropy. Any ideas what would cause entropy to be
good on one boot and bad on the next boot? Thx.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: Crypto Randomly Not Getting Initialized
Posted by Simon Matter <si...@invoca.ch>.
Hi,
> I am running Tomcat 9.0.56 in multiple AWS EC2 instances with Amazon
> Linux2 in a production environment. A couple of years ago, we started
> getting weird errors that the "Crypto Mechanism" failed to initialize.
> Through a lot of trial and error, and reasons I don't quite remember, we
> put a 2-min delay in rc.local before starting Tomcat, and the problem
> went away. I'm not a Linux nor a crypto guru. But we traced it to some
> crypto file that we assumed was not available until later in the Linux
> boot sequence. Anyway, the 2 minute delay made it go away, for over two
> years. Then all of a sudden in the last day or so, it's back with a
> vengeance. It fails with the same crypto error from 2 years ago in
> about 50% of the EC2 boot ups. I tried bumping the wait to 3 min, and
> no change.
>
> I need help. Our whole production environment is unstable now since
> every time an ASG brings a new instance online, I've got a 50-50 chance
> that tomcat is going to die (and the health check doesn't catch it, but
> that's a different issue).
>
> There are no errors in the Tomcat boot sequence logs. But the first
> time and every subsequent time I try to get a connection from the
> DataSource pool, I get the stack dump shown below.
>
> I figure it has to be a timing/race condition. But I have no clue what
> to do to fix it. I'm baffled that it worked for two years, and now
> fails every other time I start an instance. And every instance is
> running copies of the exact same Amazon Machine Image. The same EC2
> will come up clean 50% of the time the next time it boots.
Could it be that your hosts are running out of entropy on boot?
Maybe there were RNG related changes in the kernel or rng-tools?
Maybe monitor available entropy in /proc/sys/kernel/random/entropy_avail,
it should not go below 100 or so.
Regards,
Simon
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org