You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Mark Eggers <it...@yahoo.com.INVALID> on 2021/12/22 00:50:29 UTC
[OT] Re: Critical Random "Can't read cryptographic policy directory: unlimited"
Jerry:
On 12/21/2021 4:17 PM, Jerry Malcolm wrote:
> Mark,
>
> Thanks for the comments. I have somewhat of a solution. I just want to
> document what I learned in case anyone else has this problem. From what
> I have been able to determine, you are correct about it being an Amazon
> problem. I don't really understand what is happening. But here's what
> I found... After a day of testing all types of scenarios, I found that
> it only happens the first time after installing a new ami (image) on a
> brand new EC2 and bringing up the EC2 and auto-starting TC for the
> first time. But it fails EVERY time with this scenario. Bouncing TC
> fixed it every time. Bouncing the EC2 never causes it to come back. Only
> booting a brand new EC2 instance causes it. I tried removing the tomcat
> autostart, ran the ami EC2 creation, waited a bit, and went to the
> console and manually started tomcat. No problem.
>
> So all I can deduce from this is that there's a timing issue. I'm
> suspecting that AWS is starting up the EC2 before it is fully created
> (likely something not complete in the file system setup). I added a
> 2-minute sleep in rc.local to delay the startup of tomcat long enough
> for whatever wasn't finished yet to complete. With a 2-min sleep, it now
> works every time. Pretty ugly fix IMHO. But working ugly beats not
> working pretty. I'll probably play around with the 2-min sleep to see
> if I can reduce that. I may try to talk to Amazon about it. I may also
> try banging my head against the wall, probably with the same results as
> talking to amazon about this.
>
> The takeaway from this is that AWS appears to be trying to shave a
> little performance delay by jumping the gun on the EC2 boot. The result
> in this case is TC saying the crypto file doesn't exist, but it does
> exist a minute or two later. Likely nothing to do specifically with the
> crypto file. That's just likely the first file needed that wasn't ready
> yet.
>
> Moving on the next fire that needs to be put out...
>
> Thx
>
> Jerry
>
>
> On 12/20/2021 5:22 AM, Mark Thomas wrote:
>> On 20/12/2021 06:59, Jerry Malcolm wrote:
>>> I'm adding a slight variation to the error I get at times (see bottom
>>> of stack trace below)
>>
>> This is the code that throws the root exception:
>>
>> if (!Files.isDirectory(cryptoPolicyPath)
>> || !Files.isReadable(cryptoPolicyPath)) {
>> throw new SecurityException(
>> "Can't read cryptographic policy directory: " +
>> cryptoPolicyProperty);
>> }
>>
>>
>> That points very strongly to a file system issue. My recommendation is
>> to take this up with AWS support.
>>
>> Mark
>>
>>
>>>
>>> On 12/19/2021 11:55 PM, Jerry Malcolm wrote:
>>>> I have a production environment of 10+ AWS EC2 servers all built
>>>> from a single EC2 snapshot image. The master EC2 works fine. But
>>>> when we propagated the image to all of the production servers, we
>>>> started getting "Can't read cryptographic policy directory:
>>>> unlimited" errors when I try to get a jdbc connection object from
>>>> the pool. Full stack trace is below. The three critical lines in
>>>> the trace are:
>>>>
>>>> java.lang.ExceptionInInitializerError
>>>> Caused by: java.lang.SecurityException: Can not initialize
>>>> cryptographic mechanism
>>>> Caused by: java.lang.SecurityException: Can't read cryptographic
>>>> policy directory: unlimited
>>>>
>>>> On some machines rebooting the tomcat service fixes the problem and
>>>> it continues to work fine after that. On other machines, the
>>>> problem is still there after rebooting. But even on the machines
>>>> that are fixed after TC bounce, if I bounce the full EC2, the
>>>> problem is back.
>>>>
>>>> We've had this environment working for almost two years. I did a
>>>> minor version upgrade to TC 8.5.73 a month ago. And I upgraded to
>>>> java 11 probably a year ago. Otherwise, no changes that I'm aware
>>>> of. Definitely nothing in the past few days before the error hit
>>>> yesterday.
>>>>
>>>> Everything I can find on google about the error messages says to
>>>> make sure the 'unlimited' folder is present and accessible. The
>>>> folder is there and has been there untouched. And I know it's not
>>>> disappearing and reappearing to become accessible after the TC
>>>> reboot that sometimes fixes the problem.
>>>>
>>>> The mySQL RDS all of these instances talk to hasn't changed. And
>>>> other servers that aren't part of this image distribution have no
>>>> problems accessing it. It just makes no sense that the same EC2
>>>> image that works on one machine started failing on a bunch of
>>>> identical configuration EC2s yesterday.
>>>>
>>>> I know this call goes through layers of tomcat, layers of the mysql
>>>> driver, and then layers of the jvm before it occurs. But can anyone
>>>> help me understand what TC/mySQL might be trying to do with this
>>>> call to JVM crypto code and why it has started failing on all of my
>>>> servers? Any Java crypto gurus out there?
>>>>
>>>> Stack trace of error:
>>>>
>>>> java.lang.ExceptionInInitializerError
>>>> at java.base/javax.crypto.Cipher.getInstance(Cipher.java:540)
>>>> at java.base/sun.security.ssl.JsseJce.getCipher(JsseJce.java:185)
>>>> at
>>>> java.base/sun.security.ssl.SSLCipher.isTransformationAvailable(SSLCipher.java:483)
>>>>
>>>> at java.base/sun.security.ssl.SSLCipher.<init>(SSLCipher.java:472)
>>>> at java.base/sun.security.ssl.SSLCipher.<clinit>(SSLCipher.java:81)
>>>> at java.base/sun.security.ssl.CipherSuite.<clinit>(CipherSuite.java:67)
>>>> at
>>>> java.base/sun.security.ssl.SSLContextImpl.getApplicableSupportedCipherSuites(SSLContextImpl.java:348)
>>>>
>>>> at
>>>> java.base/sun.security.ssl.SSLContextImpl$AbstractTLSContext.<clinit>(SSLContextImpl.java:580)
>>>>
>>>> at java.base/java.lang.Class.forName0(Native Method)
>>>> at java.base/java.lang.Class.forName(Class.java:315)
>>>> at
>>>> java.base/java.security.Provider$Service.getImplClass(Provider.java:1918)
>>>>
>>>> at
>>>> java.base/java.security.Provider$Service.newInstance(Provider.java:1894)
>>>>
>>>> at
>>>> java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
>>>>
>>>> at
>>>> java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
>>>>
>>>> at java.base/javax.net.ssl.SSLContext.getInstance(SSLContext.java:168)
>>>> at
>>>> com.mysql.cj.protocol.ExportControlled.getSSLContext(ExportControlled.java:565)
>>>>
>>>> at
>>>> com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:302)
>>>>
>>>> at
>>>> com.mysql.cj.protocol.StandardSocketFactory.performTlsHandshake(StandardSocketFactory.java:188)
>>>>
>>>> at
>>>> com.mysql.cj.protocol.a.NativeSocketConnection.performTlsHandshake(NativeSocketConnection.java:99)
>>>>
>>>> at
>>>> com.mysql.cj.protocol.a.NativeProtocol.negotiateSSLConnection(NativeProtocol.java:331)
>>>>
>>>> at
>>>> com.mysql.cj.protocol.a.NativeAuthenticationProvider.negotiateSSLConnection(NativeAuthenticationProvider.java:777)
>>>>
>>>> at
>>>> com.mysql.cj.protocol.a.NativeAuthenticationProvider.proceedHandshakeWithPluggableAuthentication(NativeAuthenticationProvider.
>>>>
>>>> at
>>>> com.mysql.cj.protocol.a.NativeAuthenticationProvider.connect(NativeAuthenticationProvider.java:202)
>>>>
>>>> at
>>>> com.mysql.cj.protocol.a.NativeProtocol.connect(NativeProtocol.java:1348)
>>>>
>>>> at com.mysql.cj.NativeSession.connect(NativeSession.java:163)
>>>> at
>>>> com.mysql.cj.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:947)
>>>>
>>>> at
>>>> com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:817)
>>>> at com.mysql.cj.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:447)
>>>> at
>>>> com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:237)
>>>> at
>>>> com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:199)
>>>>
>>>> at
>>>> org.apache.tomcat.dbcp.dbcp2.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:52)
>>>>
>>>> at
>>>> org.apache.tomcat.dbcp.dbcp2.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:365)
>>>>
>>>> at
>>>> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.validateConnectionFactory(BasicDataSource.java:116)
>>>>
>>>> at
>>>> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createPoolableConnectionFactory(BasicDataSource.java:665)
>>>>
>>>> at
>>>> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.createDataSource(BasicDataSource.java:547)
>>>>
>>>> at
>>>> org.apache.tomcat.dbcp.dbcp2.BasicDataSource.getConnection(BasicDataSource.java:747)
>>>>
>>>> at jwm.db.DBData.getConnection(DBData.java:443)
>>>> ...
>>>> Caused by: java.lang.SecurityException: Can not initialize
>>>> cryptographic mechanism
>>>> at java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:120)
>>>> ... 81 more
>>>> Caused by: java.lang.SecurityException: Can't read cryptographic
>>>> policy directory: unlimited
>>>> at
>>>> java.base/javax.crypto.JceSecurity.setupJurisdictionPolicies(JceSecurity.java:326)
>>>>
>>>> at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:111)
>>>> at java.base/javax.crypto.JceSecurity$1.run(JceSecurity.java:108)
>>>> at java.base/java.security.AccessController.doPrivileged(Native Method)
>>>> at java.base/javax.crypto.JceSecurity.<clinit>(JceSecurity.java:107)
>>>> .. 81 more
>>>>
>>> ================= This is a variation of the error I've also found in
>>> the logs. NoClassDefFoundError that is random just adds to the
>>> strangeness of this problem...
>>>
>>> java.lang.NoClassDefFoundError: Could not initialize class
>>> sun.security.ssl.SSLContextImpl$TLSContext
>>> at java.base/java.lang.Class.forName0(Native Method)
>>> at java.base/java.lang.Class.forName(Class.java:315)
>>> at
>>> java.base/java.security.Provider$Service.getImplClass(Provider.java:1918)
>>>
>>> at
>>> java.base/java.security.Provider$Service.newInstance(Provider.java:1894)
>>> at
>>> java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:236)
>>> at
>>> java.base/sun.security.jca.GetInstance.getInstance(GetInstance.java:164)
>>> at java.base/javax.net.ssl.SSLContext.getInstance(SSLContext.java:168)
>>> at
>>> com.mysql.cj.protocol.ExportControlled.getSSLContext(ExportControlled.java:565)
>>>
>>> at
>>> com.mysql.cj.protocol.ExportControlled.performTlsHandshake(ExportControlled.java:302)
>>>
>>> at
>>> com.mysql.cj.protocol.StandardSocketFactory.performTlsHandshake(StandardSocketFactory.j
Did you post the systemd files provided by AWS? If you did, I missed it.
In my setup (local CentOS 7 and Ubuntu 20.04), I rely on a PID file
written to
/var/run/tomcat/[servicename].pid
Unfortunately, the /var/run directory is a tmpfs file system which is
created at boot time. What I had to do was to create a systemd process
that created the subdirectory, and have that as a requirement for Tomcat
to start.
Something like (tomcat-pre.service):
[Unit]
Description=Create Tomcat PID directory
[Service]
Type=oneshot
RuntimeDirectory=tomcat
User=tcadmin
Group=tcadmin
ExecStart=/bin/true
RemainAfterExit=true
[Install]
WantedBy=multi-user.target
And then, my Tomcat systemd file looks like (in part):
[Unit]
Description=Apache Tomcat loki
After=network.target tomcat-pre.service
You might check if the endorsed directory is created by a systemd
service (normally a oneshot), and then add that service to the After
line of the tomcat service systemd file.
Hopefully that helps.
. . . just my two cents
/mde/