You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@tomcat.apache.org by Amit Pande <Am...@veritas.com> on 2020/11/25 17:40:29 UTC
RE: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
Thank you Chris for the inputs. Admittedly, I didn’t know the internals of Sun JCE/JSSE vs BC JCE.
Pasting sample output and it indeed is taking minutes on RHEL 7.3. Not sure if I am indeed missing some trick here.
RHEL 7.3 ---------------
test-host-1:~ # date ; /usr/openv/java/jre/bin/keytool -providerpath /root/Downloads/bc-fips-1.0.2.jar -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair -keyalg RSA -keypass "Test123" -validity 3650 -dname "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123" -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date
Wed Nov 25 10:52:56 CST 2020 (START TIME)
Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 3,650 days
for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org
[Storing /tmp/test_bc.bcfks]
Wed Nov 25 10:58:11 CST 2020 (END TIME) (Almost 6 minutes)
test-host-1:~ # uname -a
Linux test-host-1 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
test-host-1:~ # cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.3 (Maipo)
test-host-1:~ #
test-host-1:~ # /usr/openv/java/jre/bin/java -version
java version "1.8.0_271"
Java(TM) SE Runtime Environment (build 1.8.0_271-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode)
test-host-1:~ #
RHEL 7.2 -------------
[root@test-host-2 ~]# date ; /usr/openv/java/jre/bin/keytool -providerpath /root/bc-fips-1.0.2.jar -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair -keyalg RSA -keypass "Test123" -validity 3650 -dname "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123" -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date
Wed Nov 25 11:20:06 CST 2020 (START TIME)
Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 3,650 days
for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org
[Storing /tmp/test_bc.bcfks]
Wed Nov 25 11:20:10 CST 2020 (END TIME) (Almost 4 seconds)
[root@test-host-2 ~]# uname -a
Linux test-host-2 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
[root@test-host-2 ~]# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@test-host-2 ~]#
Red Hat Enterprise Linux Server release 7.2 (Maipo)
[root@test-host-2 ~]# /usr/openv/java/jre/bin/java -version
java version "1.8.0_261"
Java(TM) SE Runtime Environment (build 1.8.0_261-b33)
Java HotSpot(TM) 64-Bit Server VM (build 25.261-b33, mixed mode)
[root@test-host-2 ~]#
Since the keytool is literally taking minutes (specifically on RHEL 7.3 as you can see above), enabling FIPS OOTB has become a challenge as it has resulted in some our test suites timing out. Tomcat is very much a core of our product and configuration and starting Tomcat in timely manner (FIPS or no FIPS) has been a critical requirement. And now, with this issue, test suites timing out, hard to convince to get the suite timeouts increased. 😊
Thanks,
Amit
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Wednesday, November 25, 2020 10:33 AM
To: users@tomcat.apache.org
Subject: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
Amit,
On 11/24/20 11:21, Amit Pande wrote:
> Probably not directly related to Tomcat but still sharing. Advanced
> apologies for that.
>
> I am using bouncy castle FIPS library and observed that specifically
> on RHEL 7.3, the library usage is causing tremendous slowness.
Note that BC is pure-Java, so you should pretty much expect "tremendous slowness". The reason that the Sun JCE/JSSE provider is (or can be) very fast is because it's using both native code (which is only marginally faster than well-JIT'ed Java bytecode, if at all) and also hardware support when it's available. There is no way that I know of for BC's pure-Java crypto implementations to use hardware acceleration. AES on silicon is *much* faster than AES in software.
> e.g. below key tool command taking several minutes to finish.
Uhh... that's not software versus hardware. You might expect an order of magnitude difference between BC and hardware-accelerated SunJCE. But minutes to generate a 2048-bit RSA key? Something is wrong.
> keytool -providerpath /root/Downloads/bc-fips-1.0.2.jar -providerclass
> org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair
> -keyalg RSA -keypass "Test123" -validity 3650 -dname
> "CN=Test_HostName, ou=My Org Unit, o=My Org" -storepass "Test123"
> -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias test_entry
>
> However, when I add the JVM option
> -J-Djava.security.egd=file:/dev/./urandom instead of the default
> /dev/random, all problems go away.
Classic entropy problem.
> It's rather strange that multiple 7.3 RHEL systems are showing this
> behavior very consistently. RHEL 7.2, 8.0 seem just fine in my
> testing. No issues when using /dev/random along with JKS type key
> stores (including RHEL 7.3).
>
> Even Tomcat is flagging the slowness to generate the secure random
> numbers.
>
> "17-Nov-2020 19:24:59.142 WARNING [Catalina-utility-2] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [68,745] milliseconds."
>
> Apparently, according to FIPS experts we should not use /dev/./urandom.
Correct. FIPS assumes that 100% of your cryptographic operations are of the *utmost highly-classified state-secret-style nuclear-launch-code
flashing-purple* security level.
Sometimes you just don't need to have perfect entropy.
For long-lived keys, really good entropy is a Good Thing. So to generate your RSA keys, I'd say go ahead and use the "high quality entropy source". But for randomness for session id generation, and for temporary, bulk encryption TLS session keys, /dev/urandom is perfectly fine.
> So, while I continue to explore further, sharing this in case any one
> has seen this behavior or be aware of such potential issue as it may
> tie in to their Tomcat (or web apps to be deployed in Tomcat).
You might be able to justify using /dev/urandom (which must be spelled /dev/./urandom to force Java to actually use it) in your web application, but I would go ahead and let keytool take as long as it needs.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
Posted by Amit Pande <Am...@veritas.com>.
Thank you Stefan, Chris for the inputs.
As I understood from our security experts, there is no moving away from /dev/random (anything else isn't (strongly) FIPS compliant) for us.
Thanks,
Amit
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Wednesday, December 2, 2020 10:29 AM
To: users@tomcat.apache.org
Subject: Re: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
Stefan,
On 11/30/20 19:17, Stefan Mayr wrote:
> Hi,
>
> Am 30.11.2020 um 17:09 schrieb Amit Pande:
>> I guess I will have to investigate the RHEL 7.3 entropy issue separately (possibly as hobby project) and look for other options to make progress.
>>
>> I still find it odd that something related to randomness (entropy generation) is so consistent (the slowness is equally slow or more on multiple RHEL 7.3 systems I have, maybe I need to look for machines from different data center or a physical 7.3 server).
>>
>> And yes, the 10 year certificate validity is just for testing
>> purposes. 😊
>>
>> Thank you for your inputs. Indeed helpful in evaluating our choices.
>>
>> Thanks,
>> Amit
>
> you might have a look at rng-tools (rngd) or haveged to boost your
> entropy pool.
>
> We use haveged in a VMware virtualized environment and this reduces a
> plain tomcat startup from multiple minutes to just a few secondes.
>
> I think Red Hat preferes rngd but there should be some articles on
> access.redhat.com to help you depending on the used hypervisor.
I would think long and hard about whether or not you want to use any of these tools. There are already ways to get "a lot of entropy really quickly" from the Linux kernel; specifically, /dev/urandom.
The whole point of both /dev/random and /dev/urandom existing side by side is so that the application can pick whether it wants "high quality entropy" (by using /dev/random) or "good enough randomness" (by using /dev/urandom).
Tools like haveged and rngd basically make /dev/random behave like /dev/urandom so the application can never have "high quality entropy"
even when it asks for it.
Have a look at this discussion on security.stackexchange to get you started down the path to paranoia:
https://security.stackexchange.com/questions/34523
My question has always been "if these things are both safe and a good idea, why does the Linux kernel not implement them directly?" There must be a reason why the kernel devs have decided not to "speed up"
/dev/random using the techniques used by both haveged and rngd. Maybe their argument is essentially "you can always just use haveged/rngd" but my guess is there is a more fundamental reason for not adopting these techniques directly in the kernel.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Stefan,
On 11/30/20 19:17, Stefan Mayr wrote:
> Hi,
>
> Am 30.11.2020 um 17:09 schrieb Amit Pande:
>> I guess I will have to investigate the RHEL 7.3 entropy issue separately (possibly as hobby project) and look for other options to make progress.
>>
>> I still find it odd that something related to randomness (entropy generation) is so consistent (the slowness is equally slow or more on multiple RHEL 7.3 systems I have, maybe I need to look for machines from different data center or a physical 7.3 server).
>>
>> And yes, the 10 year certificate validity is just for testing purposes. 😊
>>
>> Thank you for your inputs. Indeed helpful in evaluating our choices.
>>
>> Thanks,
>> Amit
>
> you might have a look at rng-tools (rngd) or haveged to boost your
> entropy pool.
>
> We use haveged in a VMware virtualized environment and this reduces a
> plain tomcat startup from multiple minutes to just a few secondes.
>
> I think Red Hat preferes rngd but there should be some articles on
> access.redhat.com to help you depending on the used hypervisor.
I would think long and hard about whether or not you want to use any of
these tools. There are already ways to get "a lot of entropy really
quickly" from the Linux kernel; specifically, /dev/urandom.
The whole point of both /dev/random and /dev/urandom existing side by
side is so that the application can pick whether it wants "high quality
entropy" (by using /dev/random) or "good enough randomness" (by using
/dev/urandom).
Tools like haveged and rngd basically make /dev/random behave like
/dev/urandom so the application can never have "high quality entropy"
even when it asks for it.
Have a look at this discussion on security.stackexchange to get you
started down the path to paranoia:
https://security.stackexchange.com/questions/34523
My question has always been "if these things are both safe and a good
idea, why does the Linux kernel not implement them directly?" There must
be a reason why the kernel devs have decided not to "speed up"
/dev/random using the techniques used by both haveged and rngd. Maybe
their argument is essentially "you can always just use haveged/rngd" but
my guess is there is a more fundamental reason for not adopting these
techniques directly in the kernel.
-chris
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
Posted by Stefan Mayr <st...@mayr-stefan.de>.
Hi,
Am 30.11.2020 um 17:09 schrieb Amit Pande:
> I guess I will have to investigate the RHEL 7.3 entropy issue separately (possibly as hobby project) and look for other options to make progress.
>
> I still find it odd that something related to randomness (entropy generation) is so consistent (the slowness is equally slow or more on multiple RHEL 7.3 systems I have, maybe I need to look for machines from different data center or a physical 7.3 server).
>
> And yes, the 10 year certificate validity is just for testing purposes. 😊
>
> Thank you for your inputs. Indeed helpful in evaluating our choices.
>
> Thanks,
> Amit
you might have a look at rng-tools (rngd) or haveged to boost your
entropy pool.
We use haveged in a VMware virtualized environment and this reduces a
plain tomcat startup from multiple minutes to just a few secondes.
I think Red Hat preferes rngd but there should be some articles on
access.redhat.com to help you depending on the used hypervisor.
Regards,
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
RE: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
Posted by Amit Pande <Am...@veritas.com>.
I guess I will have to investigate the RHEL 7.3 entropy issue separately (possibly as hobby project) and look for other options to make progress.
I still find it odd that something related to randomness (entropy generation) is so consistent (the slowness is equally slow or more on multiple RHEL 7.3 systems I have, maybe I need to look for machines from different data center or a physical 7.3 server).
And yes, the 10 year certificate validity is just for testing purposes. 😊
Thank you for your inputs. Indeed helpful in evaluating our choices.
Thanks,
Amit
-----Original Message-----
From: Christopher Schultz <ch...@christopherschultz.net>
Sent: Wednesday, November 25, 2020 9:42 PM
To: users@tomcat.apache.org
Subject: Re: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
Amit,
On 11/25/20 12:40, Amit Pande wrote:
> Thank you Chris for the inputs. Admittedly, I didn’t know the internals of Sun JCE/JSSE vs BC JCE.
>
> Pasting sample output and it indeed is taking minutes on RHEL 7.3. Not sure if I am indeed missing some trick here.
>
>
> RHEL 7.3 ---------------
>
> test-host-1:~ # date ; /usr/openv/java/jre/bin/keytool -providerpath
> /root/Downloads/bc-fips-1.0.2.jar -providerclass
> org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair
> -keyalg RSA -keypass "Test123" -validity 3650 -dname
> "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123"
> -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date
>
> Wed Nov 25 10:52:56 CST 2020 (START TIME)
>
> Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 3,650 days
> for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org [Storing
> /tmp/test_bc.bcfks]
>
> Wed Nov 25 10:58:11 CST 2020 (END TIME) (Almost 6 minutes)
>
> test-host-1:~ # uname -a
> Linux test-host-1 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT
> 2016 x86_64 x86_64 x86_64 GNU/Linux test-host-1:~ # cat
> /etc/redhat-release Red Hat Enterprise Linux Server release 7.3
> (Maipo) test-host-1:~ # test-host-1:~ # /usr/openv/java/jre/bin/java
> -version java version "1.8.0_271"
> Java(TM) SE Runtime Environment (build 1.8.0_271-b09) Java HotSpot(TM)
> 64-Bit Server VM (build 25.271-b09, mixed mode) test-host-1:~ #
>
>
> RHEL 7.2 -------------
>
> [root@test-host-2 ~]# date ; /usr/openv/java/jre/bin/keytool
> -providerpath /root/bc-fips-1.0.2.jar -providerclass
> org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair
> -keyalg RSA -keypass "Test123" -validity 3650 -dname
> "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123"
> -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date
>
> Wed Nov 25 11:20:06 CST 2020 (START TIME)
>
> Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 3,650 days
> for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org [Storing
> /tmp/test_bc.bcfks]
>
> Wed Nov 25 11:20:10 CST 2020 (END TIME) (Almost 4 seconds)
>
> [root@test-host-2 ~]# uname -a
> Linux test-host-2 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29
> EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
> [root@test-host-2 ~]# cat /etc/redhat-release Red Hat Enterprise Linux
> Server release 7.2 (Maipo)
> [root@test-host-2 ~]#
>
> Red Hat Enterprise Linux Server release 7.2 (Maipo)
> [root@test-host-2 ~]# /usr/openv/java/jre/bin/java -version java
> version "1.8.0_261"
> Java(TM) SE Runtime Environment (build 1.8.0_261-b33) Java HotSpot(TM)
> 64-Bit Server VM (build 25.261-b33, mixed mode)
> [root@test-host-2 ~]#
>
>
> Since the keytool is literally taking minutes (specifically on RHEL
> 7.3 as you can see above), enabling FIPS OOTB has become a challenge
> as it has resulted in some our test suites timing out.
What I see is that on two servers were everything is the same (except for the RHEL version... weird; RHEL 7.2 and 7.3 have identical kernel versions?), two identical command-lines take different amounts of time.
In both cases, you are using BC with FIPS enabled.
The reason is that one of the servers is starved for entropy and the other one is not. Some operations require a LOT of entropy (like generating an RSA key) and others require less entropy, but each of those operations consumes some of the system's entropy as they occur.
When you run out, processes block waiting for it, and it's only created slowly as random hardware events are sampled to generate that "high quality randomness". So it can take a while.
Some servers have hardware that generates entropy faster, some servers have usage profiles that cause certain hardware events to occur more often and therefore keep the entropy pool full. Others are constantly starved for entropy.
> Tomcat is very much a core of our product and configuration and
> starting Tomcat in timely manner (FIPS or no FIPS) has been a critical
> requirement. And now, with this issue, test suites timing out, hard to
> convince to get the suite timeouts increased. 😊
Well, you don't have too many choices. You either fix your entropy problem (which isn't exactly straightforward), or you switch to a process which does not require so much entropy (which may mean disabling FIPS), or you allow for more time to complete those operations.
Is the generation of RSA keys a part of your test suite? Maybe you can re-write your test suite to only generate the certificate if it's missing and/or is getting close to expiration. If you generate a new RSA key for each run of your test quite, you are certainly going to drain your entropy pool quickly and repeatedly.
BTW, I'm assuming that you are using (or at least just /testing/) FIPS mode because you need a "high security" environment. Nothing about a 2048-bit RSA key that is valid for 10 years should be considered "high security".
-chris
> -----Original Message-----
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Wednesday, November 25, 2020 10:33 AM
> To: users@tomcat.apache.org
> Subject: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
>
> Amit,
>
> On 11/24/20 11:21, Amit Pande wrote:
>> Probably not directly related to Tomcat but still sharing. Advanced
>> apologies for that.
>>
>> I am using bouncy castle FIPS library and observed that specifically
>> on RHEL 7.3, the library usage is causing tremendous slowness.
> Note that BC is pure-Java, so you should pretty much expect "tremendous slowness". The reason that the Sun JCE/JSSE provider is (or can be) very fast is because it's using both native code (which is only marginally faster than well-JIT'ed Java bytecode, if at all) and also hardware support when it's available. There is no way that I know of for BC's pure-Java crypto implementations to use hardware acceleration. AES on silicon is *much* faster than AES in software.
>
>> e.g. below key tool command taking several minutes to finish.
>
> Uhh... that's not software versus hardware. You might expect an order of magnitude difference between BC and hardware-accelerated SunJCE. But minutes to generate a 2048-bit RSA key? Something is wrong.
>
>> keytool -providerpath /root/Downloads/bc-fips-1.0.2.jar
>> -providerclass
>> org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair
>> -keyalg RSA -keypass "Test123" -validity 3650 -dname "CN=Test_HostName, ou=My Org Unit, o=My Org" -storepass "Test123"
>> -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias test_entry
>>
>> However, when I add the JVM option
>> -J-Djava.security.egd=file:/dev/./urandom instead of the default
>> /dev/random, all problems go away.
> Classic entropy problem.
>
>> It's rather strange that multiple 7.3 RHEL systems are showing this
>> behavior very consistently. RHEL 7.2, 8.0 seem just fine in my
>> testing. No issues when using /dev/random along with JKS type key
>> stores (including RHEL 7.3).
>>
>> Even Tomcat is flagging the slowness to generate the secure random
>> numbers.
>>
>> "17-Nov-2020 19:24:59.142 WARNING [Catalina-utility-2] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [68,745] milliseconds."
>>
>> Apparently, according to FIPS experts we should not use /dev/./urandom.
>
> Correct. FIPS assumes that 100% of your cryptographic operations are
> of the *utmost highly-classified state-secret-style
> nuclear-launch-code
> flashing-purple* security level.
>
> Sometimes you just don't need to have perfect entropy.
>
> For long-lived keys, really good entropy is a Good Thing. So to generate your RSA keys, I'd say go ahead and use the "high quality entropy source". But for randomness for session id generation, and for temporary, bulk encryption TLS session keys, /dev/urandom is perfectly fine.
>
>> So, while I continue to explore further, sharing this in case any one
>> has seen this behavior or be aware of such potential issue as it may
>> tie in to their Tomcat (or web apps to be deployed in Tomcat).
> You might be able to justify using /dev/urandom (which must be spelled /dev/./urandom to force Java to actually use it) in your web application, but I would go ahead and let keytool take as long as it needs.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org
Re: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
Posted by Christopher Schultz <ch...@christopherschultz.net>.
Amit,
On 11/25/20 12:40, Amit Pande wrote:
> Thank you Chris for the inputs. Admittedly, I didn’t know the internals of Sun JCE/JSSE vs BC JCE.
>
> Pasting sample output and it indeed is taking minutes on RHEL 7.3. Not sure if I am indeed missing some trick here.
>
>
> RHEL 7.3 ---------------
>
> test-host-1:~ # date ; /usr/openv/java/jre/bin/keytool -providerpath /root/Downloads/bc-fips-1.0.2.jar -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair -keyalg RSA -keypass "Test123" -validity 3650 -dname "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123" -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date
>
> Wed Nov 25 10:52:56 CST 2020 (START TIME)
>
> Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 3,650 days
> for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org
> [Storing /tmp/test_bc.bcfks]
>
> Wed Nov 25 10:58:11 CST 2020 (END TIME) (Almost 6 minutes)
>
> test-host-1:~ # uname -a
> Linux test-host-1 3.10.0-514.el7.x86_64 #1 SMP Wed Oct 19 11:24:13 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux
> test-host-1:~ # cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.3 (Maipo)
> test-host-1:~ #
> test-host-1:~ # /usr/openv/java/jre/bin/java -version
> java version "1.8.0_271"
> Java(TM) SE Runtime Environment (build 1.8.0_271-b09)
> Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode)
> test-host-1:~ #
>
>
> RHEL 7.2 -------------
>
> [root@test-host-2 ~]# date ; /usr/openv/java/jre/bin/keytool -providerpath /root/bc-fips-1.0.2.jar -providerclass org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair -keyalg RSA -keypass "Test123" -validity 3650 -dname "CN=Amit_HostName, ou=My_Org_Unit, o=My_Org" -storepass "Test123" -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias abcd ; date
>
> Wed Nov 25 11:20:06 CST 2020 (START TIME)
>
> Generating 2,048 bit RSA key pair and self-signed certificate (SHA256withRSA) with a validity of 3,650 days
> for: CN=Amit_HostName, OU=My_Org_Unit, O=My_Org
> [Storing /tmp/test_bc.bcfks]
>
> Wed Nov 25 11:20:10 CST 2020 (END TIME) (Almost 4 seconds)
>
> [root@test-host-2 ~]# uname -a
> Linux test-host-2 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
> [root@test-host-2 ~]# cat /etc/redhat-release
> Red Hat Enterprise Linux Server release 7.2 (Maipo)
> [root@test-host-2 ~]#
>
> Red Hat Enterprise Linux Server release 7.2 (Maipo)
> [root@test-host-2 ~]# /usr/openv/java/jre/bin/java -version
> java version "1.8.0_261"
> Java(TM) SE Runtime Environment (build 1.8.0_261-b33)
> Java HotSpot(TM) 64-Bit Server VM (build 25.261-b33, mixed mode)
> [root@test-host-2 ~]#
>
>
> Since the keytool is literally taking minutes (specifically on RHEL
> 7.3 as you can see above), enabling FIPS OOTB has become a challenge
> as it has resulted in some our test suites timing out.
What I see is that on two servers were everything is the same (except
for the RHEL version... weird; RHEL 7.2 and 7.3 have identical kernel
versions?), two identical command-lines take different amounts of time.
In both cases, you are using BC with FIPS enabled.
The reason is that one of the servers is starved for entropy and the
other one is not. Some operations require a LOT of entropy (like
generating an RSA key) and others require less entropy, but each of
those operations consumes some of the system's entropy as they occur.
When you run out, processes block waiting for it, and it's only created
slowly as random hardware events are sampled to generate that "high
quality randomness". So it can take a while.
Some servers have hardware that generates entropy faster, some servers
have usage profiles that cause certain hardware events to occur more
often and therefore keep the entropy pool full. Others are constantly
starved for entropy.
> Tomcat is very much a core of our product and configuration and
> starting Tomcat in timely manner (FIPS or no FIPS) has been a
> critical requirement. And now, with this issue, test suites timing
> out, hard to convince to get the suite timeouts increased. 😊
Well, you don't have too many choices. You either fix your entropy
problem (which isn't exactly straightforward), or you switch to a
process which does not require so much entropy (which may mean disabling
FIPS), or you allow for more time to complete those operations.
Is the generation of RSA keys a part of your test suite? Maybe you can
re-write your test suite to only generate the certificate if it's
missing and/or is getting close to expiration. If you generate a new RSA
key for each run of your test quite, you are certainly going to drain
your entropy pool quickly and repeatedly.
BTW, I'm assuming that you are using (or at least just /testing/) FIPS
mode because you need a "high security" environment. Nothing about a
2048-bit RSA key that is valid for 10 years should be considered "high
security".
-chris
> -----Original Message-----
> From: Christopher Schultz <ch...@christopherschultz.net>
> Sent: Wednesday, November 25, 2020 10:33 AM
> To: users@tomcat.apache.org
> Subject: [EXTERNAL] Re: Bouncy Castle FIPS on RHEL 7.3
>
> Amit,
>
> On 11/24/20 11:21, Amit Pande wrote:
>> Probably not directly related to Tomcat but still sharing. Advanced
>> apologies for that.
>>
>> I am using bouncy castle FIPS library and observed that specifically
>> on RHEL 7.3, the library usage is causing tremendous slowness.
> Note that BC is pure-Java, so you should pretty much expect "tremendous slowness". The reason that the Sun JCE/JSSE provider is (or can be) very fast is because it's using both native code (which is only marginally faster than well-JIT'ed Java bytecode, if at all) and also hardware support when it's available. There is no way that I know of for BC's pure-Java crypto implementations to use hardware acceleration. AES on silicon is *much* faster than AES in software.
>
>> e.g. below key tool command taking several minutes to finish.
>
> Uhh... that's not software versus hardware. You might expect an order of magnitude difference between BC and hardware-accelerated SunJCE. But minutes to generate a 2048-bit RSA key? Something is wrong.
>
>> keytool -providerpath /root/Downloads/bc-fips-1.0.2.jar -providerclass
>> org.bouncycastle.jcajce.provider.BouncyCastleFipsProvider -genkeypair
>> -keyalg RSA -keypass "Test123" -validity 3650 -dname
>> "CN=Test_HostName, ou=My Org Unit, o=My Org" -storepass "Test123"
>> -keystore "/tmp/test_bc.bcfks" -storetype BCFKS -v -alias test_entry
>>
>> However, when I add the JVM option
>> -J-Djava.security.egd=file:/dev/./urandom instead of the default
>> /dev/random, all problems go away.
> Classic entropy problem.
>
>> It's rather strange that multiple 7.3 RHEL systems are showing this
>> behavior very consistently. RHEL 7.2, 8.0 seem just fine in my
>> testing. No issues when using /dev/random along with JKS type key
>> stores (including RHEL 7.3).
>>
>> Even Tomcat is flagging the slowness to generate the secure random
>> numbers.
>>
>> "17-Nov-2020 19:24:59.142 WARNING [Catalina-utility-2] org.apache.catalina.util.SessionIdGeneratorBase.createSecureRandom Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [68,745] milliseconds."
>>
>> Apparently, according to FIPS experts we should not use /dev/./urandom.
>
> Correct. FIPS assumes that 100% of your cryptographic operations are of the *utmost highly-classified state-secret-style nuclear-launch-code
> flashing-purple* security level.
>
> Sometimes you just don't need to have perfect entropy.
>
> For long-lived keys, really good entropy is a Good Thing. So to generate your RSA keys, I'd say go ahead and use the "high quality entropy source". But for randomness for session id generation, and for temporary, bulk encryption TLS session keys, /dev/urandom is perfectly fine.
>
>> So, while I continue to explore further, sharing this in case any one
>> has seen this behavior or be aware of such potential issue as it may
>> tie in to their Tomcat (or web apps to be deployed in Tomcat).
> You might be able to justify using /dev/urandom (which must be spelled /dev/./urandom to force Java to actually use it) in your web application, but I would go ahead and let keytool take as long as it needs.
>
> -chris
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
> For additional commands, e-mail: users-help@tomcat.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@tomcat.apache.org
For additional commands, e-mail: users-help@tomcat.apache.org