CRL verification in mod_ssl

[Nicob <ni...@nicob.net>]

Hello,

I'm actually trying to setup a SSL reverse-proxy based on Apache 2.x and
mod_ssl and it seems there's a bug in the verification of the CRL.

If a CA changes its keys before expiration, the CRL is now signed by the
new key and include certificates issued by both the new and old keys.
However, mod_ssl will refuse to work if the AKID of the revoked
certificate doesn't match the issuer of the CRL.

Browsing Apache archives, I found that somebody posted a patch covering
this need (http://marc.info/?l=apache-httpd-dev&m=120350484626015), but
the code haven't been merged. I tested it and it works perfectly well.

Does this patch seems OK to you ? If yes, is it possible to include it ?

Regards,
Nicob

Re: CRL verification in mod_ssl

[Nicob <ni...@nicob.net>]

Le samedi 30 août 2008 à 14:50 +0200, Nicob a écrit :
> It implements the matching on the Authority DN (vs. Authority
> Key ID actually) during client certificate verification against a CRL
> *and* a required test during CRL validation, as described in paragraph
> 6.3.3 of RFC 3280

So, do you think that this patch could be included ?

If not, I plan to open two bug reports :
- one about the matching on the Authority DN (missing feature)
- one about the non verification of the key usage of the issuer
(security bug)

Note about the patch : it could also check at line 68 the return value
of BIO_read() before writing the NULL, even if this code is executed
only in debug mode.

Regards,
Nicob

Re: CRL verification in mod_ssl

[Nicob <ni...@nicob.net>]

> But this is a bit too obscure for me to dare to commit it directly.  
> Could someone else with a good x509 understanding look at it ?

I'm not a x509 expert but I studied the patch and it seems to implement
precisely what is described in RFC 3280 "Internet X.509 Public Key
Infrastructure Certificate and Certificate Revocation List (CRL)
Profile". It implements the matching on the Authority DN (vs. Authority
Key ID actually) during client certificate verification against a CRL
*and* a required test during CRL validation, as described in paragraph
6.3.3 of RFC 3280 :

(f)  Obtain and validate the certification path for the complete CRL
   issuer.  If a key usage extension is present in the CRL issuer's
   certificate, verify that the cRLSign bit is set.

And in the patch :

+         /* Ignore this certificate if it doesn't have the right to
+          * sign CRLs */
+         if ((testedcert->ex_flags & EXFLAG_KUSAGE) && !(testedcert->ex_kusage & KU_CRL_SIGN))
+           continue;

Regards,
Nicob

Re: CRL verification in mod_ssl

[Steve Marquess <ma...@veridicalsystems.com>]

Dr Stephen Henson wrote:

> 
>...
> 
> CRL refresh has some performance issues particularly in multi-process
> servers. For example a CRL might be 500K or more and be reloaded on each
> new connection. OpenSSL 0.9.9 does have some reload support though. If
> CRL processing was delegated to OpenSSL it would be available automatically.

Here's a real world example: I'm supporting an application with hundreds 
of servers deployed worldwide, currently referencing 46 separate CRL 
files totaling 201Mb.  Some of those have TTLs of as little as 18 hours. 
  The largest single CRL file is 30Mb, and of course is the one that is 
referenced the most.

-Steve M.

-- 
Steve Marquess
Veridical Systems, Inc.
1829 Mount Ephraim Road
Adamstown, MD  21710
301-524-9915 cell
301-831-8447 land/fax
marquess@veridicalsystems.com

Re: CRL verification in mod_ssl

[Erwann ABALEA <ea...@gmail.com>]

2008/10/20 Erwann ABALEA <ea...@gmail.com>:
> What is the decision criteria to reload a CRL? expiration of the
> "notAfter" date? An application based period would be better.

s/notAfter/nextUpdate/

-- 
Erwann.

Re: CRL verification in mod_ssl

[Erwann ABALEA <ea...@gmail.com>]

2008/10/15 Dr Stephen Henson <st...@openssl.org>:
> Erwann ABALEA wrote:
>> 2008/10/15 Dr Stephen Henson <st...@openssl.org>:
>>> Dirk-Willem van Gulik wrote:
>>>> On Aug 28, 2008, at 9:41 PM, Nicob wrote:
>> [...]
> This issue does have some security implications. For example a revoked
> client certificate could appear valid by substituting a delta CRL for a
> full CRL.

As I said, you're right. CRL verification is definitely not done
properly by the most widely adopted web server...
The threat is also linked to how much trust is put into the CA
producing those CRLs.

[...]
> I'll have a look at that in more detail. There are some security issues
> with separate CRL and certificate signing keys which were debated on the
> PKIX lists some time ago.

Mr Patterson told me about these issues, but the URLs given pointed to
a "solution" proposed by someone from Microsoft to circumvent the fact
that the CAPI doesn't handle it either. The solution was to have both
the old and new keys sign the CRL.
I'll try to google some informations about this discussion.

> Multiple paths can be used by OpenSSL but each path can contain both
> CRLs and certificates.
>
> Would there be a need to be able to restrict paths to one or the other?

No need to do that. The real need is to make sure identical
configuration variables will lead to the same behaviour.

> CRL refresh has some performance issues particularly in multi-process
> servers. For example a CRL might be 500K or more and be reloaded on each
> new connection.

CRL doesn't need to be "refreshed" on each connection, of course. It
should at least be done on a regular basis.
CRLs need to be carefully verified, I don't think you'll disagree with
that. And "properly" also means checking up-to date CRLs.

Talking big numbers... We operate several CAs, the biggest CRL we
produce is a 83MB one, today... This, in itself, is an aberration, but
as we say here, "le client est roi" (could be translated to: the
customer is king). So this CRL exists, and the webservers this
customer operates do check them. In fact, the same customer divided
its population into 4 pieces, hence 4 CAs, and the 4 CRLs all together
"weigh" 260MB. Refreshing those CRLs on each connection is of course
out of question, everybody agrees with that. But it still needs to be
refreshed on a periodic basis. Those CRLs are produced at least once a
day. Some of them every hour.
That's an extreme case, one OpenSSL can't handle properly (the memory
needed to parse the CRL is really huge). But the arguments are still
valid, for smaller CRLs.

> OpenSSL 0.9.9 does have some reload support though. If
> CRL processing was delegated to OpenSSL it would be available automatically.

What is the decision criteria to reload a CRL? expiration of the
"notAfter" date? An application based period would be better.

[PKITS tests]
> It should be OK. The script pkits-run.pl sets the necessary flags for
> each case. It wont verify all such cases by default.

Thanks, I played with it, it seems there are new execution paths for
me to discover  :)

All in all, CRL validation performed by mod_ssl exists, that's good,
since it was written when OpenSSL didn't provide a "clean" solution.
But it has some several flaws, and now OpenSSL is able to do the job,
maybe it's time to rewrite the glue between them?

-- 
Erwann.

Re: CRL verification in mod_ssl

[Dr Stephen Henson <st...@openssl.org>]

Erwann ABALEA wrote:
> Hello Mr Henson,
> 
> 2008/10/15 Dr Stephen Henson <st...@openssl.org>:
>> Dirk-Willem van Gulik wrote:
>>> On Aug 28, 2008, at 9:41 PM, Nicob wrote:
> [...]
>> While I haven't reviewed this specific patch I have a general comment.
>>
>> There is currently some questionable behaviour in mod_ssl CRL handling.
>> For example it ignores critical CRL extensions, whereas OpenSSL either
>> processes them (if they are recognised) or rejects the CRL (if they are
>> not). This is required by RFC3280 and not doing so raises some security
>> issues.
> 
> You're right. The original mod_ssl code doesn't check this, neither
> for the CRL extensions, nor for the CRLentries extensions.
> It can be added after the signature verification.
> 

This issue does have some security implications. For example a revoked
client certificate could appear valid by substituting a delta CRL for a
full CRL.

>> IMHO it would be best if mod_ssl delegated CRL handling to OpenSSL by
>> setting the necessary verification flags instead of duplicating
>> functionality. In the past mod_ssl didn't have much choice but to do
>> it's own thing because then OpenSSL CRL handling was either absent or
>> broken.
> 
> That was a proposed solution, some months ago, IIRC. Unfortunately,
> OpenSSL has the same behaviour as mod_ssl regarding the problem solved
> by this patch; it considers that the key used to sign the user
> certificate is the same key used to sign the CRL. That assumption is
> false in 2 cases (at least):
>  - a CA using keys+certificates for different usages (certificate
> signing and CRL signing)
>  - a CA renewal (with rekeying)
> 
> I proposed the same kind of patch on the OpenSSL mailing lists some
> months ago, and only debated with Mr Patterson about the merits of a
> CA renewal with/without rekeying. If you want to contact me privately
> to talk about this, feel free to do so.
> 

I'll have a look at that in more detail. There are some security issues
with separate CRL and certificate signing keys which were debated on the
PKIX lists some time ago.

For that reason some of the cases are only permitted when certain
verification flags for extended CRL support are set in OpenSSL 0.9.9.

> I'm also in favor of letting OpenSSL perform the validation path and
> CRL checking, if we can also handle the existing configurations (for
> example, a typical Apache configuration has different
> SSLCACertificatePath and SSLCARevocationPath directives, and OpenSSL
> uses a unified path for both certificates and CRLs). Problems about
> CRL refreshing should also be taken into account; killing and
> restarting a webserver every hour or every day because we downloaded a
> new CRL is not a viable solution in a production environment, and OCSP
> is not always a good answer (we're not talking about a sub-minute
> revocation status).

Multiple paths can be used by OpenSSL but each path can contain both
CRLs and certificates.

Would there be a need to be able to restrict paths to one or the other?

CRL refresh has some performance issues particularly in multi-process
servers. For example a CRL might be 500K or more and be reloaded on each
new connection. OpenSSL 0.9.9 does have some reload support though. If
CRL processing was delegated to OpenSSL it would be available automatically.

If there is sufficient demand reload support could be backported to 0.9.8.

> 
>> Things are rather better now. The 0.9.9-dev version of OpenSSL is
>> compliant with all the RFC3280 PKITS tests for example. The 0.9.8
>> handling isn't quite so complete though.
> 
> OpenSSL 0.9.9-dev is mostly compliant to PKITS, yes, but not fully.
> I'm reading my CVS copy, and based on this reading, Test19 of PKITS
> test isn't OK (this is the case with separate certificate and CRL
> keys). I'm now reading the check_crl() function, in x509_vfy.c file.
> This is also solved by this patch.
> 

It should be OK. The script pkits-run.pl sets the necessary flags for
each case. It wont verify all such cases by default.

The only one it will fail on is 4.14.30, which various docs say does not
need to be run due to a circular dependency.

Steve.

Re: CRL verification in mod_ssl

[Erwann ABALEA <ea...@gmail.com>]

Hello Mr Henson,

2008/10/15 Dr Stephen Henson <st...@openssl.org>:
> Dirk-Willem van Gulik wrote:
>> On Aug 28, 2008, at 9:41 PM, Nicob wrote:
[...]
> While I haven't reviewed this specific patch I have a general comment.
>
> There is currently some questionable behaviour in mod_ssl CRL handling.
> For example it ignores critical CRL extensions, whereas OpenSSL either
> processes them (if they are recognised) or rejects the CRL (if they are
> not). This is required by RFC3280 and not doing so raises some security
> issues.

You're right. The original mod_ssl code doesn't check this, neither
for the CRL extensions, nor for the CRLentries extensions.
It can be added after the signature verification.

> IMHO it would be best if mod_ssl delegated CRL handling to OpenSSL by
> setting the necessary verification flags instead of duplicating
> functionality. In the past mod_ssl didn't have much choice but to do
> it's own thing because then OpenSSL CRL handling was either absent or
> broken.

That was a proposed solution, some months ago, IIRC. Unfortunately,
OpenSSL has the same behaviour as mod_ssl regarding the problem solved
by this patch; it considers that the key used to sign the user
certificate is the same key used to sign the CRL. That assumption is
false in 2 cases (at least):
 - a CA using keys+certificates for different usages (certificate
signing and CRL signing)
 - a CA renewal (with rekeying)

I proposed the same kind of patch on the OpenSSL mailing lists some
months ago, and only debated with Mr Patterson about the merits of a
CA renewal with/without rekeying. If you want to contact me privately
to talk about this, feel free to do so.

I'm also in favor of letting OpenSSL perform the validation path and
CRL checking, if we can also handle the existing configurations (for
example, a typical Apache configuration has different
SSLCACertificatePath and SSLCARevocationPath directives, and OpenSSL
uses a unified path for both certificates and CRLs). Problems about
CRL refreshing should also be taken into account; killing and
restarting a webserver every hour or every day because we downloaded a
new CRL is not a viable solution in a production environment, and OCSP
is not always a good answer (we're not talking about a sub-minute
revocation status).
On the other way, letting Apache perform the certificate validation
permits the use of OCSP, transparently, if the CA is configured to do
so.

> Things are rather better now. The 0.9.9-dev version of OpenSSL is
> compliant with all the RFC3280 PKITS tests for example. The 0.9.8
> handling isn't quite so complete though.

OpenSSL 0.9.9-dev is mostly compliant to PKITS, yes, but not fully.
I'm reading my CVS copy, and based on this reading, Test19 of PKITS
test isn't OK (this is the case with separate certificate and CRL
keys). I'm now reading the check_crl() function, in x509_vfy.c file.
This is also solved by this patch.

> If the OpenSSL CRL handling needs to be made more mod_ssl friendly I'd
> be happy to coordinate changes to OpenSSL.

So do I, in my spare time. We use OpenSSL and other PKI toolkits in a
business environment, so their evolution is also our concern.

-- 
Erwann.

Re: CRL verification in mod_ssl

[Dr Stephen Henson <st...@openssl.org>]

Dirk-Willem van Gulik wrote:
> 
> On Aug 28, 2008, at 9:41 PM, Nicob wrote:
> 
>> Hello,
>>
>> I'm actually trying to setup a SSL reverse-proxy based on Apache 2.x and
>> mod_ssl and it seems there's a bug in the verification of the CRL.
>>
>> If a CA changes its keys before expiration, the CRL is now signed by the
>> new key and include certificates issued by both the new and old keys.
>> However, mod_ssl will refuse to work if the AKID of the revoked
>> certificate doesn't match the issuer of the CRL.
>>
>> Browsing Apache archives, I found that somebody posted a patch covering
>> this need (http://marc.info/?l=apache-httpd-dev&m=120350484626015), but
>> the code haven't been merged. I tested it and it works perfectly well.
>>
>> Does this patch seems OK to you ? If yes, is it possible to include it ?
> 
> I just tried that patch - and it also matched two of my edge cases.
> 
> But this is a bit too obscure for me to dare to commit it directly.
> Could someone else with a good x509 understanding look at it ?
> 
> +1 from me - willing to do the legwork if someone else gives this a good
> review as well.
> 
> 

While I haven't reviewed this specific patch I have a general comment.

There is currently some questionable behaviour in mod_ssl CRL handling.
For example it ignores critical CRL extensions, whereas OpenSSL either
processes them (if they are recognised) or rejects the CRL (if they are
not). This is required by RFC3280 and not doing so raises some security
issues.

IMHO it would be best if mod_ssl delegated CRL handling to OpenSSL by
setting the necessary verification flags instead of duplicating
functionality. In the past mod_ssl didn't have much choice but to do
it's own thing because then OpenSSL CRL handling was either absent or
broken.

Things are rather better now. The 0.9.9-dev version of OpenSSL is
compliant with all the RFC3280 PKITS tests for example. The 0.9.8
handling isn't quite so complete though.

If the OpenSSL CRL handling needs to be made more mod_ssl friendly I'd
be happy to coordinate changes to OpenSSL.

Steve.

Re: CRL verification in mod_ssl

[Dirk-Willem van Gulik <di...@webweaving.org>]

On Aug 28, 2008, at 9:41 PM, Nicob wrote:

> Hello,
>
> I'm actually trying to setup a SSL reverse-proxy based on Apache 2.x  
> and
> mod_ssl and it seems there's a bug in the verification of the CRL.
>
> If a CA changes its keys before expiration, the CRL is now signed by  
> the
> new key and include certificates issued by both the new and old keys.
> However, mod_ssl will refuse to work if the AKID of the revoked
> certificate doesn't match the issuer of the CRL.
>
> Browsing Apache archives, I found that somebody posted a patch  
> covering
> this need (http://marc.info/?l=apache-httpd-dev&m=120350484626015),  
> but
> the code haven't been merged. I tested it and it works perfectly well.
>
> Does this patch seems OK to you ? If yes, is it possible to include  
> it ?

I just tried that patch - and it also matched two of my edge cases.

But this is a bit too obscure for me to dare to commit it directly.  
Could someone else with a good x509 understanding look at it ?

+1 from me - willing to do the legwork if someone else gives this a  
good review as well.


Dw