You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by "J.Lance Wilkinson" <jl...@psulias.psu.edu> on 2010/12/01 05:37:44 UTC
[users@httpd] Apache HTTPD 2.2.6 + mod_ssl 2.2.6 -- odd error...
My organization recently switched its SSL Certificate vendor and the new
supplier (COMODO) insists (reasonably) that we use 2048-bit Private and Public
keys.
So I take a running Apache installation, HTTPD v2.2.6, with mod_ssl v2.2.6 and
openssl v0.9.8g running on Solaris 10, currently using a Thawte certificate,
and upgrade it for the new vendor's certificates.
I implement the new certificates. reboot httpd, and both aspects where the new
certificate is used in the server (mod_ssl and an additional module, mod_cosign
from http://weblogin.org) seem to be working properly. That is, mod_cosign
works as expected providing single signon features, and mod_ssl appears to be
encrypting properly. Short of sniffing the wire to verify the data between
browser and server, the little padlock icons are proudly displayed by the
browser and page info displays confirm security by the vendor expected, dates
expected, etc.
But my httpd log files present an unexpected error each and every time a
browser visits an SSL encrypted page (2 examples cited):
User interface error
unable to load Private Key
22188:error:0906A068:PEM routines:PEM_do_header:bad password
read:/on10/build-nd/G10U10B0B/usr/src/common/openssl/crypto/pem/pem_lib.c:401:
User interface error
unable to load Private Key
22439:error:0906A068:PEM routines:PEM_do_header:bad password
read:/on10/build-nd/G10U10B0B/usr/src/common/openssl/crypto/pem/pem_lib.c:401:
Any idea what these might be?
I have already verified that the private key file is NOT password protected.
I've also seen notations on both sites for Apache and mod_ssl:
"Why does my 2048-bit private key not work?"
http://www.modssl.org/docs/2.8/ssl_faq.html
http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#keysize
both seem to say say that 2048-bit private keys are NOT ALLOWED because of
incompatibility w/ certain web browsers. Meanwhile it's not clear that I could
even generate a 2048-bit public key without having a 2048-bit private key. So
how could these COMODO certs EVER work if this was the issue?
Count this with a layer of extreme urgency, as this new vendor is my only
source for certificates now, and I have two production webservers with current
certs expiring in about 30 hours that I need to replace w/ these new certs.
Another server in the organization running RHEL v2.2.3 has no such issues;
naturally the powers that be have no examples of v2.2.6 on Solaris to compare
against.
--
J.Lance Wilkinson ("Lance") InterNet: Lance.Wilkinson@psu.edu
Systems Design Specialist - Lead Phone: (814) 865-4870
Digital Library Technologies FAX: (814) 863-3560
E3 Paterno Library
Penn State University
University Park, PA 16802
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache HTTPD 2.2.6 + mod_ssl 2.2.6 -- odd error...
Posted by Sander Temme <sc...@apache.org>.
On Dec 1, 2010, at 7:30 AM, J.Lance Wilkinson wrote:
> In hindsight, the fact these messages were not prefixed by typical
> Apache error log tokens like like [timestamp] [severity] [client]
> should have been the tipoff here.
That is true, log messages from the server (through ap_log_error() variants) have a well defined format.
> MANAGE SSL CERTIFICATES. The output from backtick'd calls to the
> OPENSSL utility was being output to syserr which of course PHP (using
> mod_php5) was shunting to the Apache error log for lack of a better
> place to put it.
Apache redirects the stderr stream to its error log after it starts up. PHP et. al. have no influence over this.
> Should be safe to proceed with the other applications of these new
> certificates that are so critical today.
>
> Thanks for everybody's patience and indulgence -- clearly this should
> never have been posted here, but I was desperate.
Good to hear, happy to help out. That's why we have this list!
S.
--
Sander Temme
sctemme@apache.org
PGP FP: FC5A 6FC6 2E25 2DFD 8007 EE23 9BB8 63B0 F51B B88A
View my availability: http://tungle.me/sctemme
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache HTTPD 2.2.6 + mod_ssl 2.2.6 -- odd error...
Posted by "J.Lance Wilkinson" <jl...@psulias.psu.edu>.
J.Lance Wilkinson wrote:
> Sander Temme wrote:
>> On Nov 30, 2010, at 8:37 PM, J.Lance Wilkinson wrote:
>>
>>> But my httpd log files present an unexpected error each and every
>>> time a browser visits an SSL encrypted page (2 examples cited):
>>
>> So there is no discernible negative impact on the client?
>
> Correct. At the moment, the only negative impact is the considerably
> larger error log files being generated.
>
> Furthermore, watching the error log with tail -f, I can say that the
> errors are NOT being thrown EVERY time a browser visits an SSL
> encrypted page.
>>
>>> User interface error unable to load Private Key 22439:error:0906A068:PEM
>>> routines:PEM_do_header:bad password
>>> read:/on10/build-nd/G10U10B0B/usr/src/common/openssl/crypto/pem/pem_lib.c:401:
In hindsight, the fact these messages were not prefixed by typical
Apache error log tokens like like [timestamp] [severity] [client]
should have been the tipoff here.
These errors are being thrown not by Apache, mod_ssl or mod_cosign.
That explains why they don't appear at server startup, and normal
client activity doesn't seem to be affected.
The errors are being thrown by explicit executions of the OPENSSL
utility from within PHP scripts on the application that was used to
test out the new certificates -- an application used to, ironically,
MANAGE SSL CERTIFICATES. The output from backtick'd calls to the
OPENSSL utility was being output to syserr which of course PHP (using
mod_php5) was shunting to the Apache error log for lack of a better
place to put it.
Should be safe to proceed with the other applications of these new
certificates that are so critical today.
Thanks for everybody's patience and indulgence -- clearly this should
never have been posted here, but I was desperate.
--
J.Lance Wilkinson ("Lance") InterNet: Lance.Wilkinson@psu.edu
Systems Design Specialist - Lead Phone: (814) 865-4870
Digital Library Technologies FAX: (814) 863-3560
E3 Paterno Library
Penn State University
University Park, PA 16802
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache HTTPD 2.2.6 + mod_ssl 2.2.6 -- odd error...
Posted by "J.Lance Wilkinson" <jl...@psulias.psu.edu>.
Sander Temme wrote:
> On Nov 30, 2010, at 8:37 PM, J.Lance Wilkinson wrote:
>
>> But my httpd log files present an unexpected error each and every time a
>> browser visits an SSL encrypted page (2 examples cited):
>
> So there is no discernible negative impact on the client?
Correct. At the moment, the only negative impact is the considerably
larger error log files being generated.
Furthermore, watching the error log with tail -f, I can say that the
errors are NOT being thrown EVERY time a browser visits an SSL
encrypted page.
>
>> User interface error unable to load Private Key 22439:error:0906A068:PEM
>> routines:PEM_do_header:bad password
>> read:/on10/build-nd/G10U10B0B/usr/src/common/openssl/crypto/pem/pem_lib.c:401:
>>
The errors seem to be thrown, however, when visiting a [previously
authenticated] directory/resource protected by the single signon module
mod_cosign (http://weblogin.org).
>> Any idea what these might be?
>
> It's hard to guess what's going on here without a backtrace. A cursory
> glance at the OpenSSL source suggests that things FAIL when this error is
> triggered, so successful SSL connections seem unlikely under those
> circumstances. I would not be surprised if this should cause your server to
> fail to start.
I'm seeking local assistance in getting that backtrace Sander
suggested.
> So the fact that it doesn't happen when the server starts (which is when we
> read the SSL private keys and certificates from disk), and does not cause
> the connections to the browser to fail, suggests that this does not have
> anything to do with mod_ssl.
I agree, especially now that the latest observations show it not
being thrown when the only module dealing w/ Public/Private keys
involved is mod_ssl.
>
> What other modules do you have that might be reading a private key from a
> PEM blob on every request?
That would be mod_cosign's CosignCrypto directive.
>
>> I have already verified that the private key file is NOT password
>> protected. I've also seen notations on both sites for Apache and mod_ssl:
>>
>> "Why does my 2048-bit private key not work?"
>> http://www.modssl.org/docs/2.8/ssl_faq.html
>> http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#keysize
>>
>> both seem to say say that 2048-bit private keys are NOT ALLOWED because of
>> incompatibility w/ certain web browsers. Meanwhile it's not clear that I
>> could even generate a 2048-bit public key without having a 2048-bit
>> private key. So how could these COMODO certs EVER work if this was the
>> issue?
>
> Surely that is very old and no longer relevant. If you visit
> https://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#keysize , you will find
> it protected by a 4096 bit key.
Not sure how I would see that, closest I can see is the "Subject's
Public Key" is "Size: 526 Bytes / 4208 Bits". But I believe you.
Ironic, isn't it, that the documentation is so clearly refuted by
reality on the same page ;-P
>
>> Count this with a layer of extreme urgency, as this new vendor is my only
>> source for certificates now, and I have two production webservers with
>> current certs expiring in about 30 hours that I need to replace w/ these
>> new certs.
>
> Besides the weird error messages, what is the impact on functionality at
> this point?
Honestly, I'm not sure. The particular site I chose to verify the
new certificates installation procedure is a low-volume site. The
site that I have to apply these certs to TODAY is a very HIGH volume
site, so at the very least that server's log files are going to
explode.
The site in question is the University Libraries' bread-and-butter
site, the Online Public Access Catalog, https://cat.libraries.psu.edu.
Whenever any of our literally millions of users authenticates and
goes to access their personal information (reserves, personalized
searches, requests & recalls, renewals, etc.) we're going to see
similar messages logged. And if indeed there's something more
sinister taking place, it's going to happen a LOT more.
--
J.Lance Wilkinson ("Lance") InterNet: Lance.Wilkinson@psu.edu
Systems Design Specialist - Lead Phone: (814) 865-4870
Digital Library Technologies FAX: (814) 863-3560
E3 Paterno Library
Penn State University
University Park, PA 16802
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache HTTPD 2.2.6 + mod_ssl 2.2.6 -- odd error...
Posted by Sander Temme <sc...@apache.org>.
On Nov 30, 2010, at 8:37 PM, J.Lance Wilkinson wrote:
> But my httpd log files present an unexpected error each and every time a
> browser visits an SSL encrypted page (2 examples cited):
So there is no discernible negative impact on the client?
> User interface error
> unable to load Private Key
> 22439:error:0906A068:PEM routines:PEM_do_header:bad password
> read:/on10/build-nd/G10U10B0B/usr/src/common/openssl/crypto/pem/pem_lib.c:401:
>
> Any idea what these might be?
It's hard to guess what's going on here without a backtrace. A cursory glance at the OpenSSL source suggests that things FAIL when this error is triggered, so successful SSL connections seem unlikely under those circumstances. I would not be surprised if this should cause your server to fail to start.
So the fact that it doesn't happen when the server starts (which is when we read the SSL private keys and certificates from disk), and does not cause the connections to the browser to fail, suggests that this does not have anything to do with mod_ssl.
What other modules do you have that might be reading a private key from a PEM blob on every request?
> I have already verified that the private key file is NOT password protected. I've also seen notations on both sites for Apache and mod_ssl:
>
> "Why does my 2048-bit private key not work?"
> http://www.modssl.org/docs/2.8/ssl_faq.html
> http://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#keysize
>
> both seem to say say that 2048-bit private keys are NOT ALLOWED because of incompatibility w/ certain web browsers. Meanwhile it's not clear that I could even generate a 2048-bit public key without having a 2048-bit private key. So how could these COMODO certs EVER work if this was the issue?
Surely that is very old and no longer relevant. If you visit https://httpd.apache.org/docs/2.0/ssl/ssl_faq.html#keysize , you will find it protected by a 4096 bit key.
> Count this with a layer of extreme urgency, as this new vendor is my only
> source for certificates now, and I have two production webservers with current
> certs expiring in about 30 hours that I need to replace w/ these new certs.
Besides the weird error messages, what is the impact on functionality at this point?
S.
--
Sander Temme
sctemme@apache.org
PGP FP: FC5A 6FC6 2E25 2DFD 8007 EE23 9BB8 63B0 F51B B88A
View my availability: http://tungle.me/sctemme
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org