[users@httpd] Problem with mod_authnz_ldap w/ldaps?

[Christopher Schultz <ch...@christopherschultz.net>]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

All,

I've been using mod_authnz_ldap for a while with OpenLDAP over TLS and
things have been going fairly well. This weekend, however, I had 3
servers stop working (returning HTTP 500 responses) for pages which
were protected with HTTP Basic auth + mod_authnz_ldap. Two of them
were unavailable while the third one was working, and then the third
one stopped working as well.

I checked everything: connectivity was okay (checked with openssl),
certs were okay (also checked with openssl), and I even installed
OpenLDAP-clients on the server to check to see if 'ldapsearch' worked
(it did).

After several "graceful" restarts, nothing was working. On a
particularly low-activity server, I ran "/etc/init.d/httpd restart"
and the page came right back up with no errors.

Here's what I think has happened.

First, we are using Let's Encrypt for not only our httpd server
certificates but also for the OpenLDAP server.

Second, our most recent certificate for the LDAP server has a "not
before" date of "Oct 28 09:47:18 2018 GMT". That's very likely to be
roughly the time all of this started becoming a problem.

Note: We are using "LDAPVerifyServerCert Off" though that it a vestige
of having self-signed certificates before moving to Let's Encrypt. So
the certificate chain itself shouldn't be a problem.

Third, the two servers that stopped working (roughly) at the same time
have a system clock tracking UTC. Before restarting httpd on the
second of the two servers, I grabbed the date-of-last restart -- that
is, the timestamp of the oldest httpd process on the box:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     15973  0.1  0.6 415204 24060 ?        Ss   Oct22  12:29
/usr/sbin/http

So only graceful restarts have occurred since Oct 22... before the
certificate was re-issued.

Fourth, we use logrotate and the latest logrotation on the
initially-broken servers (using log-file timestamps) shows Oct 28
03:51 UTC.

For the server that *wasn't* suffering until later, we have:
tz: America/New_York (/etc/timezone... /etc/localtime says UTC)
logrotate @ Oct 28 03:33 UCT

logrotate performs "/sbin/service httpd reload" which I believe
performs a "graceful" reload of httpd.

These things don't entirely line-up... I was thinking I'd find that
logrotate on one server ran on a several-hour delay from the other two
and that would explain why one server was working while the others
weren't, but it appears that logrotate ran roughly at the same time on
all of the servers (at least, within a 30 minutes or so).

Is anyone aware of any issues with mod_authnz_ldap caching TLS
certificates or anything like that? I literally changed nothing on the
LDAP server side or the httpd server side, but an httpd "restart"
fixed this problem while no number of "graceful" reloads seemed to
improve things.

Thanks,
- -chris
-----BEGIN PGP SIGNATURE-----
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlvXC3wACgkQHPApP6U8
pFh/cRAAq10ZkYWnI5Rzjrmx93DCRxRajo/XXmFC/9LeJEylCtLslmUd7oBqTSBn
ikV56lWRIoGjVUVG3rGRConQzV76gbokai5q39a3Pf4mcQ2a20pec34aHtNNf7gE
j4YFmSqWnAhZ6NHVAQZg4zLxV4JCS/OrVbU9jlgevZhXhaOFI5cwiCz51HyPE1z6
OPqeXz91P4AJvIH7hd5tXHmIVRopjOhJID15gGkzlNIRNrB00OMGFcTQbcUhzLj+
jClAdsbMCYnjuKRvTUnNVwTTBujj9G6wPBgvt0egdk15UhUDRyJnVciNYF85VgeX
66WrMB0GiPDepNTABWSj1gDd5pNaXd7gun2Lhbn3X7oAvawlYAAvsglvXbJedIzA
y4gJgxn2zd9PU3F2Wf58j/zmMv7YQijcPT4yI+yWTwfYNx+QlcCVboxusY/i8YNE
iN8lUFqh6kwICJazyLXpxNW6gMNoEEW9BRpaEC2doF1pxNk6LmI8ZdN5d24TMyj3
xTMje7jiNA7moYxx2wUR081IpyJezHzTvtmrKfnWz1szI5uYadott0eRq+i6JGw+
7fb3OmA++MJUeB+uM5/O1QndR5zQ8B9XJ4cVTqaZLdG2CIiPJp4T6+D7qWpyWnEi
Qbb0WxFauNFgsN417qrnEjjXLFAdeFJwq81sv4UTtaP3Yfhla74=
=MvMp
-----END PGP SIGNATURE-----

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org