You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@airavata.apache.org by "Marcus Christie (JIRA)" <ji...@apache.org> on 2018/05/23 15:54:00 UTC
[jira] [Comment Edited] (AIRAVATA-2627) Letsencrypt auto renewal is
preventing Apache from restarting
[ https://issues.apache.org/jira/browse/AIRAVATA-2627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16487338#comment-16487338 ]
Marcus Christie edited comment on AIRAVATA-2627 at 5/23/18 3:53 PM:
--------------------------------------------------------------------
Letsencrypt renewal failed on the dreg jetstream instance. From /var/log/httpd/error_log:
{noformat}
[Wed May 23 04:36:15.704698 2018] [mpm_prefork:notice] [pid 29883] AH00171: Graceful restart requested, doing restart
AH00112: Warning: DocumentRoot [/www/default] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
[Wed May 23 04:36:31.015404 2018] [auth_digest:notice] [pid 29883] AH01757: generating secret for digest authentication ...
[Wed May 23 04:36:31.111576 2018] [lbmethod_heartbeat:notice] [pid 29883] AH02282: No slotmem from mod_heartmonitor
[Wed May 23 04:36:31.111741 2018] [ssl:emerg] [pid 29883] (2)No such file or directory: AH02201: Init: Can't open server certificate file /var/lib/letsencrypt/JbZ1--OTKoDFcaH4fCFIxdjQmOmpNE6Win6w4Eclqgw.crt
[Wed May 23 04:36:31.111763 2018] [ssl:emerg] [pid 29883] AH02312: Fatal error initialising mod_ssl, exiting.
{noformat}
From /var/log/letsencrypt/letsencrypt.log
{noformat}
2018-05-23 04:36:31,084:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:
Domain: dreg.dnasequence.org
Type: connection
Detail: Timeout after connect (your server may be slow or overloaded)
To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you're using the webroot plugin, you should also verify that you are serving files from the webroot path you provided.
2018-05-23 04:36:31,085:INFO:certbot.auth_handler:Cleaning up challenges
2018-05-23 04:36:31,207:ERROR:certbot.util:Error while running apachectl graceful.
Job for httpd.service invalid.
2018-05-23 04:36:31,207:WARNING:certbot.renewal:Attempting to renew cert (dreg.dnasequence.org) from /etc/letsencrypt/renewal/dreg.dnasequence.org.conf produced an unexpected error: Error while running apachectl graceful.
Job for httpd.service invalid.
. Skipping.
2018-05-23 04:36:31,311:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 422, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1102, in renew_cert
_get_and_save_cert(le_client, config, lineage=lineage)
File "/usr/lib/python2.7/site-packages/certbot/main.py", line 113, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 297, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains)
File "/usr/lib/python2.7/site-packages/certbot/client.py", line 294, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File "/usr/lib/python2.7/site-packages/certbot/client.py", line 330, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 82, in handle_authorizations
self._respond(resp, best_effort)
File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 159, in _respond
self._cleanup_challenges(active_achalls)
File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 304, in _cleanup_challenges
self.auth.cleanup(achalls)
File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 2109, in cleanup
self.restart()
File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 1989, in restart
self._reload()
File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 2000, in _reload
raise errors.MisconfigurationError(str(err))
MisconfigurationError: Error while running apachectl graceful.
Job for httpd.service invalid.
2018-05-23 04:36:31,311:ERROR:certbot.renewal:All renewal attempts failed. The following certs could not be renewed:
2018-05-23 04:36:31,311:ERROR:certbot.renewal: /etc/letsencrypt/live/dreg.dnasequence.org/fullchain.pem (failure)
{noformat}
This matches pretty well with this certbot issue: https://github.com/certbot/certbot/issues/5439
So I tried to fix by upgrading with yum:
{noformat}
yum makecache fast
yum update python2-certbot-apache
yum update certbot
systemctl start certbot-renew
{noformat}
This upgraded from 0.22 to 0.24.
The letsencrypt renewal worked successfully this time.
was (Author: marcuschristie):
Letsencrypt renewal failed on the dreg jetstream instance. From /var/log/httpd/error_log:
{noformat}
[Wed May 23 04:36:15.704698 2018] [mpm_prefork:notice] [pid 29883] AH00171: Graceful restart requested, doing restart
AH00112: Warning: DocumentRoot [/www/default] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
AH00112: Warning: DocumentRoot [/var/lib/letsencrypt/tls_sni_01_page/] does not exist
[Wed May 23 04:36:31.015404 2018] [auth_digest:notice] [pid 29883] AH01757: generating secret for digest authentication ...
[Wed May 23 04:36:31.111576 2018] [lbmethod_heartbeat:notice] [pid 29883] AH02282: No slotmem from mod_heartmonitor
[Wed May 23 04:36:31.111741 2018] [ssl:emerg] [pid 29883] (2)No such file or directory: AH02201: Init: Can't open server certificate file /var/lib/letsencrypt/JbZ1--OTKoDFcaH4fCFIxdjQmOmpNE6Win6w4Eclqgw.crt
[Wed May 23 04:36:31.111763 2018] [ssl:emerg] [pid 29883] AH02312: Fatal error initialising mod_ssl, exiting.
{noformat}
From /var/log/letsencrypt/letsencrypt.log
{noformat}
2018-05-23 04:36:31,084:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:
Domain: dreg.dnasequence.org
Type: connection
Detail: Timeout after connect (your server may be slow or overloaded)
To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you're using the webroot plugin, you should also verify that you are serving files from the webroot path you provided.
2018-05-23 04:36:31,085:INFO:certbot.auth_handler:Cleaning up challenges
2018-05-23 04:36:31,207:ERROR:certbot.util:Error while running apachectl graceful.
Job for httpd.service invalid.
2018-05-23 04:36:31,207:WARNING:certbot.renewal:Attempting to renew cert (dreg.dnasequence.org) from /etc/letsencrypt/renewal/dreg.dnasequence.org.conf produced an unexpected error: Error while running apachectl graceful.
Job for httpd.service invalid.
. Skipping.
2018-05-23 04:36:31,311:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 422, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File "/usr/lib/python2.7/site-packages/certbot/main.py", line 1102, in renew_cert
_get_and_save_cert(le_client, config, lineage=lineage)
File "/usr/lib/python2.7/site-packages/certbot/main.py", line 113, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File "/usr/lib/python2.7/site-packages/certbot/renewal.py", line 297, in renew_cert
new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains)
File "/usr/lib/python2.7/site-packages/certbot/client.py", line 294, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File "/usr/lib/python2.7/site-packages/certbot/client.py", line 330, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 82, in handle_authorizations
self._respond(resp, best_effort)
File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 159, in _respond
self._cleanup_challenges(active_achalls)
File "/usr/lib/python2.7/site-packages/certbot/auth_handler.py", line 304, in _cleanup_challenges
self.auth.cleanup(achalls)
File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 2109, in cleanup
self.restart()
File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 1989, in restart
self._reload()
File "/usr/lib/python2.7/site-packages/certbot_apache/configurator.py", line 2000, in _reload
raise errors.MisconfigurationError(str(err))
MisconfigurationError: Error while running apachectl graceful.
Job for httpd.service invalid.
2018-05-23 04:36:31,311:ERROR:certbot.renewal:All renewal attempts failed. The following certs could not be renewed:
2018-05-23 04:36:31,311:ERROR:certbot.renewal: /etc/letsencrypt/live/dreg.dnasequence.org/fullchain.pem (failure)
{noformat}
This matches pretty well with this certbot issue: https://github.com/certbot/certbot/issues/5439
So I tried to fix by upgrading with yum:
{noformat}
yum makecache fast
yum update python2-certbot-apache
yum update certbot
systemctl start certbot-renew
{noformat}
The letsencrypt renewal worked successfully this time.
> Letsencrypt auto renewal is preventing Apache from restarting
> -------------------------------------------------------------
>
> Key: AIRAVATA-2627
> URL: https://issues.apache.org/jira/browse/AIRAVATA-2627
> Project: Airavata
> Issue Type: Bug
> Components: PGA PHP Web Gateway
> Reporter: Marcus Christie
> Assignee: Marcus Christie
> Priority: Major
>
> The {{certbot renew --quiet}} command in the crontab is apparently causing Apache to fail to reload:
> From the systemd journal ({{journalctl -xe}}):
> {noformat}
> -- Unit session-34124.scope has begun starting up.
> Jan 09 12:50:01 gridfarm004.ucs.indiana.edu CROND[11610]: (root) CMD (/usr/lib64/sa/sa1 1 1)
> Jan 09 12:52:01 gridfarm004.ucs.indiana.edu systemd[1]: Started Session 34125 of user root.
> -- Subject: Unit session-34125.scope has finished start-up
> -- Defined-By: systemd
> -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
> --
> -- Unit session-34125.scope has finished starting up.
> --
> -- The start-up result is done.
> Jan 09 12:52:01 gridfarm004.ucs.indiana.edu systemd[1]: Starting Session 34125 of user root.
> -- Subject: Unit session-34125.scope has begun start-up
> -- Defined-By: systemd
> -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
> --
> -- Unit session-34125.scope has begun starting up.
> Jan 09 12:52:01 gridfarm004.ucs.indiana.edu CROND[11692]: (root) CMD (/usr/bin/certbot renew --quiet)
> Jan 09 12:52:03 gridfarm004.ucs.indiana.edu httpd[11725]: AH00112: Warning: DocumentRoot [/www/default] does not exist
> Jan 09 12:52:03 gridfarm004.ucs.indiana.edu httpd[11725]: AH00526: Syntax error on line 10 of /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:
> Jan 09 12:52:03 gridfarm004.ucs.indiana.edu httpd[11725]: SSLCertificateFile: file '/var/lib/letsencrypt/YDnHNU3oKDOaT_oO2qXSoXR65gUb7k66KB0dF4nwT-8.crt' does not exist or is empty
> Jan 09 12:52:03 gridfarm004.ucs.indiana.edu systemd[1]: httpd.service: control process exited, code=exited status=1
> Jan 09 12:52:03 gridfarm004.ucs.indiana.edu systemd[1]: Reload failed for The Apache HTTP Server.
> -- Subject: Unit httpd.service has finished reloading its configuration
> -- Defined-By: systemd
> -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
> --
> -- Unit httpd.service has finished reloading its configuration
> --
> -- The result is failed.
> Jan 09 12:52:03 gridfarm004.ucs.indiana.edu httpd[11735]: AH00112: Warning: DocumentRoot [/www/default] does not exist
> Jan 09 12:52:03 gridfarm004.ucs.indiana.edu systemd[1]: Reloaded The Apache HTTP Server.
> -- Subject: Unit httpd.service has finished reloading its configuration
> -- Defined-By: systemd
> -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
> --
> -- Unit httpd.service has finished reloading its configuration
> --
> -- The result is done.
> Jan 09 12:52:05 gridfarm004.ucs.indiana.edu httpd[11757]: AH00112: Warning: DocumentRoot [/www/default] does not exist
> Jan 09 12:52:05 gridfarm004.ucs.indiana.edu httpd[11757]: AH00526: Syntax error on line 10 of /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:
> Jan 09 12:52:05 gridfarm004.ucs.indiana.edu httpd[11757]: SSLCertificateFile: file '/var/lib/letsencrypt/9qLZfLerTerU_bGLYPfXWXq-EXktXgYfNQAEQcdHSpE.crt' does not exist or is empty
> Jan 09 12:52:05 gridfarm004.ucs.indiana.edu systemd[1]: httpd.service: control process exited, code=exited status=1
> Jan 09 12:52:05 gridfarm004.ucs.indiana.edu systemd[1]: Reload failed for The Apache HTTP Server.
> -- Subject: Unit httpd.service has finished reloading its configuration
> -- Defined-By: systemd
> -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
> --
> -- Unit httpd.service has finished reloading its configuration
> --
> -- The result is failed.
> Jan 09 12:52:05 gridfarm004.ucs.indiana.edu httpd[11767]: AH00112: Warning: DocumentRoot [/www/default] does not exist
> Jan 09 12:52:05 gridfarm004.ucs.indiana.edu systemd[1]: Reloaded The Apache HTTP Server.
> -- Subject: Unit httpd.service has finished reloading its configuration
> -- Defined-By: systemd
> -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
> --
> -- Unit httpd.service has finished reloading its configuration
> --
> -- The result is done.
> Jan 09 12:52:07 gridfarm004.ucs.indiana.edu httpd[11796]: AH00112: Warning: DocumentRoot [/www/default] does not exist
> Jan 09 12:52:07 gridfarm004.ucs.indiana.edu httpd[11796]: AH00526: Syntax error on line 10 of /etc/httpd/conf.d/le_tls_sni_01_cert_challenge.conf:
> Jan 09 12:52:07 gridfarm004.ucs.indiana.edu httpd[11796]: SSLCertificateFile: file '/var/lib/letsencrypt/I69cuV1431Lfk88VjtDFxlBPEnagdg5atz9dhGhsxfY.crt' does not exist or is empty
> Jan 09 12:52:07 gridfarm004.ucs.indiana.edu systemd[1]: httpd.service: control process exited, code=exited status=1
> Jan 09 12:52:07 gridfarm004.ucs.indiana.edu systemd[1]: Reload failed for The Apache HTTP Server.
> ...
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)