Let's Encrypt Renewal Triggers Apache Crash

This issue is summarized here: https://forum.virtualmin.com/t/apache-crash-during-lets-encrypt-renewal/...

Essentially, LE requests and receives approval for a cert. Apache gets a restart request, but after the cert file is written, the key file has not yet finished writing to the filesystem, httpd detects the cert/key mismatch and the entire server goes down.

14:45:41.573 - LE reports that it is done 14:45:42.307 - Graceful requested 14:45:42.631 - ssl.cert written (ssl.key not yet modified) 14:45:43.047 - apache is reading configs, ssl mismatch, crash 14:45:43.619 - ssl.key written

Is it possible to - specify a slightly longer delay in requesting the restart of httpd, or - specify the daily schedule that LE checks domains for renewal so that if the issue persists, it manifests during non-peak, non-business-hours

Status: 
Active

Comments

Ilia's picture
Submitted by Ilia on Sat, 07/04/2020 - 09:35

Hi,

Thanks for the heads up.

I remember reading about this in the past on ACME Tiny issue tracker on GitHub. I remember, someone proposed to solve this using an artificially created delay.

Honestly, I don't understand how would that be possible, as when we run the script, we neither cache it nor run it in a background (sub-process)?

Moreover, I could never reproduce this issue or have encountered it myself.

If you have steady steps to reproduce it, share it with us, it can be easily fixed.

I would recommend using an official certbot client for requesting SSL certificates.

Ilia's picture
Submitted by Ilia on Sat, 07/04/2020 - 09:36

Notice: I marked this issue as non-private and cross-referenced it to your public post on forums.

The only way I can see this happen is if Apache was restarted at around the same for some other reason.

Would it be possible to not have LE run during business hours?

Ilia's picture
Submitted by Ilia on Sat, 07/11/2020 - 04:47

It's not normal, neither expected, as all renewals on our side and on all of our servers work without an issue.

It would be interesting to see an errors from a global Apache log, when this happens?

Would it be possible to not have LE run during business hours?

I think the easiest way you could achieve your goal, with rough success though, is to go to Server Configuration/SSL Certificate/Let's Encrypt and simply update renewal only, at a very early hour, let's say 4-5 am. Presumably, it might do the trick.