We are doing our best to keep our services always up and running. Despite all commitment and caution it can happen that you experience a disruption of our services. While you and we rely on a number of third party services, the problem (and solution) may even be beyond our power. On this page we will log incidents with some background information of what happened and once the cause is known, the details will be shared.
Let’s Encrypt certificate issues (April 30, 2021)
On Friday April 30, 2021, around 7:40 PM UTC, a significant number of HTTPS monitors of multiple Uptrends customers started reporting errors, stating that the HTTPS certificate could not be validated. Not all monitors showed this problem; it happened only for sites using a TLS certificate issued by the Let’s Encrypt certificate authority.
Background: HTTPS monitors perform certificate checks
HTTPS monitors check the availability of the specified URL. They also check the validity of the HTTPS certificate presented by the server, if the option Check SSL certificate errors on the Advanced tab of the monitor settings is active. Certificates are only valid if they haven’t expired yet. Aside from expiring automatically at some point (typically after a year), certificates can also be revoked by the certificate authority. Therefore, in order to perform a solid check and to ensure that the certificate can be trusted, the HTTPS certificate check also needs to verify that the certificate hasn’t been revoked. Without that, the check is essentially inconclusive.
What was the problem?
The revocation checks happen in two ways: through OCSP, and through a certificate revocation list (CRL). Several hours after the incident started, reports by Let’s Encrypt staff revealed that they had been serving an expired CRL, which caused CRL checks to fail and report errors. Consequently, Uptrends monitors reported a possible insecure situation as the validity of these certificates simply couldn’t be determined.
This wasn’t affecting only Uptrends monitors: anyone using .NET or Java code to access sites and APIs would have run into this issue. The problem was solved by Let’s Encrypt on Saturday May 1, 2021, at 12:04 AM UTC.
Browsers did not report this problem
Browsers often use their own internal certificate revocation lists, which do not rely on certificate authorities. As a result, affected web sites showed up OK in a browser.
Conclusion, recommendations and follow-up
There was a genuine problem. Therefore, the error messages (errors or alerts? both?) generated by the Uptrends HTTPS errors were correct, since we could not guarantee the validity of certificates, and the security they are meant to provide.
However, we realize that it was virtually impossible for you to take any action to solve the issue, as the disruption was entirely caused by external factors. To give you more options in the future, our engine teams will consider adding additional settings that let you decide the level of certificate checks you want to execute (including revocation checks or not).
When a problem like this happens, and you’re certain that you want to temporarily ignore this type of error, you can bypass certificate checks by deactivating the Check SSL certificate errors on the Advanced tab of the monitor settings.
The Let’s Encrypt status report for this issue is posted at https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/608c9dd384a5cf052fc6ed24.