Sisir.me

Root Cause Analysis - Linkfy TLS Certificate Failure: A Kubernetes ACME Incident

Published on Sat Sep 12 2026

Note: An LLM was used extensively to investigate and resolve this incident, and this root cause analysis was created with the help of the LLM. I used my own Kubernetes, networking, and infrastructure knowledge together with the LLM to narrow down the problem quickly, validate the findings, and resolve the incident.

When Linkfy’s SSL Certificates Started Failing: A Kubernetes ACME Incident

Recently, users of Linkfy, a SaaS platform I manage, started experiencing problems with SSL certificates for their custom domains.

Linkfy allows customers to use their own domains through a whitelabel setup. When a new domain is configured, the platform automatically provisions a TLS certificate using Let’s Encrypt through cert-manager.

Suddenly, certificate issuance started failing.

The interesting part was that, although the issue initially pointed directly at cert-manager or Let’s Encrypt. The actual problem was deeper inside the Kubernetes control plane.

The symptom

The initial cert-manager error looked like this:

Waiting for HTTP-01 challenge propagation:
wrong status code '503', expected '200'

The certificate request was using the normal Let’s Encrypt HTTP-01 challenge mechanism.

The expected flow was:

Let's Encrypt
      |
      | HTTP request
      v
Ingress
      |
      v
ACME solver Service
      |
      v
ACME solver Pod
      |
      v
HTTP 200

Instead, Let’s Encrypt was receiving:

HTTP 503

At first glance, this looked like an ingress, cert-manager, DNS, or solver problem.

It wasn’t.

First investigation: the ACME solver pods

The first thing I checked was whether cert-manager was actually creating the temporary ACME solver pods.

Eventually, the solver pod existed and was healthy:

1/1 Running

The corresponding Service also existed.

But there was an important problem.

The Service had no endpoints.

That meant Kubernetes had a Service such as:

cm-acme-http-solver-49pbc

but no backend:

Endpoints: <none>

This explained the HTTP 503.

Nginx was receiving the request, forwarding it toward the Kubernetes Service, and Kubernetes had no backend endpoint to send the request to.

So the next question became: Why wasn’t Kubernetes creating an endpoint for a healthy pod?

Looking at the EndpointSlice

Kubernetes now primarily uses EndpointSlices to represent Service backends.The Service selector matched the ACME solver pod correctly. The pod was also Ready. So the expected EndpointSlice should have been created automatically. But it wasn’t. This shifted the investigation away from cert-manager and toward the Kubernetes control plane.

The scheduler was also failing

Around the same time, I discovered something more serious. The Kubernetes scheduler was producing Unauthorized errors. The scheduler’s job is to watch for unscheduled pods and assign them to nodes.

That explained another part of the incident. The ACME solver pods were initially stuck in Pending because the scheduler wasn’t functioning correctly.

I verified several things:

  • The scheduler client certificate was valid.
  • The certificate and private key matched.
  • The certificate was issued for system:kube-scheduler.
  • The scheduler ClusterRoleBinding was correct.
  • The scheduler kubeconfig was valid.
  • Kubernetes authorization checks succeeded when tested manually.
  • There was no proxy configuration interfering with API access.

Yet the scheduler process itself was still receiving:

Unauthorized

This was an important clue. The configuration was correct. The running process was the problem.

The stale container behind the static pod

Kubernetes control-plane components in this cluster are managed as static pods. The scheduler manifest was located at:

/etc/kubernetes/manifests/kube-scheduler.yaml

At first, I was looking at the scheduler as a Kubernetes Pod. But in this case, the distinction between the Pod and the container running inside it turned out to be important.

I inspected the actual container through the container runtime rather than relying only on Kubernetes’ view of the Pod. That revealed something surprising: the scheduler’s actual container had been running since July 7.

The Pod existed. Kubernetes knew about the Pod. But the underlying scheduler container was effectively stale.

This explained why some of the normal Kubernetes-level actions did not resolve the problem. Removing the static Pod object did not address the underlying container that was still running with its broken API authentication state.

I needed kubelet to actually recreate the scheduler container.

A harmless change to the static Pod manifest triggered kubelet to reconcile the manifest and create a new container. Once the new container started, the Unauthorized errors disappeared and the ACME solver pods were immediately able to schedule.

This distinction was one of the most important findings in the incident:

The problem wasn’t simply a stale Kubernetes Pod. The actual control-plane container running behind the Pod was stale.

But the certificate still wasn’t working

At this point the solver pod was running. However, the Service still had no endpoint. That led to the second major discovery. The kube-controller-manager was also stale. Again, the container runtime showed that its actual container had been created on July 7.

The controller-manager was also producing:

error retrieving resource lock kube-system/kube-controller-manager:
Unauthorized

The controller-manager is responsible for many Kubernetes reconciliation loops, including the controllers involved in maintaining Service and EndpointSlice state. Therefore, we now had two independent control-plane components with broken API authentication:

                 Kubernetes API
                       |
          +------------+------------+
          |                         |
          v                         v
   kube-scheduler          kube-controller-manager
          |                         |
          X                         X
    Unauthorized              Unauthorized
          |                         |
          v                         v
  Solver pod stuck           EndpointSlice missing

The result was exactly what we were seeing.

Recreating kube-controller-manager

I again forced kubelet to recreate the actual static container by making a harmless change to:

/etc/kubernetes/manifests/kube-controller-manager.yaml

The new container was created successfully, and as expected the Unauthorized errors stopped Almost immediately, Kubernetes created the missing EndpointSlice. The Service finally had a backend:

192.168.167.187:8089

with the endpoint marked ready.

The ACME HTTP-01 path was now complete:

Let's Encrypt
      |
      v
Ingress
      |
      v
ACME solver Service
      |
      v
192.168.167.187:8089
      |
      v
HTTP 200

The certificate recovered

After the control-plane components were healthy again, cert-manager was able to complete the ACME challenge. The Order eventually reached:

STATE: valid

The CertificateRequest became:

READY: True

And the TLS Certificate was also:

READY: True

There was no need to reinstall cert-manager or rebuild the Let’s Encrypt integration. The ACME system itself was working correctly.

Root cause

The primary root cause was stale Kubernetes control-plane static pod containers.

Both:

kube-scheduler
kube-controller-manager

had been running since July 7 and were experiencing API authentication failures. The scheduler failure caused newly created ACME solver pods to remain unscheduled. The controller-manager failure prevented the necessary Service endpoint reconciliation.

Together, this produced the external symptom:

Let's Encrypt
      |
      v
HTTP-01 challenge
      |
      v
Ingress
      |
      v
Kubernetes Service
      |
      X
No backend endpoint
      |
      v
HTTP 503

This made the incident initially look like a cert-manager or Let’s Encrypt problem, but the actual failure was in the Kubernetes control plane.

Why was this difficult to diagnose?

  • The most misleading part was that Kubernetes appeared healthy at first.
  • The API server was accessible.
  • The node was Ready.
  • The control-plane pods appeared to exist.
  • The scheduler and controller-manager manifests were correct.
  • The certificates were valid.
  • RBAC was correct.

But the actual containers running underneath those static pods were stale.

That distinction between the Kubernetes object’s apparent state and the actual container runtime state was the key to solving the problem.

Checking crictl was what exposed it.

For example:

crictl ps --name kube-scheduler

and:

crictl ps --name kube-controller-manager

showed the real container creation and start times.

What fixed the incident?

The final remediation was surprisingly simple. I forced kubelet to recreate the stale static pods by making harmless changes to their manifests:

/etc/kubernetes/manifests/kube-scheduler.yaml
/etc/kubernetes/manifests/kube-controller-manager.yaml

Once the containers were genuinely recreated:

  1. Scheduler API authentication recovered.
  2. ACME solver pods were scheduled.
  3. Controller-manager API authentication recovered.
  4. EndpointSlices were created.
  5. ACME solver Services received endpoints.
  6. HTTP-01 validation succeeded.
  7. Let’s Encrypt issued the certificate.

No cert-manager reinstall was required.

Lessons learned

1. Don’t assume a Kubernetes pod object means the underlying container is healthy

For static pods in particular, check the container runtime when something doesn’t make sense.

crictl ps
crictl inspect <container-id>

The runtime can tell you when the actual container was created and started.

2. Follow the dependency chain

The original error was:

HTTP 503

It would have been easy to focus exclusively on ingress. Instead, following the chain backwards was more useful:

503
 ↓
Service
 ↓
No endpoints
 ↓
EndpointSlice controller
 ↓
controller-manager
 ↓
Unauthorized
 ↓
stale container

At the same time:

ACME solver Pending
 ↓
scheduler
 ↓
Unauthorized
 ↓
stale container

The two symptoms eventually converged on the same underlying problem: stale control-plane components.

3. Don’t reinstall the application that reports the error too quickly

cert-manager was reporting the failure, but cert-manager wasn’t broken. It was correctly reporting that the HTTP-01 challenge endpoint returned 503. The actual failure was lower in the stack.

4. Whitelabel systems need resource hygiene

Linkfy can create a large number of Kubernetes and cert-manager resources because every customer domain can potentially require its own certificate. Domains with incorrect or missing DNS records can leave behind failed certificate requests and related resources.

That means a production whitelabel system should have monitoring and cleanup policies around:

  • CertificateRequests
  • Orders
  • Challenges
  • failed certificates
  • abandoned domains
  • invalid DNS configurations

This incident reinforced the importance of keeping those resources under control.

Final outcome

The certificate issuance pipeline is now working again. The important conclusion is that Let’s Encrypt, cert-manager, ingress, and the ACME solver were not the fundamental problem.

The actual issue was a stale Kubernetes control plane. Once the scheduler and controller-manager were genuinely recreated, the entire certificate issuance pipeline recovered without reinstalling or replacing the certificate-management components.

This was a good reminder that Kubernetes incidents often require looking one layer below what the Kubernetes API appears to be telling you.

Root Cause Summary

Primary root cause: Stale kube-scheduler and kube-controller-manager static pod containers with broken API authentication state.

Impact: Linkfy users could not successfully provision TLS certificates for custom/whitelabel domains.

Immediate symptom: Let’s Encrypt HTTP-01 validation received HTTP 503 instead of 200.

Failure chain:

Stale scheduler
      ↓
ACME solver pods not scheduled

Stale controller-manager
      ↓
EndpointSlices not reconciled
      ↓
ACME Service had no endpoints
      ↓
Ingress returned HTTP 503
      ↓
Let's Encrypt HTTP-01 validation failed
      ↓
Certificate issuance failed

Resolution: Force recreation of the stale scheduler and controller-manager static pod containers.

Cert-manager reinstall required: No.

Let’s Encrypt problem: No.

Ingress problem: No.

Main lesson: When Kubernetes behaves inconsistently, inspect the actual control-plane containers through the container runtime instead of relying solely on the Kubernetes API view.