I use the same public domain for services across my homelab, but those services don’t all live in Kubernetes.
HAProxy runs on my pfSense firewalls and remains the general-purpose reverse proxy for the environment. It handles public ingress and also fronts services running outside Kubernetes. Inside my k3s cluster, Traefik handles Kubernetes ingress.
The basic traffic flow looks like this:
Internet
|
Cloudflare
|
pfSense / HAProxy
/ \
/ \
Non-Kubernetes Traefik
services |
|
Kubernetes services
For Kubernetes services, I use cert-manager with Cloudflare DNS-01 challenges to obtain a wildcard certificate. That certificate is configured as Traefik’s default TLS certificate, so I don’t have to configure a certificate individually on every Ingress.
It’s a simple setup and, once configured, certificate renewal is completely automatic.
There was just one problem: cert-manager couldn’t issue the certificate.
The DNS-01 challenge remained pending even though the _acme-challenge TXT record existed in Cloudflare and was publicly resolvable.
The problem wasn’t Cloudflare, Let’s Encrypt, or DNS propagation.
It was split-horizon DNS.
The certificate setup
Rather than issuing a separate certificate for every Kubernetes application, cert-manager requests a wildcard certificate covering the domain used by the cluster.
The certificate is obtained using an ACME DNS-01 challenge through Cloudflare:
cert-manager
|
| create TXT record
v
Cloudflare DNS
|
v
_acme-challenge.example.com
|
| validate
v
ACME CA
DNS-01 works particularly well for a homelab because the application itself doesn’t have to be publicly reachable. Domain ownership is demonstrated through DNS rather than by having the ACME server connect to the application.
Once issued, I configure the resulting wildcard certificate as Traefik’s default certificate.
An Ingress can therefore be fairly minimal:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example
spec:
rules:
- host: example.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: example
port:
number: 80
There’s no TLS Secret that has to be specified for every application. If the Ingress doesn’t specify another certificate, Traefik presents the wildcard certificate.
This is specifically how I’m handling certificates inside Kubernetes. HAProxy hasn’t gone away. It continues to handle the firewall/edge side of the environment and services that aren’t running in the cluster.
The two layers solve different problems:
HAProxy
General-purpose / edge reverse proxy
Runs on pfSense
Handles Kubernetes and non-Kubernetes destinations
Traefik
Kubernetes ingress controller
Runs inside k3s
Routes requests to Kubernetes Services
That separation let me migrate applications into Kubernetes without redesigning the ingress architecture for everything else in the homelab.
The DNS-01 challenge that wouldn’t complete
When cert-manager attempted to issue the wildcard certificate, the challenge remained pending.
The obvious first check was Cloudflare.
cert-manager had successfully created the TXT record:
_acme-challenge.example.com TXT "<challenge>"
Querying a public resolver returned it:
dig TXT _acme-challenge.example.com @1.1.1.1
So did Google:
dig TXT _acme-challenge.example.com @8.8.8.8
The record existed at the authoritative nameservers and was visible through public recursive DNS.
That ruled out several obvious explanations. The Cloudflare credentials worked. cert-manager could create the record. The record wasn’t waiting to propagate to the public resolvers I tested.
But cert-manager still didn’t consider the challenge ready.
DNS-01 includes a self-check
Creating the record isn’t the end of cert-manager’s involvement.
Before proceeding with ACME validation, cert-manager performs its own DNS self-check to verify that the challenge record is actually resolvable.
That makes sense. Successfully sending an API request to a DNS provider doesn’t guarantee that the resulting TXT record can already be resolved.
The important question, therefore, became:
What DNS server was cert-manager asking?
My manual tests were querying 1.1.1.1 and 8.8.8.8.
cert-manager wasn’t.
Its DNS queries were following the normal Kubernetes DNS path.
And the normal Kubernetes DNS path in my network is intentionally different from the public DNS.
My internal and public DNS don’t return the same answers
I use split-horizon DNS for my public domain.
Externally, Cloudflare is authoritative and provides the public view of the domain.
Internally, my firewall provides the internal view. This lets me use the same hostnames inside and outside my network while returning private addresses internally.
For example, public DNS might resolve:
service.example.com -> public address
while the internal DNS view resolves:
service.example.com -> 10.x.x.x
That’s intentional.
It means internal clients can reach services directly over the appropriate internal networks while I can continue using the same FQDNs and certificates everywhere. It also forms part of the security model for some services.
This blog is one example. The WordPress admin interface isn’t exposed through the same HAProxy frontend to both internal and external clients. On the main frontend handling WAN traffic, requests to /wp-admin are rejected. When I access the same hostname internally, split-horizon DNS directs the connection to a separate HAProxy frontend where administrative traffic is permitted.
Conceptually, the same FQDN can therefore take two different paths:
External client
|
v
Public DNS
|
v
WAN HAProxy frontend
|
+-- /wp-admin --> rejected
|
+-- public paths --> backend
Internal client
|
v
Internal DNS
|
v
Internal HAProxy frontend
|
+-- /wp-admin --> permitted
|
+-- other paths --> backendThat makes split-horizon DNS more than a convenience in this environment. It lets the same hostname and certificate work on both sides, while HAProxy applies different access policies based on which frontend the client can reach.
Kubernetes participates in that same DNS environment:
Pod
|
v
CoreDNS
|
v
pfSense / internal DNS
|
v
internal view of example.com
The problem was that my internal DNS server was authoritative for its view of the public domain.
And that view didn’t contain the temporary ACME TXT record that cert-manager had just created in Cloudflare.
So there were effectively two versions of the same record.
Public DNS:
_acme-challenge.example.com
|
v
Cloudflare
|
v
TXT "<challenge>"
Internal DNS:
_acme-challenge.example.com
|
v
pfSense / internal DNS
|
v
no TXT record
When I queried Cloudflare through a public resolver, everything looked correct.
When cert-manager performed the same lookup through the Kubernetes resolver, it got the internal view instead.
The TXT record existed.
It just didn’t exist in cert-manager’s view of DNS.
Testing DNS from inside Kubernetes
This is an important distinction when troubleshooting DNS-01.
Running dig from your laptop doesn’t necessarily test the same DNS path your application is using.
A temporary pod makes that easy to verify:
kubectl run dns-test \
--rm -it \
--restart=Never \
--image=busybox:1.36 \
-- nslookup -type=TXT _acme-challenge.example.com
Then compare that result with the public DNS view:
dig TXT _acme-challenge.example.com @1.1.1.1
In a split-horizon environment, those queries can return different answers despite asking for exactly the same FQDN.
You can also inspect cert-manager’s view of the challenge directly:
kubectl get challenges -A
and:
kubectl describe challenge -A
The distinction I cared about was whether cert-manager was failing to create the DNS record or failing to observe it afterward.
In this case, creation was working.
The self-check was failing.
Giving cert-manager a different DNS path
I didn’t want to change the cluster’s normal DNS behavior.
The split-horizon configuration exists for a reason. Kubernetes applications should continue using internal DNS and resolving internal versions of my public hostnames.
ACME validation is the exception.
cert-manager supports overriding the recursive nameservers used for DNS-01 self-checks. I configured it to use public resolvers instead:
--dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53
--dns01-recursive-nameservers-only
The resulting lookup path is now:
cert-manager
|
v
1.1.1.1 / 8.8.8.8
|
v
public DNS
|
v
Cloudflare authoritative DNS
|
v
_acme-challenge TXT
instead of:
cert-manager
|
v
CoreDNS
|
v
pfSense / internal DNS
|
v
internal copy of the zone
The first option tells cert-manager which recursive nameservers to use. The second forces the DNS-01 self-check through those recursive nameservers rather than relying on the cluster’s normal resolver path. These were the two settings that resolved the original issue.
Configuring the resolver with Helm
Since cert-manager is deployed through Helm, the configuration can be expressed as additional controller arguments:
extraArgs:
- --dns01-recursive-nameservers=1.1.1.1:53,8.8.8.8:53
- --dns01-recursive-nameservers-only
After updating the release, the cert-manager controller restarts with the new arguments.
The important part isn’t specifically Cloudflare’s 1.1.1.1 and Google’s 8.8.8.8. They’re simply public recursive resolvers that will follow the public DNS view of the domain.
The requirement is that the resolver used for the ACME self-check can see the TXT record created in the public zone.
Why I didn’t change CoreDNS
I could have approached this as a Kubernetes DNS problem and modified CoreDNS.
That would have been solving the wrong problem.
My normal DNS path is doing exactly what I configured it to do:
Normal Kubernetes DNS
Pod
|
v
CoreDNS
|
v
pfSense / internal DNS
|
+--> Internal hosts
|
+--> Internal view of public domain
Applications inside the cluster should use that path.
Only the ACME self-check needs this:
cert-manager DNS-01 self-check
|
v
Public recursive resolver
|
v
Public authoritative DNS
|
v
Cloudflare TXT record
Overriding cert-manager’s resolver keeps that exception local to cert-manager instead of changing DNS behavior for every pod in the cluster.
HAProxy and Traefik don’t change the DNS-01 validation path
Having two proxy layers can make the TLS architecture look more complicated than it actually is, but neither proxy is involved in DNS-01 validation.
For a public Kubernetes application, an HTTP request might follow:
Internet
|
v
Cloudflare
|
v
pfSense / HAProxy
|
v
Traefik
|
v
Kubernetes Service
|
v
Pod
A non-Kubernetes application might instead follow:
Internet
|
v
Cloudflare
|
v
pfSense / HAProxy
|
v
VM / physical service / other host
The ACME DNS-01 validation path is separate:
cert-manager -> Cloudflare DNS API -> TXT record
^
|
ACME validation
There is no HTTP challenge passing through HAProxy or Traefik.
That’s one of the reasons I prefer DNS-01 here. Certificate issuance isn’t dependent on how a particular application is exposed—or whether it’s externally exposed at all.
The initial certificate wasn’t the only thing that needed fixing
It’s possible to work around this problem manually.
I could have added the challenge TXT record to the internal DNS view. I could have temporarily changed DNS behavior. Either might have allowed the initial certificate to issue.
But cert-manager isn’t only responsible for initial issuance.
Eventually it has to renew the certificate.
Renewal uses the same DNS-01 process:
Create challenge
|
v
Publish TXT record
|
v
Self-check
|
v
ACME validation
|
v
Renew certificate
A workaround that only makes the first issuance succeed leaves a certificate that’s likely to fail when cert-manager tries to renew it later.
Changing the DNS-01 self-check resolver fixes the path permanently. Future renewals use the public DNS view without requiring changes to the rest of the network.
The useful troubleshooting test
The shortest version of this problem is:
Is the DNS-01 challenge pending?
|
v
Does the TXT record exist in the DNS provider?
|
yes
|
v
Does a public resolver return it?
|
yes
|
v
Does a pod using normal cluster DNS return it?
|
no
|
v
Check for split-horizon DNS
If the TXT record is correct in Cloudflare and publicly resolvable, waiting longer for “DNS propagation” isn’t necessarily going to accomplish anything.
The resolver cert-manager is using may simply never receive that version of the zone.
In my case, all of the individual components were behaving correctly:
Cloudflare created and served the public TXT record.
pfSense served the internal version of the domain.
CoreDNS forwarded Kubernetes DNS queries through the expected internal path.
cert-manager performed its self-check before requesting ACME validation.
The failure came from combining those behaviors.
Forcing cert-manager’s DNS-01 self-check through public recursive resolvers lets the rest of the split-horizon design remain unchanged while ensuring that ACME validation sees the record cert-manager actually created.
For anyone using internal DNS for the same domain they host publicly, it’s a small configuration detail that’s very easy to miss—and one worth fixing before the first certificate reaches its renewal window.
