In this article

Drinking the Kubernetes Kool-Aid

Generating and using Let’s Encrypt wildcard certificates with cert‑manager

Wildcard certificates are amazingly easy to set up and manage with cert-manager. I don’t know how I managed without them for so long.

This illustration barely made the certifi-cut
Generic clipart of TLS certificate

This illustration barely made the certifi-cut

Let’s say you manage a Kubernetes cluster that hosts various environments of a web app. example.com hosts your production environment, and you also have test and acceptance environments at test.example.com and acc.example.com. Of course, you have configured a free Let’s Encrypt certificate for each environment so that each instance is accessible over HTTPS.

This works well as long as you don’t hit Let’s Encrypt’s rate limits. For example, we can find the following limit in Let’s Encrypt’s documentation:

Up to 50 certificates can be issued per registered domain (or IPv4 address, or IPv6 /64 range) every 7 days.

Now, you may think that 50 is quite a generous number – and it is, especially for a service that is provided for free – but depending on your use case, it’s a number you can reach very easily. For example, if you want to give your team the ability to dynamically create test environments, you will be able to create at most 50 environments per week. But what if each environments requires the use of four subdomains (e.g., demo.example.com, demo.api.example.com, demo.admin.example.com, and demo.assets.example.com)? Now you are limited to creating only 504=12\lfloor \frac{50}{4} \rfloor = 12 environments per week, which is unlikely to be enough for an entire development team.

There are several ways to get around this limit:

  • Let’s Encrypt provides a staging environment with much higher rate limits. The certificates it generates aren’t trusted by web browsers and other web clients, but this can nonetheless be a very viable alternative if you only need to test the mechanism of requesting certificates.

  • You can submit a request to Let’s Encrypt to increase your rate limits. I haven’t tried this myself, but the process can reportedly take several weeks and there is no guarantee that Let’s Encrypt will accept your request, given that it is likely intended for hosting providers who request certificates for a large variety of domains on behalf of their customers.

  • If your DNS provider supports it, you can request a single wildcard certificate that covers your entire domain. All you have to do is keep the certificate synchronised between every environment that uses it.

In this guide I’ll explain how you can request a wildcard certificate for your Kubernetes cluster with cert-manager, which you’re probably already using for your existing certificates.

I will assume that you are already familiar with deploying web applications on Kubernetes that make use of “normal” single-domain certificates using cert-manager, that you control the domain’s DNS records, and that you manage the cluster and all applications deployed on it.

DNS-01 challenges

Certificate authorities such as Let’s Encrypt will only issue certificates for domains that you control. When you request a certificate for example.com, Let’s Encrypt wants proof that you control the domain via a so-called challenge.

With HTTP-01 challenges, Let’s Encrypt provides you with a token that you put at http://<DOMAIN>/.well-known/acme-challenge/<TOKEN>. This allows Let’s Encrypt to verify that you own <DOMAIN>, which could be something like demo.example.com. What it does not prove, is that you are also the rightful owner of all other subdomains of example.com, such as foo.example.com, bar.example.com or baz.example.com.

DNS-01 challenges work in a similar way as HTTP-01 challenges, but via a TXT record for _acme-challenge.<DOMAIN> containing the token. This proves that you own the entire domain and should be allowed to request wildcard certificates for it.

Configuring a DNS-01 challenge provider

First, make sure you have an issuer that supports DNS-01 challenges. If you don’t have one yet, create one using the snippet below. Make sure you modify the email and dns01 solver for your use case.

In this example we use Cloudflare, which is natively supported by cert-manager. However, if you live in Europe you may want to consider using a European alternative that is safe from America’s tantrums and has a community-maintained webhook, such as Hetzner or deSEC.

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt-prod-dns
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: bofh@example.com
privateKeySecretRef:
name: letsencrypt-prod-dns
solvers:
- dns01:
cloudflare:
# This references a Secret that contains your Cloudflare API token
apiTokenSecretRef:
name: cloudflare-api-token
key: api-token

Set up secret synchronisation

Certificates that you create using cert-manager are stored as a Kubernetes Secret. An important characteristic of Secrets is that they are scoped to a specific Namespace. An application that is deployed in an example-demo namespace therefore won’t be able to access a wildcard certificate generated in the default namespace.

You could manually copy and update the secret from default to any namespace that needs it, but this approach doesn’t scale very well and is likely prone to errors.

Reflector is a Kubernetes add-on that automates the synchronisation of resources such as Secrets and ConfigMaps. It can be easily installed via Helm:

Terminal
helm upgrade --install reflector oci://ghcr.io/emberstack/helm-charts/reflector

Creating a wildcard certificate

Next, create a wildcard certificate for your domain. The snippet below contains the YAML for a Certificate resource that you can use as a starting point. I have highlighted all fields that you should adjust for your own use case:

  • The name and secretName can technically be whatever you want, but should ideally describe the domain(s) covered by the wildcard certificate.

  • The issuerRef must point to a ClusterIssuer (or Issuer) that supports DNS-01 challenges. Here, we reference our newly created letsencrypt-prod-dns ClusterIssuer.

  • dnsNames must be a list of all domains that are covered by the wildcard certificate. In this example, we list the domains needed for the production environment (example.com and *.example.com) and the test environments (*.example.com, *.admin.example.com, *.api.example.com, and *.assets.example.com).

  • The annotations tell Reflector which namespaces are allowed to use the wildcard certificate. Here, I allow any namespace that starts with example- to use the certificate, but you can also limit this to specific namespaces by hardcoding them as a comma-separated string.

Note that DNS changes may take time to propagate. It usually takes several minutes for the certificate to be issued.

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: example-com
# We use the “default” namespace here, but it can also be something else
namespace: default
spec:
# Name of the secret that we will later reference
secretName: example-com-tls
issuerRef:
# Use the new “letsencrypt-prod-dns” cluster issuer
name: letsencrypt-prod-dns
kind: ClusterIssuer
dnsNames:
# This is a list of all domains that will be included in the certificate
- "example.com"
- "*.example.com"
- "*.admin.example.com"
- "*.api.example.com"
- "*.assets.example.com"
secretTemplate:
annotations:
# Instruct Reflector to automatically sync this secret to other namespaces
reflector.v1.k8s.emberstack.com/reflection-allowed: "true"
reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "example-.*"
reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true"
reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "example-.*"

Using the wildcard certificate

Once the wildcard certificate is ready, you can start using it in your ingresses.

Let’s say that you want to create a new demolition.example.com that you host in a new example-demolition namespace.

Create an empty Secret and add an annotation that tells Reflector to synchronise its contents with the Secret containing the wildcard certificate in default:

apiVersion: v1
kind: Secret
metadata:
# This secret name must be referenced in your ingress
name: example-com-tls
namespace: example-demolition
annotations:
# Make Reflector keep this secret in sync with the original in “default”
reflector.v1.k8s.emberstack.com/reflects: "default/example-com-tls"

Finally, make sure to set the secretName to the name of the secret now containing the wildcard certificate:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: example-app
namespace: example-demolition
annotations: {}
# We already have a certificate, so no Cert-Manager annotation is needed
spec:
rules:
- host: demolition.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: example-app
port:
name: http
tls:
- hosts:
- demolition.example.com
# Reference the secret we created earlier
secretName: example-com-tls

That’s it, you’re done!

This article appears in