Generating and using Let’s Encrypt wildcard certificates with cert‑manager

Let’s say you manage a Kubernetes cluster that hosts various environments of a web app. example.com hosts your production environment, and you also have test and acceptance environments at test.example.com and acc.example.com. Of course, you have configured a free Let’s Encrypt certificate for each environment so that each instance is accessible over HTTPS.

This works well as long as you don’t hit Let’s Encrypt’s rate limits. For example, we can find the following limit in Let’s Encrypt’s documentation:

Up to 50 certificates can be issued per registered domain (or IPv4 address, or IPv6 /64 range) every 7 days.

Now, you may think that 50 is quite a generous number – and it is, especially for a service that is provided for free – but depending on your use case, it’s a number you can reach very easily. For example, if you want to give your team the ability to dynamically create test environments, you will be able to create at most 50 environments per week (side note: Assuming, of course, that you will not need to request any other certificates for your domain.). But what if each environments requires the use of four subdomains (e.g., demo.example.com, demo.api.example.com, demo.admin.example.com, and demo.assets.example.com)? Now you are limited to creating only $\lfloor \frac{50}{4} \rfloor = 12$ environments per week, which is unlikely to be enough for an entire development team.

There are several ways to get around this limit:

• Let’s Encrypt provides a staging environment with much higher rate limits. The certificates it generates aren’t trusted by web browsers and other web clients, but this can nonetheless be a very viable alternative if you only need to test the mechanism of requesting certificates (side note: I once forgot to set Let’s Encrypt’s staging environment for my deployment test scripts and blew through the weekly rate limit in less than an hour. Fortunately it wasn’t connected to our real domain, because we didn’t get to deploy anything else that required a valid certificate the rest of the week…).

• You can submit a request to Let’s Encrypt to increase your rate limits. I haven’t tried this myself, but the process can reportedly take several weeks and there is no guarantee that Let’s Encrypt will accept your request, given that it is likely intended for hosting providers who request certificates for a large variety of domains on behalf of their customers.

• If your DNS provider supports it, you can request a single wildcard certificate that covers your entire domain. All you have to do is keep the certificate synchronised between every environment that uses it.

In this guide I’ll explain how you can request a wildcard certificate for your Kubernetes cluster with cert-manager, which you’re probably already using for your existing certificates.

I will assume that you are already familiar with deploying web applications on Kubernetes that make use of “normal” single-domain certificates using cert-manager, that you control the domain’s DNS records, and that you manage the cluster and all applications deployed on it (side note: If this is the case for you, you can still follow this guide. Just use Issuer whenever the guide mentions a ClusterIssuer.).

Certificate authorities such as Let’s Encrypt will only issue certificates for domains that you control. When you request a certificate for example.com, Let’s Encrypt wants proof that you control the domain via a so-called challenge.

With HTTP-01 challenges, Let’s Encrypt provides you with a token that you put at http://<DOMAIN>/.well-known/acme-challenge/<TOKEN>. This allows Let’s Encrypt to verify that you own <DOMAIN>, which could be something like demo.example.com. What it does not prove, is that you are also the rightful owner of all other subdomains of example.com, such as foo.example.com, bar.example.com or baz.example.com.

DNS-01 challenges work in a similar way as HTTP-01 challenges, but via a TXT record for _acme-challenge.<DOMAIN> containing the token. This proves that you own the entire domain and should be allowed to request wildcard certificates for it.

First, make sure you have an issuer that supports DNS-01 challenges. If you don’t have one yet, create one using the snippet below. Make sure you modify the email and dns01 solver for your use case.

In this example we use Cloudflare, which is natively supported by cert-manager. However, if you live in Europe you may want to consider using a European alternative that is safe from America’s tantrums and has a community-maintained webhook, such as Hetzner or deSEC.

Certificates that you create using cert-manager are stored as a Kubernetes Secret. An important characteristic of Secrets is that they are scoped to a specific Namespace. An application that is deployed in an example-demo namespace therefore won’t be able to access a wildcard certificate generated in the default namespace.

You could manually copy and update the secret from default to any namespace that needs it, but this approach doesn’t scale very well and is likely prone to errors.

Reflector is a Kubernetes add-on that automates the synchronisation of resources such as Secrets and ConfigMaps. It can be easily installed via Helm:

Next, create a wildcard certificate for your domain. The snippet below contains the YAML for a Certificate resource that you can use as a starting point. I have highlighted all fields that you should adjust for your own use case:

• The name and secretName can technically be whatever you want, but should ideally describe the domain(s) covered by the wildcard certificate.

• The issuerRef must point to a ClusterIssuer (or Issuer) that supports DNS-01 challenges. Here, we reference our newly created letsencrypt-prod-dns ClusterIssuer.

• dnsNames must be a list of all domains that are covered by the wildcard certificate. In this example, we list the domains needed for the production environment (example.com and *.example.com) and the test environments (*.example.com, *.admin.example.com, *.api.example.com, and *.assets.example.com).

• The annotations tell Reflector which namespaces are allowed to use the wildcard certificate. Here, I allow any namespace that starts with example- to use the certificate, but you can also limit this to specific namespaces by hardcoding them as a comma-separated string.

Note that DNS changes may take time to propagate. It usually takes several minutes for the certificate to be issued.

Once the wildcard certificate is ready, you can start using it in your ingresses.

Let’s say that you want to create a new demolition.example.com that you host in a new example-demolition namespace.

Create an empty Secret and add an annotation that tells Reflector to synchronise its contents with the Secret containing the wildcard certificate in default:

Finally, make sure to set the secretName to the name of the secret now containing the wildcard certificate:

That’s it, you’re done!