Drinking the Kubernetes Kool-Aid
Generating and using Let’s Encrypt wildcard certificates with cert‑manager
Wildcard certificates are amazingly easy to set up and manage with cert-manager. I don’t know how I managed without them for so long.
Let’s say you manage a Kubernetes cluster that hosts various environments of a web app. example.com hosts your production environment, and you also have test and acceptance environments at test.example.com and acc.example.com. Of course, you have configured a free Let’s Encrypt certificate for each environment so that each instance is accessible over HTTPS.
This works well as long as you don’t hit Let’s Encrypt’s rate limits. For example, we can find the following limit in Let’s Encrypt’s documentation:
Up to 50 certificates can be issued per registered domain (or IPv4 address, or IPv6 /64 range) every 7 days.
Now, you may think that 50 is quite a generous number – and it is, especially for a service that is provided for free – but depending on your use case, it’s a number you can reach very easily. For example, if you want to give your team the ability to dynamically create test environments, you will be able to create at most 50 environments per week. But what if each environments requires the use of four subdomains (e.g., demo.example.com, demo.api.example.com, demo.admin.example.com, and demo.assets.example.com)? Now you are limited to creating only environments per week, which is unlikely to be enough for an entire development team.
There are several ways to get around this limit:
-
Let’s Encrypt provides a staging environment with much higher rate limits. The certificates it generates aren’t trusted by web browsers and other web clients, but this can nonetheless be a very viable alternative if you only need to test the mechanism of requesting certificates.
-
You can submit a request to Let’s Encrypt to increase your rate limits. I haven’t tried this myself, but the process can reportedly take several weeks and there is no guarantee that Let’s Encrypt will accept your request, given that it is likely intended for hosting providers who request certificates for a large variety of domains on behalf of their customers.
-
If your DNS provider supports it, you can request a single wildcard certificate that covers your entire domain. All you have to do is keep the certificate synchronised between every environment that uses it.
In this guide I’ll explain how you can request a wildcard certificate for your Kubernetes cluster with cert-manager, which you’re probably already using for your existing certificates.
I will assume that you are already familiar with deploying web applications on Kubernetes that make use of “normal” single-domain certificates using cert-manager, that you control the domain’s DNS records, and that you manage the cluster and all applications deployed on it.
DNS-01 challenges
Certificate authorities such as Let’s Encrypt will only issue certificates for domains that you control. When you request a certificate for example.com, Let’s Encrypt wants proof that you control the domain via a so-called challenge.
With HTTP-01 challenges, Let’s Encrypt provides you with a token that you put at http://<DOMAIN>/.well-known/acme-challenge/<TOKEN>. This allows Let’s Encrypt to verify that you own <DOMAIN>, which could be something like demo.example.com. What it does not prove, is that you are also the rightful owner of all other subdomains of example.com, such as foo.example.com, bar.example.com or baz.example.com.
DNS-01 challenges work in a similar way as HTTP-01 challenges, but via a TXT record for _acme-challenge.<DOMAIN> containing the token. This proves that you own the entire domain and should be allowed to request wildcard certificates for it.
Configuring a DNS-01 challenge provider
First, make sure you have an issuer that supports DNS-01 challenges. If you don’t have one yet, create one using the snippet below. Make sure you modify the email and dns01 solver for your use case.
In this example we use Cloudflare, which is natively supported by cert-manager. However, if you live in Europe you may want to consider using a European alternative that is safe from America’s tantrums and has a community-maintained webhook, such as Hetzner or deSEC.
apiVersion: cert-manager.io/v1kind: ClusterIssuermetadata: name: letsencrypt-prod-dnsspec: acme: server: https://acme-v02.api.letsencrypt.org/directory email: bofh@example.com privateKeySecretRef: name: letsencrypt-prod-dns solvers: - dns01: cloudflare: # This references a Secret that contains your Cloudflare API token apiTokenSecretRef: name: cloudflare-api-token key: api-tokenSet up secret synchronisation
Certificates that you create using cert-manager are stored as a Kubernetes Secret. An important characteristic of Secrets is that they are scoped to a specific Namespace. An application that is deployed in an example-demo namespace therefore won’t be able to access a wildcard certificate generated in the default namespace.
You could manually copy and update the secret from default to any namespace that needs it, but this approach doesn’t scale very well and is likely prone to errors.
Reflector is a Kubernetes add-on that automates the synchronisation of resources such as Secrets and ConfigMaps. It can be easily installed via Helm:
helm upgrade --install reflector oci://ghcr.io/emberstack/helm-charts/reflectorCreating a wildcard certificate
Next, create a wildcard certificate for your domain. The snippet below contains the YAML for a Certificate resource that you can use as a starting point. I have highlighted all fields that you should adjust for your own use case:
-
The
nameandsecretNamecan technically be whatever you want, but should ideally describe the domain(s) covered by the wildcard certificate. -
The
issuerRefmust point to a ClusterIssuer (or Issuer) that supports DNS-01 challenges. Here, we reference our newly createdletsencrypt-prod-dnsClusterIssuer. -
dnsNamesmust be a list of all domains that are covered by the wildcard certificate. In this example, we list the domains needed for the production environment (example.comand*.example.com) and the test environments (*.example.com,*.admin.example.com,*.api.example.com, and*.assets.example.com). -
The
annotationstell Reflector which namespaces are allowed to use the wildcard certificate. Here, I allow any namespace that starts withexample-to use the certificate, but you can also limit this to specific namespaces by hardcoding them as a comma-separated string.
Note that DNS changes may take time to propagate. It usually takes several minutes for the certificate to be issued.
apiVersion: cert-manager.io/v1kind: Certificatemetadata: name: example-com # We use the “default” namespace here, but it can also be something else namespace: defaultspec: # Name of the secret that we will later reference secretName: example-com-tls issuerRef: # Use the new “letsencrypt-prod-dns” cluster issuer name: letsencrypt-prod-dns kind: ClusterIssuer dnsNames: # This is a list of all domains that will be included in the certificate - "example.com" - "*.example.com" - "*.admin.example.com" - "*.api.example.com" - "*.assets.example.com" secretTemplate: annotations: # Instruct Reflector to automatically sync this secret to other namespaces reflector.v1.k8s.emberstack.com/reflection-allowed: "true" reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces: "example-.*" reflector.v1.k8s.emberstack.com/reflection-auto-enabled: "true" reflector.v1.k8s.emberstack.com/reflection-auto-namespaces: "example-.*"Using the wildcard certificate
Once the wildcard certificate is ready, you can start using it in your ingresses.
Let’s say that you want to create a new demolition.example.com that you host in a new example-demolition namespace.
Create an empty Secret and add an annotation that tells Reflector to synchronise its contents with the Secret containing the wildcard certificate in default:
apiVersion: v1kind: Secretmetadata: # This secret name must be referenced in your ingress name: example-com-tls namespace: example-demolition annotations: # Make Reflector keep this secret in sync with the original in “default” reflector.v1.k8s.emberstack.com/reflects: "default/example-com-tls"Finally, make sure to set the secretName to the name of the secret now containing the wildcard certificate:
apiVersion: networking.k8s.io/v1kind: Ingressmetadata: name: example-app namespace: example-demolition annotations: {} # We already have a certificate, so no Cert-Manager annotation is neededspec: rules: - host: demolition.example.com http: paths: - path: / pathType: Prefix backend: service: name: example-app port: name: http tls: - hosts: - demolition.example.com # Reference the secret we created earlier secretName: example-com-tlsThat’s it, you’re done!