Configuring mutual TLS with Traefik as an ingress controller
Let’s assume the following context: there is a client application that connects to a server application deployed inside a Kubernetes cluster. The client and the server have special requirements that necessitate configuring mutual TLS. The cluster uses Traefik as an ingress controller to manage incoming traffic. How can mTLS be configured? I found myself in this situation a while ago. There are a few tricky parts about this setup, so I wanted to write a post about it.
I’ll start by launching a local Kubernetes cluster using Minikube. I’m using a profile called mtls-demo since I manage multiple local clusters with Minikube:
1# Launch a local Kubernetes cluster using the mtls-demo profile.
2-> % minikube start -p mtls-demo
3...
4🏄 Done! kubectl is now configured to use "mtls-demo" cluster and "default" namespace by default
5
6# Switch to the mtls-demo profile. Useful when opening a new terminal.
7-> % minikube profile mtls-demo
8✅ minikube profile was successfully set to mtls-demo
9
10# Check the status of the cluster.
11-> % minikube status
For the target service, I’ll use whoami. It’s a small Go server that prints HTTP requests to the output. This will help in inspecting the request body and headers.
Let’s create a namespace for whoami:
1-> % cat whoami-namespace.yaml
2apiVersion: v1
3kind: Namespace
4metadata:
5 name: whoami
6-> % k apply -f whoami-namespace.yaml
Then a pod in the whoami namespace running the whoami container that exposes port 80 for incoming requests:
1-> % cat whoami-pod.yaml
2apiVersion: v1
3kind: Pod
4metadata:
5 name: whoami
6 namespace: whoami
7 labels:
8 app: whoami
9spec:
10 containers:
11 - name: whoami
12 image: traefik/whoami
13 command:
14 - "/whoami"
15 - "--verbose"
16 ports:
17 - name: web
18 containerPort: 80
19 protocol: TCP
20-> % k apply -f whoami-pod.yaml
21pod/whoami created
22-> % k -n whoami get po
23NAME READY STATUS RESTARTS AGE
24whoami 1/1 Running 0 76s
And finally a service:
1-> % cat whoami-service.yaml
2apiVersion: v1
3kind: Service
4metadata:
5 name: whoami
6 namespace: whoami
7spec:
8 selector:
9 app: whoami
10 type: ClusterIP
11 ports:
12 - name: whoamiweb
13 protocol: TCP
14 port: 8080
15 targetPort: web
16-> % k apply -f whoami-service.yaml
17service/whoami created
18-> % k -n whoami get svc
19NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
20whoami ClusterIP 10.108.194.137 <none> 8080/TCP 48s
Let’s install the Traefik ingress controller using Helm. I’ll set some options through the values.yaml file:
1-> % cat traefik-values.yaml
2logs:
3 general:
4 level: "TRACE"
5 access:
6 enabled: true
7 fields:
8 headers:
9 defaultMode: "keep"
10ingressRoute:
11 dashboard:
12 enabled: true
13 entryPoints: [web, websecure]
14-> % helm install traefik oci://ghcr.io/traefik/helm/traefik --namespace traefik --create-namespace -f traefik-values.yaml
15Pulled: ghcr.io/traefik/helm/traefik:36.3.0
16Digest: sha256:4c9930fb5db2eb0c31d445933e2a9c86796073a2c35e72c2588cecb217a085d2
17NAME: traefik
18LAST DEPLOYED: Wed Jul 2 12:07:11 2025
19NAMESPACE: traefik
20STATUS: deployed
21REVISION: 1
22TEST SUITE: None
23NOTES:
24traefik with docker.io/traefik:v3.4.3 has been deployed successfully on traefik namespace !
I’ve enabled logging of requests and headers to gain more insight into what the ingress controller is processing. I’ve also exposed the Traefik dashboard to verify that it’s working. To access it:
1-> % minikube tunnel
The minikube tunnel command requires sudo permission because privileged ports are exposed. After opening the tunnel, access the dashboard at localhost/dashboard/ (note the required trailing slash).
Next, I'll set up an ingress rule so Traefik knows how to forward traffic to the whoami service:
1-> % cat whoami-ingress.yaml
2apiVersion: networking.k8s.io/v1
3kind: Ingress
4metadata:
5 name: whoami
6 namespace: whoami
7spec:
8 ingressClassName: traefik
9 rules:
10 - host: localhost
11 http:
12 paths:
13 - path: /
14 pathType: Prefix
15 backend:
16 service:
17 name: whoami
18 port:
19 name: whoamiweb
20-> % k apply -f whoami-ingress.yaml
21ingress.networking.k8s.io/whoami created
22-> % k -n whoami get ingress
23NAME CLASS HOSTS ADDRESS PORTS AGE
24whoami traefik localhost 127.0.0.1 80 14s
Traefik’s default behavior is to look in all namespaces and process all ingress resources with the traefik ingress class. This rule forwards requests having localhost as a host and '/' as a path prefix to the whoami service on port 8080.
Let’s test it:
1-> % curl http://localhost
2Hostname: whoami
3IP: 127.0.0.1
4IP: ::1
5IP: 10.244.0.3
6IP: fe80::dc26:15ff:fee1:cbe7
7RemoteAddr: 10.244.0.4:47296
8GET / HTTP/1.1
9Host: localhost
10User-Agent: curl/8.7.1
11Accept: */*
12Accept-Encoding: gzip
13X-Forwarded-For: 10.244.0.1
14X-Forwarded-Host: localhost
15X-Forwarded-Port: 80
16X-Forwarded-Proto: http
17X-Forwarded-Server: traefik-6489fdb447-jgqgx
18X-Real-Ip: 10.244.0.1
So far so good. Let’s move on to setting up TLS. For a full mutual TLS setup, I’ll need a CA certificate, a server certificate, and a client certificate. mTLS works as follows (with some details omitted):
- The client sends a request to the server to ask for its certificate.
- The server sends the client back its certificate.
- The client verifies the server’s certificate against its trust store, i.e. it goes through all the certificate signers it trusts and if the certificate of the server was signed by one of them then the client can trust the server.
Usually this is where TLS usually ends, but in mutual TLS, the server also verifies the client:
- The client sends its certificate.
- The server verifies it against its own trust store.
If the certificate is signed by a trusted CA, the TLS handshake is successful.
The CA (Certificate Authority) certificate is the certificate that belongs to a trusted well-known entity (e.g. Google or Amazon) and it’s used to sign the other certificates. A collection of CA certificates forms a trust store.
Let’s generate a CA certificate, a server certificate and a client certificate. I’ll use the localhost domain for all of them and for the Organisation Unit I’ll set the identifier of the certificate, i.e. ca, server, client to be able to distinguish between them.
For this, the openssl tool can be used. I have a few utility scripts for generating the certificates which are pretty intuitive to use:
1## Generate the CA certificate.
2
3# Fetch the script for generating self-signed certificates:
4wget https://gist.githubusercontent.com/vlasebian/1a5ff3925fe5e21c18484d5929e753ff/raw/5588e2b99151f26d2edb0b47555daaf51098cf9a/gen_self_signed_cert.sh
5
6# Check the script before, don't run random stuff from the internet!
7chmod +x gen_self_signed_cert.sh
8./gen_self_signed_cert.sh -o ca
9
10## Generate the server and client certificates.
11
12# Fetch the script for generating certificates:
13wget https://gist.githubusercontent.com/vlasebian/274c7ca97ee392117e780352bea0daa6/raw/56d4a23fd96b77a19ac9fb1e57c798b350b46930/gen_cert.sh
14
15# Check the script before, don't run random stuff from the internet!
16chmod +x gen_cert.sh
17./gen_cert.sh --cacrt root.crt --cakey root.key -o server
18./gen_cert.sh --cacrt root.crt --cakey root.key -o client
19
20## Inspect certificates:
21
22# Fetch the script for inspecting certificates:
23wget https://gist.githubusercontent.com/vlasebian/918023f03b3866b4b0c691f8622d8e0b/raw/dd0bdb44935f8156dcaab50f65c3f7bca8f6dedb/inspect_cert.sh
24
25# Check the script before, don't run random stuff from the internet!
26chmod +x inspect_cert.sh
27./inspect_cert.sh server.crt
The set of certificates looks like this:
1-> % ls *.key *.crt
2server.crt server.key client.crt client.key ca.crt ca.key
Now that we have the certificates, let’s add the tls certificate to the server. I’ll configure tls on the whoami ingress resource so only traffic that matches the ingress rule will be decrypted by the provided certificate. To do that, I have to make the server certificate available in the cluster. This can be done by using a secret. The secret will be used by the whoami ingress resource so it must be created in the whoami namespace:
1-> % kubectl create secret tls server-tls-certificate --cert=server.crt --key=server.key -n whoami
2secret/server-tls-certificate created
3-> % k -n whoami get secret
4NAME TYPE DATA AGE
5server-tls-certificate kubernetes.io/tls 2 9s
I’ll update the whoami ingress resource that we applied previously to pick up the TLS secret and reapply it:
1-> % cat whoami-ingress.yaml
2apiVersion: networking.k8s.io/v1
3kind: Ingress
4metadata:
5 name: whoami
6 namespace: whoami
7spec:
8 ingressClassName: traefik
9 rules:
10 - host: localhost
11 http:
12 paths:
13 - path: /
14 pathType: Prefix
15 backend:
16 service:
17 name: whoami
18 port:
19 name: whoamiweb
20 tls:
21 - hosts:
22 - localhost
23 secretName: server-tls-certificate
24-> % k apply -f whoami-ingress.yaml
25ingress.networking.k8s.io/whoami configured
26-> % k -n whoami describe ingress/whoami
27Name: whoami
28Labels: <none>
29Namespace: whoami
30Address: 127.0.0.1
31Ingress Class: traefik
32Default backend: <default>
33TLS:
34 server-tls-certificate terminates localhost
35Rules:
36 Host Path Backends
37 ---- ---- --------
38 localhost
39 / whoami:whoamiweb (10.244.0.3:80)
40Annotations: <none>
41Events: <none>
Let’s send a request to the endpoint using curl:
1-> % curl https://localhost
2curl: (60) SSL certificate problem: unable to get local issuer certificate
3More details here: https://curl.se/docs/sslcerts.html
4
5curl failed to verify the legitimacy of the server and therefore could not
6establish a secure connection to it. To learn more about this situation and
7how to fix it, please visit the web page mentioned above.
We have an error because we didn’t provide any CA certificate (or CA path to a directory of certificates) to verify the server certificate against. Let’s pass the ca.crt that we used to sign the server certificate with:
1-> % curl --cacert ca.crt https://localhost
2Hostname: whoami
3IP: 127.0.0.1
4IP: ::1
5IP: 10.244.0.3
6IP: fe80::dc26:15ff:fee1:cbe7
7RemoteAddr: 10.244.0.4:35884
8GET / HTTP/1.1
9Host: localhost
10User-Agent: curl/8.7.1
11Accept: */*
12Accept-Encoding: gzip
13X-Forwarded-For: 10.244.0.1
14X-Forwarded-Host: localhost
15X-Forwarded-Port: 443
16X-Forwarded-Proto: https
17X-Forwarded-Server: traefik-6489fdb447-jgqgx
18X-Real-Ip: 10.244.0.1
The request was successful now. You can also bypass the server certificate check by passing the -k option to curl.
Let’s check the actual TLS handshake by providing -v (verbose) flag to curl:
1-> % curl -v --cacert ca.crt https://localhost
2* Host localhost:443 was resolved.
3* IPv6: ::1
4* IPv4: 127.0.0.1
5* Trying [::1]:443...
6* Connected to localhost (::1) port 443
7* ALPN: curl offers h2,http/1.1
8* (304) (OUT), TLS handshake, Client hello (1):
9* CAfile: ca.crt
10* CApath: none
11* (304) (IN), TLS handshake, Server hello (2):
12* (304) (IN), TLS handshake, Unknown (8):
13* (304) (IN), TLS handshake, Certificate (11):
14* (304) (IN), TLS handshake, CERT verify (15):
15* (304) (IN), TLS handshake, Finished (20):
16* (304) (OUT), TLS handshake, Finished (20):
17* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
18* ALPN: server accepted h2
19* Server certificate:
20* subject: C=US; ST=California; L=San Francisco; O=My Company; OU=server; CN=localhost; emailAddress=admin@example.com
21* start date: Jul 1 22:33:24 2025 GMT
22* expire date: Jun 29 22:33:24 2035 GMT
23* common name: localhost (matched)
24* issuer: C=US; ST=California; L=San Francisco; O=My Company; OU=CA; CN=localhost; emailAddress=admin@example.com
25* SSL certificate verify ok.
26* using HTTP/2
27* [HTTP/2] [1] OPENED stream for https://localhost/
28...
The server is now configured. But how can we ensure it also authenticates the client? Since Kubernetes Ingress doesn’t support full mTLS natively, we’ll use Traefik features.
I will create another secret to store the CA certificate. The certificate provided by the client will be verified against this certificate.
1# Traefik requires that the key of the CA certificate is either tls.ca or ca.crt.
2kubectl create secret generic ca-tls-certificate --from-file=ca.crt -n traefik
To set the tls options for client authentication, I’ll use the TLSOption Custom Resource Definition (CRD). I’ll create this in the traefik namespace:
1apiVersion: traefik.io/v1alpha1
2kind: TLSOption
3metadata:
4 name: request-client-cert
5 namespace: traefik
6spec:
7 clientAuth:
8 secretNames:
9 - ca-tls-certificate # CA certificate used by the server to verify the client certificates against
10 clientAuthType: RequireAndVerifyClientCert
11-> % k apply -f traefik-custom-tlsoption.yaml
12tlsoption.traefik.io/request-client-cert created
Then I’ll update the whoami ingress resource to pick up the tls option by using annotations:
1-> % cat whoami-ingress.yaml
2apiVersion: networking.k8s.io/v1
3kind: Ingress
4metadata:
5 name: whoami
6 namespace: whoami
7 annotations:
8 # Annotation value format is: <namespace>-<tls-option-name>@kubernetescrd
9 traefik.ingress.kubernetes.io/router.tls.options: traefik-request-client-cert@kubernetescrd
10spec:
11 ingressClassName: traefik
12 rules:
13 - host: localhost
14 http:
15 paths:
16 - path: /
17 pathType: Prefix
18 backend:
19 service:
20 name: whoami
21 port:
22 name: whoamiweb
23 tls:
24 - hosts:
25 - localhost
26 secretName: server-tls-certificate
27-> % k apply -f whoami-ingress.yaml
28ingress.networking.k8s.io/whoami configured
Let’s send the previous curl request:
1-> % curl --cacert ca.crt https://localhost
2curl: (56) LibreSSL SSL_read: LibreSSL/3.3.6: error:1404C45C:SSL routines:ST_OK:reason(1116), errno 0
I have an error caused by not providing the client certificate requested by the server. Let’s fix that:
1-> % curl --cacert ca.crt --cert client.crt --key client.key https://localhost
2
3Hostname: whoami
4IP: 127.0.0.1
5IP: ::1
6IP: 10.244.0.3
7IP: fe80::dc26:15ff:fee1:cbe7
8RemoteAddr: 10.244.0.4:41722
9GET / HTTP/1.1
10Host: localhost
11User-Agent: curl/8.7.1
12Accept: */*
13Accept-Encoding: gzip
14X-Forwarded-For: 10.244.0.1
15X-Forwarded-Host: localhost
16X-Forwarded-Port: 443
17X-Forwarded-Proto: https
18X-Forwarded-Server: traefik-6489fdb447-jgqgx
19X-Real-Ip: 10.244.0.1
Now the request is processed. Let’s send it again with a -v flag to check the TLS handshake process:
1-> % curl -v --cacert ca.crt --cert client.crt --key client.key https://localhost
2
3* Host localhost:443 was resolved.
4* IPv6: ::1
5* IPv4: 127.0.0.1
6* Trying [::1]:443...
7* Connected to localhost (::1) port 443
8* ALPN: curl offers h2,http/1.1
9* (304) (OUT), TLS handshake, Client hello (1):
10* CAfile: ca.crt
11* CApath: none
12* (304) (IN), TLS handshake, Server hello (2):
13* (304) (IN), TLS handshake, Unknown (8):
14* (304) (IN), TLS handshake, Request CERT (13):
15* (304) (IN), TLS handshake, Certificate (11):
16* (304) (IN), TLS handshake, CERT verify (15):
17* (304) (IN), TLS handshake, Finished (20):
18* (304) (OUT), TLS handshake, Certificate (11):
19* (304) (OUT), TLS handshake, CERT verify (15):
20* (304) (OUT), TLS handshake, Finished (20):
21* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
22* ALPN: server accepted h2
23* Server certificate:
24* subject: C=US; ST=California; L=San Francisco; O=My Company; OU=server; CN=localhost; emailAddress=admin@example.com
25* start date: Jul 1 22:33:24 2025 GMT
26* expire date: Jun 29 22:33:24 2035 GMT
27* common name: localhost (matched)
28* issuer: C=US; ST=California; L=San Francisco; O=My Company; OU=CA; CN=localhost; emailAddress=admin@example.com
29* SSL certificate verify ok.
30* using HTTP/2
31* [HTTP/2] [1] OPENED stream for https://localhost/
32...
If I analyze the output of curl, there are two CERT verify operations, one IN and one OUT. If I compare it with the previous curl request where we passed the -v option, before configuring the TLSOptions in Traefik, there was only one CERT verify operation, for IN. This time the client certificate was also verified.
There are multiple behaviours that can be configured through the clientAuthType field. All the options for the clientAuthType are described here.
One more interesting behaviour would be to make the client certificate required, but let the application process it instead of Traefik. That means we have to forward the headers containing the client certificate to the application. By setting clientAuthType to RequireAnyClientCert Traefik will request the certificate but will not verify it:
1-> % cat traefik-custom-tlsoption.yaml
2apiVersion: traefik.io/v1alpha1
3kind: TLSOption
4metadata:
5 name: request-client-cert
6 namespace: traefik
7spec:
8 clientAuth:
9 clientAuthType: RequireAnyClientCert
10-> % k apply -f traefik-custom-tlsoption.yaml
11tlsoption.traefik.io/request-client-cert configured
Notice that I dropped the secret containing the CA. That won’t be needed anymore as Traefik won’t verify the client certificate anymore, this responsability will be passed to the application.
To forward the TLS certificate in the application, I have to set a middleware that tells Traefik to pass the client certificate to the application:
1-> % cat traefik-pass-tls-middleware.yaml
2apiVersion: traefik.io/v1alpha1
3kind: Middleware
4metadata:
5 name: pass-client-cert
6 namespace: traefik
7spec:
8 passTLSClientCert:
9 pem: true
10-> % k apply -f traefik-pass-tls-middleware.yaml
11middleware.traefik.io/pass-client-cert created
And I have to provide the middleware on the specific ingress through an ingress annotation:
1-> % cat whoami-ingress.yaml
2apiVersion: networking.k8s.io/v1
3kind: Ingress
4metadata:
5 name: whoami
6 namespace: whoami
7 annotations:
8 traefik.ingress.kubernetes.io/router.tls.options: traefik-request-client-cert@kubernetescrd
9 traefik.ingress.kubernetes.io/router.middlewares: traefik-pass-client-cert@kubernetescrd
10spec:
11 ingressClassName: traefik
12 rules:
13 - host: localhost
14 http:
15 paths:
16 - path: /
17 pathType: Prefix
18 backend:
19 service:
20 name: whoami
21 port:
22 name: whoamiweb
23 tls:
24 - hosts:
25 - localhost
26 secretName: server-tls-certificate
27-> % k apply -f whoami-ingress.yaml
28ingress.networking.k8s.io/whoami configured
Now the client certificate is passed to the application through the request headers:
1-> % curl --cacert ca.crt --cert client.crt --key client.key https://localhost
2
3Hostname: whoami
4IP: 127.0.0.1
5IP: ::1
6IP: 10.244.0.3
7IP: fe80::dc26:15ff:fee1:cbe7
8RemoteAddr: 10.244.0.4:37888
9GET / HTTP/1.1
10Host: localhost
11User-Agent: curl/8.7.1
12Accept: */*
13Accept-Encoding: gzip
14X-Forwarded-For: 10.244.0.1
15X-Forwarded-Host: localhost
16X-Forwarded-Port: 443
17X-Forwarded-Proto: https
18X-Forwarded-Server: traefik-6489fdb447-jgqgx
19X-Forwarded-Tls-Client-Cert: MIICbjCCAhSgAwIBAgIURmC/W6MegyfmdP/9P2VGeP/T1qQwCgYIKoZIzj0EAwIwgZIxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNpc2NvMRMwEQYDVQQKDApNeSBDb21wYW55MQswCQYDVQQLDAJDQTESMBAGA1UEAwwJbG9jYWxob3N0MSAwHgYJKoZIhvcNAQkBFhFhZG1pbkBleGFtcGxlLmNvbTAeFw0yNTA3MDEyMjI2NDBaFw0zNTA2MjkyMjI2NDBaMIGWMQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzETMBEGA1UECgwKTXkgQ29tcGFueTEPMA0GA1UECwwGY2xpZW50MRIwEAYDVQQDDAlsb2NhbGhvc3QxIDAeBgkqhkiG9w0BCQEWEWFkbWluQGV4YW1wbGUuY29tMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE4e9Ee+hCUmQcy/F/+6rfnBcPkdFh6ilSAluB6LLpgpu/VO4bxU+iVsVYN7G8VKjiFfeqZGQzkqBycYvpwz8dqKNCMEAwHQYDVR0OBBYEFLoXGaucJeOAUR8TlQw3zdOzrBiSMB8GA1UdIwQYMBaAFHzXpd9kLcMiT05oLSfYcxmFyz8vMAoGCCqGSM49BAMCA0gAMEUCIBVC7lBi2+VaDhHi/moq23NQQm0Fj9PpewKlPi2VteEMAiEAjaO1qhzdkQ2O4ECxM+9zqywhiagWKOKuIzG1d/jnIBM=
20X-Real-Ip: 10.244.0.1