Let’s assume the following context: there is a client application that connects to a server application deployed inside a Kubernetes cluster. The client and the server have special requirements that necessitate configuring mutual TLS. The cluster uses Traefik as an ingress controller to manage incoming traffic. How can mTLS be configured? I found myself in this situation a while ago. There are a few tricky parts about this setup, so I wanted to write a post about it.

I’ll start by launching a local Kubernetes cluster using Minikube. I’m using a profile called mtls-demo since I manage multiple local clusters with Minikube:

# Launch a local Kubernetes cluster using the mtls-demo profile.
-> % minikube start -p mtls-demo
...
🏄  Done! kubectl is now configured to use "mtls-demo" cluster and "default" namespace by default

# Switch to the mtls-demo profile. Useful when opening a new terminal.
-> % minikube profile mtls-demo
✅  minikube profile was successfully set to mtls-demo

# Check the status of the cluster.
-> % minikube status

For the target service, I’ll use whoami. It’s a small Go server that prints HTTP requests to the output. This will help in inspecting the request body and headers.

Let’s create a namespace for whoami:

-> % cat whoami-namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: whoami
-> % k apply -f whoami-namespace.yaml

Then a pod in the whoami namespace running the whoami container that exposes port 80 for incoming requests:

-> % cat whoami-pod.yaml
apiVersion: v1
kind: Pod
metadata:
  name: whoami
  namespace: whoami
  labels:
    app: whoami
spec:
  containers:
  - name: whoami
    image: traefik/whoami
    command:
    - "/whoami"
    - "--verbose"
    ports:
    - name: web
      containerPort: 80
      protocol: TCP
-> % k apply -f whoami-pod.yaml
pod/whoami created
-> % k -n whoami get po
NAME     READY   STATUS    RESTARTS   AGE
whoami   1/1     Running   0          76s

And finally a service:

-> % cat whoami-service.yaml
apiVersion: v1
kind: Service
metadata:
  name: whoami
  namespace: whoami
spec:
  selector:
    app: whoami
  type: ClusterIP
  ports:
  - name: whoamiweb
    protocol: TCP
    port: 8080
    targetPort: web
-> % k apply -f whoami-service.yaml
service/whoami created
-> % k -n whoami get svc
NAME     TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
whoami   ClusterIP   10.108.194.137   <none>        8080/TCP   48s

Let’s install the Traefik ingress controller using Helm. I’ll set some options through the values.yaml file:

-> % cat traefik-values.yaml
logs:
  general:
    level: "TRACE"
  access:
    enabled: true
    fields:
      headers:
        defaultMode: "keep"
ingressRoute:
  dashboard:
    enabled: true
    entryPoints: [web, websecure]
-> % helm install traefik oci://ghcr.io/traefik/helm/traefik --namespace traefik --create-namespace -f traefik-values.yaml
Pulled: ghcr.io/traefik/helm/traefik:36.3.0
Digest: sha256:4c9930fb5db2eb0c31d445933e2a9c86796073a2c35e72c2588cecb217a085d2
NAME: traefik
LAST DEPLOYED: Wed Jul  2 12:07:11 2025
NAMESPACE: traefik
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
traefik with docker.io/traefik:v3.4.3 has been deployed successfully on traefik namespace !

I’ve enabled logging of requests and headers to gain more insight into what the ingress controller is processing. I’ve also exposed the Traefik dashboard to verify that it’s working. To access it:

-> % minikube tunnel

The minikube tunnel command requires sudo permission because privileged ports are exposed. After opening the tunnel, access the dashboard at localhost/dashboard/ (note the required trailing slash).

Next, I’ll set up an ingress rule so Traefik knows how to forward traffic to the whoami service:

-> % cat whoami-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami
  namespace: whoami
spec:
  ingressClassName: traefik
  rules:
  - host: localhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: whoami
            port:
              name: whoamiweb
-> % k apply -f whoami-ingress.yaml
ingress.networking.k8s.io/whoami created
-> % k -n whoami get ingress
NAME     CLASS     HOSTS       ADDRESS     PORTS   AGE
whoami   traefik   localhost   127.0.0.1   80      14s

Traefik’s default behavior is to look in all namespaces and process all ingress resources with the traefik ingress class. This rule forwards requests having localhost as a host and ’/’ as a path prefix to the whoami service on port 8080.

Let’s test it:

-> % curl http://localhost
Hostname: whoami
IP: 127.0.0.1
IP: ::1
IP: 10.244.0.3
IP: fe80::dc26:15ff:fee1:cbe7
RemoteAddr: 10.244.0.4:47296
GET / HTTP/1.1
Host: localhost
User-Agent: curl/8.7.1
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.244.0.1
X-Forwarded-Host: localhost
X-Forwarded-Port: 80
X-Forwarded-Proto: http
X-Forwarded-Server: traefik-6489fdb447-jgqgx
X-Real-Ip: 10.244.0.1

So far so good. Let’s move on to setting up TLS. For a full mutual TLS setup, I’ll need a CA certificate, a server certificate, and a client certificate. mTLS works as follows (with some details omitted):

• The client sends a request to the server to ask for its certificate.
• The server sends the client back its certificate.
• The client verifies the server’s certificate against its trust store, i.e. it goes through all the certificate signers it trusts and if the certificate of the server was signed by one of them then the client can trust the server.

Usually this is where TLS usually ends, but in mutual TLS, the server also verifies the client:

• The client sends its certificate.
• The server verifies it against its own trust store.

If the certificate is signed by a trusted CA, the TLS handshake is successful. The CA (Certificate Authority) certificate is the certificate that belongs to a trusted well-known entity (e.g. Google or Amazon) and it’s used to sign the other certificates. A collection of CA certificates forms a trust store.

Let’s generate a CA certificate, a server certificate and a client certificate. I’ll use the localhost domain for all of them and for the Organisation Unit I’ll set the identifier of the certificate, i.e. ca, server, client to be able to distinguish between them.

For this, the openssl tool can be used. I have a few utility scripts for generating the certificates which are pretty intuitive to use:

## Generate the CA certificate.

# Fetch the script for generating self-signed certificates:
wget https://gist.githubusercontent.com/vlasebian/1a5ff3925fe5e21c18484d5929e753ff/raw/5588e2b99151f26d2edb0b47555daaf51098cf9a/gen_self_signed_cert.sh

# Check the script before, don't run random stuff from the internet!
chmod +x gen_self_signed_cert.sh
./gen_self_signed_cert.sh -o ca

## Generate the server and client certificates.

# Fetch the script for generating certificates:
wget https://gist.githubusercontent.com/vlasebian/274c7ca97ee392117e780352bea0daa6/raw/56d4a23fd96b77a19ac9fb1e57c798b350b46930/gen_cert.sh

# Check the script before, don't run random stuff from the internet!
chmod +x gen_cert.sh
./gen_cert.sh --cacrt root.crt --cakey root.key -o server
./gen_cert.sh --cacrt root.crt --cakey root.key -o client

## Inspect certificates:

# Fetch the script for inspecting certificates:
wget https://gist.githubusercontent.com/vlasebian/918023f03b3866b4b0c691f8622d8e0b/raw/dd0bdb44935f8156dcaab50f65c3f7bca8f6dedb/inspect_cert.sh

# Check the script before, don't run random stuff from the internet!
chmod +x inspect_cert.sh
./inspect_cert.sh server.crt

The set of certificates looks like this:

-> % ls *.key *.crt
server.crt   server.key   client.crt client.key ca.crt   ca.key

Now that we have the certificates, let’s add the tls certificate to the server. I’ll configure tls on the whoami ingress resource so only traffic that matches the ingress rule will be decrypted by the provided certificate. To do that, I have to make the server certificate available in the cluster. This can be done by using a secret. The secret will be used by the whoami ingress resource so it must be created in the whoami namespace:

-> % kubectl create secret tls server-tls-certificate --cert=server.crt --key=server.key -n whoami
secret/server-tls-certificate created
-> % k -n whoami get secret
NAME                     TYPE                DATA   AGE
server-tls-certificate   kubernetes.io/tls   2      9s

I’ll update the whoami ingress resource that we applied previously to pick up the TLS secret and reapply it:

-> % cat whoami-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami
  namespace: whoami
spec:
  ingressClassName: traefik
  rules:
  - host: localhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: whoami
            port:
              name: whoamiweb
  tls:
    - hosts:
      - localhost
      secretName: server-tls-certificate
-> % k apply -f whoami-ingress.yaml
ingress.networking.k8s.io/whoami configured
-> % k -n whoami describe ingress/whoami
Name:             whoami
Labels:           <none>
Namespace:        whoami
Address:          127.0.0.1
Ingress Class:    traefik
Default backend:  <default>
TLS:
  server-tls-certificate terminates localhost
Rules:
  Host        Path  Backends
  ----        ----  --------
  localhost
              /   whoami:whoamiweb (10.244.0.3:80)
Annotations:  <none>
Events:       <none>

Let’s send a request to the endpoint using curl:

-> % curl https://localhost
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

We have an error because we didn’t provide any CA certificate (or CA path to a directory of certificates) to verify the server certificate against. Let’s pass the ca.crt that we used to sign the server certificate with:

-> % curl --cacert ca.crt https://localhost
Hostname: whoami
IP: 127.0.0.1
IP: ::1
IP: 10.244.0.3
IP: fe80::dc26:15ff:fee1:cbe7
RemoteAddr: 10.244.0.4:35884
GET / HTTP/1.1
Host: localhost
User-Agent: curl/8.7.1
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.244.0.1
X-Forwarded-Host: localhost
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik-6489fdb447-jgqgx
X-Real-Ip: 10.244.0.1

The request was successful now. You can also bypass the server certificate check by passing the -k option to curl.

Let’s check the actual TLS handshake by providing -v (verbose) flag to curl:

-> % curl -v --cacert ca.crt https://localhost
* Host localhost:443 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:443...
* Connected to localhost (::1) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: ca.crt
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=My Company; OU=server; CN=localhost; emailAddress=admin@example.com
*  start date: Jul  1 22:33:24 2025 GMT
*  expire date: Jun 29 22:33:24 2035 GMT
*  common name: localhost (matched)
*  issuer: C=US; ST=California; L=San Francisco; O=My Company; OU=CA; CN=localhost; emailAddress=admin@example.com
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://localhost/
...

The server is now configured. But how can we ensure it also authenticates the client? Since Kubernetes Ingress doesn’t support full mTLS natively, we’ll use Traefik features.

I will create another secret to store the CA certificate. The certificate provided by the client will be verified against this certificate.

# Traefik requires that the key of the CA certificate is either tls.ca or ca.crt.
kubectl create secret generic ca-tls-certificate --from-file=ca.crt -n traefik

To set the tls options for client authentication, I’ll use the TLSOption Custom Resource Definition (CRD). I’ll create this in the traefik namespace:

apiVersion: traefik.io/v1alpha1
kind: TLSOption
metadata:
  name: request-client-cert
  namespace: traefik
spec:
  clientAuth:
    secretNames:
      - ca-tls-certificate # CA certificate used by the server to verify the client certificates against
    clientAuthType: RequireAndVerifyClientCert
-> % k apply -f traefik-custom-tlsoption.yaml
tlsoption.traefik.io/request-client-cert created

Then I’ll update the whoami ingress resource to pick up the tls option by using annotations:

-> % cat whoami-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami
  namespace: whoami
  annotations:
    # Annotation value format is: <namespace>-<tls-option-name>@kubernetescrd
    traefik.ingress.kubernetes.io/router.tls.options: traefik-request-client-cert@kubernetescrd
spec:
  ingressClassName: traefik
  rules:
  - host: localhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: whoami
            port:
              name: whoamiweb
  tls:
    - hosts:
      - localhost
      secretName: server-tls-certificate
-> % k apply -f whoami-ingress.yaml
ingress.networking.k8s.io/whoami configured

Let’s send the previous curl request:

-> % curl --cacert ca.crt https://localhost
curl: (56) LibreSSL SSL_read: LibreSSL/3.3.6: error:1404C45C:SSL routines:ST_OK:reason(1116), errno 0

I have an error caused by not providing the client certificate requested by the server. Let’s fix that:

-> % curl --cacert ca.crt --cert client.crt --key client.key https://localhost

Hostname: whoami
IP: 127.0.0.1
IP: ::1
IP: 10.244.0.3
IP: fe80::dc26:15ff:fee1:cbe7
RemoteAddr: 10.244.0.4:41722
GET / HTTP/1.1
Host: localhost
User-Agent: curl/8.7.1
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.244.0.1
X-Forwarded-Host: localhost
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik-6489fdb447-jgqgx
X-Real-Ip: 10.244.0.1

Now the request is processed. Let’s send it again with a -v flag to check the TLS handshake process:

-> % curl -v --cacert ca.crt --cert client.crt --key client.key https://localhost

* Host localhost:443 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:443...
* Connected to localhost (::1) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: ca.crt
*  CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Request CERT (13):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Certificate (11):
* (304) (OUT), TLS handshake, CERT verify (15):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=My Company; OU=server; CN=localhost; emailAddress=admin@example.com
*  start date: Jul  1 22:33:24 2025 GMT
*  expire date: Jun 29 22:33:24 2035 GMT
*  common name: localhost (matched)
*  issuer: C=US; ST=California; L=San Francisco; O=My Company; OU=CA; CN=localhost; emailAddress=admin@example.com
*  SSL certificate verify ok.
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://localhost/
...

If I analyze the output of curl, there are two CERT verify operations, one IN and one OUT. If I compare it with the previous curl request where we passed the -v option, before configuring the TLSOptions in Traefik, there was only one CERT verify operation, for IN. This time the client certificate was also verified.

There are multiple behaviours that can be configured through the clientAuthType field. All the options for the clientAuthType are described here.

One more interesting behaviour would be to make the client certificate required, but let the application process it instead of Traefik. That means we have to forward the headers containing the client certificate to the application. By setting clientAuthType to RequireAnyClientCert Traefik will request the certificate but will not verify it:

-> % cat traefik-custom-tlsoption.yaml
apiVersion: traefik.io/v1alpha1
kind: TLSOption
metadata:
  name: request-client-cert
  namespace: traefik
spec:
  clientAuth:
    clientAuthType: RequireAnyClientCert
-> % k apply -f traefik-custom-tlsoption.yaml
tlsoption.traefik.io/request-client-cert configured

Notice that I dropped the secret containing the CA. That won’t be needed anymore as Traefik won’t verify the client certificate anymore, this responsability will be passed to the application.

To forward the TLS certificate in the application, I have to set a middleware that tells Traefik to pass the client certificate to the application:

-> % cat traefik-pass-tls-middleware.yaml
apiVersion: traefik.io/v1alpha1
kind: Middleware
metadata:
  name: pass-client-cert
  namespace: traefik
spec:
  passTLSClientCert:
    pem: true
-> % k apply -f traefik-pass-tls-middleware.yaml
middleware.traefik.io/pass-client-cert created

And I have to provide the middleware on the specific ingress through an ingress annotation:

-> % cat whoami-ingress.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: whoami
  namespace: whoami
  annotations:
    traefik.ingress.kubernetes.io/router.tls.options: traefik-request-client-cert@kubernetescrd
    traefik.ingress.kubernetes.io/router.middlewares: traefik-pass-client-cert@kubernetescrd
spec:
  ingressClassName: traefik
  rules:
  - host: localhost
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: whoami
            port:
              name: whoamiweb
  tls:
    - hosts:
      - localhost
      secretName: server-tls-certificate
-> % k apply -f whoami-ingress.yaml
ingress.networking.k8s.io/whoami configured

Now the client certificate is passed to the application through the request headers:

-> % curl --cacert ca.crt --cert client.crt --key client.key https://localhost

Hostname: whoami
IP: 127.0.0.1
IP: ::1
IP: 10.244.0.3
IP: fe80::dc26:15ff:fee1:cbe7
RemoteAddr: 10.244.0.4:37888
GET / HTTP/1.1
Host: localhost
User-Agent: curl/8.7.1
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.244.0.1
X-Forwarded-Host: localhost
X-Forwarded-Port: 443
X-Forwarded-Proto: https
X-Forwarded-Server: traefik-6489fdb447-jgqgx
X-Forwarded-Tls-Client-Cert: MIICbjCCAhSgAwIBAgIURmC/W6MegyfmdP/9P2VGeP/T1qQwCgYIKoZIzj0EAwIwgZIxCzAJBgNVBAYTAlVTMRMwEQYDVQQIDApDYWxpZm9ybmlhMRYwFAYDVQQHDA1TYW4gRnJhbmNpc2NvMRMwEQYDVQQKDApNeSBDb21wYW55MQswCQYDVQQLDAJDQTESMBAGA1UEAwwJbG9jYWxob3N0MSAwHgYJKoZIhvcNAQkBFhFhZG1pbkBleGFtcGxlLmNvbTAeFw0yNTA3MDEyMjI2NDBaFw0zNTA2MjkyMjI2NDBaMIGWMQswCQYDVQQGEwJVUzETMBEGA1UECAwKQ2FsaWZvcm5pYTEWMBQGA1UEBwwNU2FuIEZyYW5jaXNjbzETMBEGA1UECgwKTXkgQ29tcGFueTEPMA0GA1UECwwGY2xpZW50MRIwEAYDVQQDDAlsb2NhbGhvc3QxIDAeBgkqhkiG9w0BCQEWEWFkbWluQGV4YW1wbGUuY29tMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE4e9Ee+hCUmQcy/F/+6rfnBcPkdFh6ilSAluB6LLpgpu/VO4bxU+iVsVYN7G8VKjiFfeqZGQzkqBycYvpwz8dqKNCMEAwHQYDVR0OBBYEFLoXGaucJeOAUR8TlQw3zdOzrBiSMB8GA1UdIwQYMBaAFHzXpd9kLcMiT05oLSfYcxmFyz8vMAoGCCqGSM49BAMCA0gAMEUCIBVC7lBi2+VaDhHi/moq23NQQm0Fj9PpewKlPi2VteEMAiEAjaO1qhzdkQ2O4ECxM+9zqywhiagWKOKuIzG1d/jnIBM=
X-Real-Ip: 10.244.0.1