Enable TLS with cert-manager

This HowTo walks you through enabling TLS for every Shoehorn service using cert-manager. The Shoehorn Helm chart ships with cert-manager integration built in, so in most cases you only need to flip a handful of values.

You’ll enable, in order:

1. cert-manager itself (bundled with the chart, or bring your own)
2. A ClusterIssuer (self-signed CA for internal traffic, Let’s Encrypt for the edge)
3. gRPC mTLS between all internal services (API, Forge, Worker, Crawler, Eventbus, Cerbos)
4. PostgreSQL server TLS
5. Meilisearch HTTPS
6. Redpanda Kafka TLS
7. Edge TLS on the public ingress (Ingress / Traefik IngressRoute / Envoy Gateway HTTPRoute)

By the end, every connection — from the browser to the pod, and between pods — is encrypted, with certificates that rotate automatically.

Prerequisites

• A running Kubernetes cluster (v1.28+) with the Shoehorn Helm chart installed or ready to install. See Deploying Shoehorn with Helm.
• kubectl and helm configured against the cluster.
• Cluster admin permissions (cert-manager installs CRDs and ClusterIssuers).
• For public-facing TLS: a DNS record pointing to your ingress controller, and outbound access to Let’s Encrypt from the cluster.

Step 1: Install cert-manager

You have two choices. Pick one.

Option A — Bundled install (simplest)

Let the Shoehorn chart install cert-manager as a dependency. This is ideal for new clusters and single-tenant deployments.

values.yaml

certManager:
  install: true            # Install cert-manager as a sub-chart
  installCRDs: true        # Install cert-manager CRDs
  createClusterIssuer: true  # Create the Shoehorn self-signed CA ClusterIssuer
  issuer:
    name: shoehorn-ca-issuer
    kind: ClusterIssuer

Then run the usual helm upgrade --install. The chart pulls cert-manager v1.20.0 from https://charts.jetstack.io and wires it in.

Option B — External cert-manager (recommended for shared clusters)

If cert-manager is already installed in the cluster (or managed by a platform team), leave the bundled install off. Shoehorn will create its own ClusterIssuer and use the existing cert-manager.

values.yaml

certManager:
  install: false           # Do NOT install cert-manager (use existing)
  createClusterIssuer: true  # Still let Shoehorn create its self-signed CA for internal services
  issuer:
    name: shoehorn-ca-issuer
    kind: ClusterIssuer

If you want Shoehorn to use your own existing issuer instead of creating one, set createClusterIssuer: false and point certManager.issuer.name at it.

Verify cert-manager is healthy before moving on:

kubectl -n cert-manager get pods
kubectl get crd | grep cert-manager.io

Step 2: Understand the default ClusterIssuer

When certManager.createClusterIssuer: true, the chart bootstraps a three-step chain:

1. A temporary ClusterIssuer named <release>-selfsigned-issuer (type selfSigned).
2. A root CA Certificate named <release>-ca with a 10-year duration, stored in the secret <release>-ca-secret.
3. The real ClusterIssuer named shoehorn-ca-issuer that uses the root CA to sign every internal certificate.

All internal service certificates (mTLS, Postgres, Meilisearch, Redpanda) are signed by shoehorn-ca-issuer and rotate every 90 days automatically (with renewal 15 days before expiry). You can customize the CA duration and organization name:

certManager:
  issuer:
    ca:
      organizationName: "Acme Corp"
      duration: 87600h       # 10 years
      renewBefore: 8760h     # 1 year

Note: This self-signed CA is only used for internal cluster traffic, where the client is another Shoehorn pod that trusts the CA via the mounted secret. It is not trusted by browsers. For public-facing TLS, use Let’s Encrypt — see Step 7.

Step 3: Enable gRPC mTLS between services

Shoehorn’s microservices talk to each other over gRPC. When mTLS is enabled, every call is encrypted and authenticated in both directions.

global:
  mtls:
    enabled: true
    createCertificates: true       # Let the chart create the cert via cert-manager
    issuerName: shoehorn-ca-issuer
    issuerKind: ClusterIssuer
    # These paths are mounted automatically into every backend pod
    certFile: "/etc/shoehorn/certs/tls.crt"
    keyFile: "/etc/shoehorn/certs/tls.key"
    caFile: "/etc/shoehorn/certs/ca.crt"

The chart creates a single wildcard Certificate covering every backend service (API, Forge, Worker, Crawler, Eventbus, Cerbos) with DNS SANs for both the fully qualified and short service names. All services mount the same secret (<release>-grpc-mtls-cert) so there’s only one cert to rotate.

Apply the values, then verify:

kubectl -n shoehorn get certificate
kubectl -n shoehorn describe certificate shoehorn-grpc-mtls

You should see Ready: True and a Not After timestamp ~90 days out.

Step 4: PostgreSQL TLS

Encrypt every connection between Shoehorn services and PostgreSQL.

postgresql:
  enabled: true
  tls:
    enabled: true
    createCertificate: true
    issuerName: shoehorn-ca-issuer
    issuerKind: ClusterIssuer
    certFile: "/etc/postgresql/certs/tls.crt"
    keyFile: "/etc/postgresql/certs/tls.key"

The chart creates a Postgres server certificate covering the headless service DNS and the StatefulSet pod DNS, and mounts it into the Postgres pod. Shoehorn services connecting over sslmode=require will now see a valid certificate signed by the Shoehorn CA.

Using an external database (RDS, Cloud SQL, …)? Skip this step. External DB TLS is configured through your cloud provider and passed to Shoehorn via the database connection string. See postgresql.external in values.yaml.

Step 5: Meilisearch HTTPS

meilisearch:
  tls:
    enabled: true
    createCertificate: true
    issuerName: shoehorn-ca-issuer
    issuerKind: ClusterIssuer

Meilisearch will start on HTTPS and the API service will automatically use https:// when connecting. No additional client config is needed.

Step 6: Redpanda Kafka TLS

redpanda:
  tls:
    enabled: true
    createCertificate: true
    issuerName: shoehorn-ca-issuer
    issuerKind: ClusterIssuer

Redpanda brokers terminate TLS on the Kafka listener. The EventBus, Worker, and any other Kafka clients use the mounted CA to verify broker certificates.

Step 7: Edge TLS (public ingress)

Internal TLS is signed by the Shoehorn self-signed CA. For traffic coming into the cluster from browsers and external clients, you need a certificate that public clients trust — almost always from Let’s Encrypt.

The Shoehorn chart supports three ingress strategies. Pick the one that matches your ingress controller.

Option A — Standard Kubernetes Ingress (works with any controller)

This is the most portable option and integrates cleanly with cert-manager via annotations. First, create a Let’s Encrypt ClusterIssuer (this is separate from the internal shoehorn-ca-issuer):

letsencrypt-clusterissuer.yaml

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: platform@example.com
    privateKeySecretRef:
      name: letsencrypt-prod-account
    solvers:
      - http01:
          ingress:
            class: nginx   # or traefik, envoy, etc.

kubectl apply -f letsencrypt-clusterissuer.yaml

Then configure Shoehorn’s ingress to request a certificate from it:

values.yaml

ingressRoute:
  enabled: false
httpRoute:
  enabled: false
ingress:
  enabled: true
  className: nginx   # or your ingress class
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
  tls:
    - hosts:
        - shoehorn.example.com
      secretName: shoehorn-tls  # cert-manager creates this automatically
  hosts:
    - host: shoehorn.example.com
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: web
              port: 4173

cert-manager will see the annotation, solve the ACME HTTP-01 challenge, and write the certificate into the shoehorn-tls secret. Your ingress controller reloads and starts serving HTTPS.

Option B — Traefik IngressRoute

If you use Traefik, the Shoehorn chart defaults to using Traefik’s built-in ACME certResolver for edge TLS. This works without cert-manager on the edge, because Traefik itself talks to Let’s Encrypt.

ingressRoute:
  enabled: true
  tls:
    enabled: true
    certResolver: letsencrypt   # Traefik resolver name

Your Traefik install must have a certResolver named letsencrypt configured — see the Traefik ACME docs. In this mode Traefik, not cert-manager, manages the public certificate. Internal TLS (Steps 3–6) still goes through cert-manager.

If you’d rather have cert-manager manage the public cert and keep Traefik IngressRoute, switch to Option A (standard Ingress) or create the Secret out-of-band with a cert-manager Certificate resource and reference it from a custom IngressRoute. The bundled ingressRoute.yaml template does not currently expose tls.secretName.

Option C — Envoy Gateway HTTPRoute

Envoy Gateway handles TLS on the Gateway resource, not on the HTTPRoute. Configure cert-manager on your existing Gateway (outside the chart) and leave the Shoehorn HTTPRoute plain:

# In your envoy-gateway-system namespace
apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: eg
  namespace: envoy-gateway-system
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
spec:
  gatewayClassName: eg
  listeners:
    - name: https
      port: 443
      protocol: HTTPS
      hostname: shoehorn.example.com
      tls:
        mode: Terminate
        certificateRefs:
          - name: shoehorn-tls

Then in Shoehorn values:

ingressRoute:
  enabled: false
ingress:
  enabled: false
httpRoute:
  enabled: true
  gatewayName: eg
  gatewayNamespace: envoy-gateway-system

cert-manager will populate shoehorn-tls for the Gateway; the HTTPRoute just binds Shoehorn’s services to it.

Step 8: Verify end-to-end

After applying all the values above, run a full check:

# 1. All certificates ready?
kubectl get certificate -A

# Expected: every cert has READY=True

# 2. Issuer healthy?
kubectl get clusterissuer
kubectl describe clusterissuer shoehorn-ca-issuer

# 3. Pods restarted with TLS secrets mounted?
kubectl -n shoehorn get pods
kubectl -n shoehorn exec deploy/shoehorn-api -- ls /etc/shoehorn/certs

# 4. Public endpoint serves a trusted cert?
curl -vI https://shoehorn.example.com 2>&1 | grep -E "subject|issuer|HTTP/"

From inside a pod, confirm the API reaches Postgres over TLS:

kubectl -n shoehorn exec deploy/shoehorn-api -- \
  sh -c 'openssl s_client -connect shoehorn-postgresql.shoehorn-data:5432 -starttls postgres -showcerts 2>/dev/null | openssl x509 -noout -subject -issuer'

The issuer should be Shoehorn Platform Root CA (or your configured organization name).

Rotation and renewal

cert-manager handles all renewals automatically:

Certificate	Duration	Renew before
Root CA (internal)	10 years	1 year
Service certs (mTLS, Postgres, Meilisearch, Redpanda)	90 days	15 days
Let’s Encrypt (edge)	90 days	30 days (cert-manager default)

When a certificate is renewed, the Secret is updated in place. Pods mounting the secret via a projected volume pick up the change automatically on the next read (gRPC clients reload on reconnect). If you need to force a pod restart after a manual rotation, run kubectl rollout restart deploy/<name>.

Troubleshooting

Certificate stuck in False / pending

kubectl describe certificate <name> -n <namespace>
kubectl describe certificaterequest -n <namespace>
kubectl describe order -n <namespace>           # for ACME
kubectl describe challenge -n <namespace>       # for ACME

Most failures are: DNS not propagated, HTTP-01 challenge blocked by firewall, wrong ingressClassName on the ClusterIssuer solver, or ACME rate limits.

Pods can’t connect to Postgres after enabling TLS

Make sure Shoehorn services are using sslmode=require (or stricter) in the connection string, and that the Postgres pod has actually restarted after the certificate was issued. Check kubectl -n shoehorn-data logs statefulset/shoehorn-postgresql for TLS startup errors.

Browser shows “untrusted certificate”

You’re almost certainly serving the internal self-signed CA instead of a public cert. Double-check that your ingress resource (Option A/B/C in Step 7) is configured to use Let’s Encrypt, not shoehorn-ca-issuer.

mTLS “x509: certificate signed by unknown authority”

Both client and server need to mount the same CA. Verify global.mtls.createCertificates: true is set, or — if you’re providing certs externally — that every service has the CA mounted at global.mtls.caFile.

ArgoCD shows cert-manager resources as OutOfSync

cert-manager injects caBundle fields post-creation. Add the ignore rules shown in the ArgoCD deployment guide under “Common gotchas”.

Related

• Deploying Shoehorn with Helm — full chart reference
• Security Overview — platform security model
• Multi-tenant deployment — RLS and tenant isolation
• cert-manager docs — upstream reference