Deploying Shoehorn with ArgoCD

Production-ready ArgoCD manifests for deploying the Shoehorn Helm chart without common anti-patterns.

Prerequisites

• ArgoCD v2.9+ installed in your cluster
• kubectl access to the ArgoCD namespace
• Shoehorn secrets already created (see Helm deployment guide)
• OCI registry credential configured in ArgoCD (type helm, enableOCI: true)

Important: How ArgoCD Deploys Helm Charts

ArgoCD runs helm template and applies the output via kubectl apply. It never runs helm install or helm upgrade. This means:

• Helm hooks (pre-install, post-install) are converted to ArgoCD sync hooks with different lifecycle semantics
• .Release.IsInstall / .Release.IsUpgrade are not meaningful
• Use ArgoCD sync hooks (PreSync, PostSync, SyncFail) instead of Helm hooks when needed

Step 1: Lock Down the Default AppProject

Before creating anything, restrict the default project so Applications cannot escape tenant boundaries:

kubectl patch appproject default -n argocd --type merge -p '{"spec":{"sourceRepos":[],"destinations":[]}}'

Step 2: Create the Shoehorn AppProject

Scope permissions to only what Shoehorn needs.

argocd/project.yaml

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: shoehorn
  namespace: argocd
spec:
  description: Shoehorn Intelligent Developer Platform
  sourceRepos:
    # Explicit repo -- never use '*' which allows any source
    - "ghcr.io/shoehorn-dev/helm-charts"
  destinations:
    # Single namespace on the local cluster only
    - namespace: shoehorn
      server: https://kubernetes.default.svc
  clusterResourceWhitelist:
    - group: ""
      kind: Namespace
    - group: "cert-manager.io"
      kind: ClusterIssuer
  namespaceResourceBlacklist:
    # Secrets managed outside ArgoCD (pre-created or via External Secrets)
    - group: ""
      kind: Secret

Step 3: Create the Application

argocd/application.yaml

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: shoehorn
  namespace: argocd
  annotations:
    # Notify on sync status (if using argocd-notifications)
    notifications.argoproj.io/subscribe.on-sync-succeeded.slack: shoehorn-deploys
    notifications.argoproj.io/subscribe.on-sync-failed.slack: shoehorn-deploys
    # Scope manifest generation to this chart only (critical for monorepo perf)
    argocd.argoproj.io/manifest-generate-paths: .
  finalizers:
    # Ensures child resources are cleaned up on Application deletion.
    # CAUTION: Remove this finalizer BEFORE renaming or moving the Application,
    # otherwise ArgoCD cascade-deletes all managed resources.
    - resources-finalizer.argocd.argoproj.io
spec:
  project: shoehorn

  source:
    # OCI source -- no oci:// prefix (ArgoCD adds it automatically)
    repoURL: ghcr.io/shoehorn-dev/helm-charts
    chart: shoehorn
    # Always pin to a specific semver -- never use '*', 'latest', or mutable tags
    targetRevision: 0.1.0
    helm:
      releaseName: shoehorn
      # Inline values -- for larger configs, use multi-source with a Git values repo
      values: |
        global:
          domain: shoehorn.example.com
          environment: production
          logLevel: info
          storageClass: "gp3"
          organization:
            name: "Acme Corp"
            slug: "acme-corp"

        image:
          # Pin image tag to a release. Avoid 'latest' for reproducibility.
          tag: "v2026.3.0"
          pullPolicy: IfNotPresent

        replicaCount:
          api: 2
          web: 2
          worker: 3
          crawler: 2
          forge: 2
          eventbus: 1

        auth:
          provider: zitadel
          zitadel:
            externalUrl: https://auth.example.com
            projectId: "CHANGE_ME"
            clientId: "CHANGE_ME"

        postgresql:
          enabled: true
          persistence:
            size: 20Gi

        meilisearch:
          enabled: true
          persistence:
            size: 10Gi

        valkey:
          enabled: true

        redpanda:
          enabled: true

        cerbos:
          enabled: true

        ingressRoute:
          enabled: true
          tls:
            enabled: true
            certResolver: letsencrypt

  destination:
    server: https://kubernetes.default.svc
    namespace: shoehorn

  # --- Sync Policy ---
  syncPolicy:
    automated:
      prune: true          # Clean up resources removed from the chart
      selfHeal: true       # Revert manual drift back to Git state
      allowEmpty: false    # Safety net -- never delete ALL resources
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true            # Avoids 262KB annotation size limit
      - PrunePropagationPolicy=foreground
      - PruneLast=true                  # Apply new resources before pruning old ones
      - RespectIgnoreDifferences=true
    retry:
      limit: 3
      backoff:
        duration: 30s
        factor: 2
        maxDuration: 5m

  # --- Ignore Differences ---
  # Fields managed by other controllers that ArgoCD should not fight over
  ignoreDifferences:
    # cert-manager injects caBundle after creation
    - group: admissionregistration.k8s.io
      kind: MutatingWebhookConfiguration
      jsonPointers:
        - /webhooks/0/clientConfig/caBundle
    - group: admissionregistration.k8s.io
      kind: ValidatingWebhookConfiguration
      jsonPointers:
        - /webhooks/0/clientConfig/caBundle
    # HPA manages replica count -- don't fight it
    - group: apps
      kind: Deployment
      jsonPointers:
        - /spec/replicas

Step 4: Apply

kubectl apply -f argocd/project.yaml
kubectl apply -f argocd/application.yaml

Or with the CLI:

argocd app create shoehorn -f argocd/application.yaml
argocd app sync shoehorn

Anti-Patterns Avoided

This configuration explicitly avoids the following common production issues:

Anti-Pattern	Risk	What We Do Instead
sourceRepos: ['*'] on AppProject	Any repo can deploy to your namespace	Explicit ghcr.io/shoehorn-dev/helm-charts
Auto-sync without prune: true	Removed chart resources linger as orphans	Prune enabled with PruneLast=true ordering
Missing allowEmpty: false	A bad commit can delete all resources	Set to false as a safety net
targetRevision: * or latest	Non-reproducible deploys, silent upgrades	Pin to semver 0.1.0
Image tag latest	Containers change without Git audit trail	Pin to v2026.3.0
Secrets in Application values	Credentials committed to Git in plaintext	Secrets pre-created or via External Secrets Operator
Client-side apply (default)	Fails on resources >262KB (CRDs, large ConfigMaps)	ServerSideApply=true
No retry policy	Transient network errors fail the sync permanently	Retry 3x with exponential backoff
oci:// prefix in repoURL	Connection failure (ArgoCD adds protocol itself)	Bare registry URL
No ignoreDifferences for controller-managed fields	Endless OutOfSync loop (cert-manager, HPA)	Explicit ignores for caBundle and replicas
Using argocd app set parameter overrides	Bypasses Git as source of truth	All values in the Application spec, committed to Git
Renaming Application with finalizer attached	Cascade-deletes all production resources	Documented warning in annotation
Default AppProject left open	Escape hatch for any Application	Locked down with empty destinations/sources

Managing Secrets

Never store secrets in the Application values. Use one of these approaches:

Option A: External Secrets Operator (Recommended)

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: database-credentials
  namespace: shoehorn
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager    # or vault, gcp, azure
    kind: ClusterSecretStore
  target:
    name: database-credentials
  data:
    - secretKey: postgres_password
      remoteRef:
        key: shoehorn/database
        property: postgres_password
    - secretKey: db_password
      remoteRef:
        key: shoehorn/database
        property: db_password

Deploy the ESO controller independently from ArgoCD (or in a separate “infra” Application with Prune=false). Never let the secret management system be managed by the same Application that depends on it.

Option B: SealedSecrets

kubeseal --format yaml \
  --controller-name sealed-secrets \
  --controller-namespace kube-system \
  < secret.yaml > sealed-secret.yaml

Commit sealed-secret.yaml to Git. The SealedSecrets controller decrypts it in-cluster.

Option C: Pre-created Secrets

Create secrets before ArgoCD syncs (as shown in the Helm guide). The namespaceResourceBlacklist on the AppProject ensures ArgoCD does not manage or prune them.

Upgrading Shoehorn

1. Update targetRevision in application.yaml to the new chart version
2. Review the changelog for breaking changes
3. Commit and push — ArgoCD auto-syncs the upgrade
4. Monitor:

argocd app get shoehorn
argocd app wait shoehorn --health

Automate version bumps with Renovate or Dependabot creating PRs against your GitOps repo.

Multi-Source Values (Recommended for Teams)

For larger configurations, separate the chart source from the values file using multi-source Applications (ArgoCD 2.6+):

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: shoehorn
  namespace: argocd
spec:
  project: shoehorn
  sources:
    # Source 1: The Helm chart from OCI
    - repoURL: ghcr.io/shoehorn-dev/helm-charts
      chart: shoehorn
      targetRevision: 0.1.0
      helm:
        releaseName: shoehorn
        valueFiles:
          - $values/shoehorn/values-production.yaml
    # Source 2: Values from your Git repo
    - repoURL: https://github.com/acme-corp/gitops-config
      targetRevision: main
      ref: values
  destination:
    server: https://kubernetes.default.svc
    namespace: shoehorn

This keeps values version-controlled in your own repo with full PR review workflow.

Multi-Cluster Deployment

Use an ApplicationSet with preserveResourcesOnDeletion to prevent accidental deletion:

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: shoehorn
  namespace: argocd
spec:
  generators:
    - list:
        elements:
          - cluster: staging
            url: https://staging-cluster.example.com
            revision: 0.2.0-rc.1
            domain: shoehorn-staging.example.com
          - cluster: production
            url: https://kubernetes.default.svc
            revision: 0.1.0
            domain: shoehorn.example.com
  template:
    metadata:
      name: "shoehorn-{{cluster}}"
    spec:
      project: shoehorn
      source:
        repoURL: ghcr.io/shoehorn-dev/helm-charts
        chart: shoehorn
        targetRevision: "{{revision}}"
        helm:
          values: |
            global:
              domain: "{{domain}}"
              environment: "{{cluster}}"
      destination:
        server: "{{url}}"
        namespace: shoehorn
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
          allowEmpty: false
  # Prevent cascade-deletion if an ApplicationSet element is removed
  syncPolicy:
    preserveResourcesOnDeletion: true

Resource Tracking

If you see app.kubernetes.io/instance label conflicts (ArgoCD vs Helm both setting it), switch to annotation-based tracking in argocd-cm:

data:
  application.resourceTrackingMethod: annotation+label

Use annotation+label as a migration step before moving to annotation only. Sync all Applications before changing the method to avoid orphaning resources.

Monitoring

# Application health and sync status
argocd app get shoehorn

# Watch a sync in progress
argocd app sync shoehorn --watch

# Sync history
argocd app history shoehorn

# Diff what would change
argocd app diff shoehorn