Skip to content

ArgoCD GitOps: Deep Dive

This document explains how ArgoCD implements GitOps, how the Application CRD works field by field, and how every piece in this demo fits together. It is meant to be read alongside the YAML files in this repo. If you want step-by-step setup instructions, see the README instead.


GitOps is a set of practices where Git is the single source of truth for your infrastructure and application configuration. Four principles define it.

Git as the single source of truth. Every desired state, from Kubernetes manifests to Helm values, lives in a Git repository. The cluster should reflect what Git says. If it does not, something is wrong.

Declarative configuration. You describe what you want, not how to get there. Kubernetes already works this way with YAML manifests. GitOps extends that pattern to the delivery pipeline itself.

Automated reconciliation. A controller watches Git and the cluster continuously. When they diverge, it takes action. No human needs to run kubectl apply. No CI pipeline needs cluster credentials.

Observable state. You can always tell whether the cluster matches Git. ArgoCD surfaces this as sync status: “Synced” means they match, “OutOfSync” means they do not. This makes drift immediately visible.

The practical effect: your Git history becomes your deployment history. Want to roll back? Revert a commit. Want to know who changed what? Check git log. Want to audit production? Compare the cluster state against the HEAD of your main branch.


ArgoCD runs inside your Kubernetes cluster as a set of controllers. Three components do the heavy lifting.

The repo server clones Git repositories and generates Kubernetes manifests from them. It understands plain YAML, Helm charts, Kustomize overlays, and Jsonnet. When ArgoCD needs to know what resources should exist, it asks the repo server to render the manifests from a given repo, revision, and path.

In this demo, the repo server clones https://github.com/savitojs/k8s-learn-by-doing.git and renders manifests from paths like demos/simple-app/manifests (plain YAML), demos/helm/chart (Helm), and demos/kustomize/overlays/development (Kustomize).

The application controller is the reconciliation loop. It compares the desired state (manifests from the repo server) against the live state (resources in the cluster). When they differ, it either reports “OutOfSync” or, if auto-sync is enabled, applies the changes.

This controller also handles self-healing. If someone manually edits a resource in the cluster, the controller detects the drift and reverts it to match Git.

The API server exposes the ArgoCD REST and gRPC APIs. The web UI and the argocd CLI both talk to it. It handles authentication, RBAC, and provides the interface for viewing application status, triggering manual syncs, and managing projects.

  1. The application controller detects that an Application is OutOfSync (either by polling or webhook notification).
  2. It asks the repo server to render the desired manifests for the configured repo, revision, and path.
  3. It diffs the desired manifests against the live cluster state.
  4. It applies the diff to the cluster, creating, updating, or deleting resources as needed.
  5. It watches the rollout until resources reach a healthy state (Pods running, Deployments available, etc.).
  6. It updates the Application status to “Synced” and “Healthy”.

By default, ArgoCD polls Git every 3 minutes. You can also configure webhooks for near-instant sync.


The Application custom resource is the core of ArgoCD. It tells ArgoCD: watch this Git path and deploy those manifests to that cluster and namespace.

Here is the simple-app Application from this demo, annotated:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: simple-app
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io/foreground
spec:
project: default
source:
repoURL: https://github.com/savitojs/k8s-learn-by-doing.git
targetRevision: HEAD
path: demos/simple-app/manifests
destination:
server: https://kubernetes.default.svc
namespace: simple-app
syncPolicy:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:
- Validate=true
- CreateNamespace=true
- PrunePropagationPolicy=foreground
- PruneLast=true
  • name: The application’s identity in ArgoCD. Shows up in the UI and CLI.
  • namespace: Must be argocd. ArgoCD only watches for Application resources in its own namespace.
  • finalizers: Covered in detail in section 5. Controls what happens when you delete this Application.

Which ArgoCD Project this application belongs to. Projects define RBAC boundaries: which repos are allowed, which namespaces can be targeted, and which resource types can be created. The default project has no restrictions. See section 7 for the custom development project in this demo.

Where ArgoCD should pull manifests from.

  • repoURL: The Git repository URL. Must be registered in ArgoCD’s repo configuration (done in terraform/values.yaml in this demo).
  • targetRevision: The Git ref to track. HEAD means the tip of the default branch. You could also use a tag (v1.2.3), a branch name (release/2.0), or a specific commit SHA.
  • path: The directory within the repo containing the manifests. ArgoCD auto-detects the format: if it finds a Chart.yaml, it treats it as Helm. If it finds a kustomization.yaml, it treats it as Kustomize. Otherwise, it treats all .yaml/.json files as plain manifests.

For Helm sources, you can add a helm block with valueFiles, values, or parameters. For Kustomize sources, you can add a kustomize block with namePrefix, images, etc. Neither is needed in this demo because the defaults work.

Where to deploy the rendered manifests.

  • server: The Kubernetes API server URL. https://kubernetes.default.svc means “the same cluster ArgoCD is running in.” For multi-cluster setups, you would register external clusters and use their server URLs here.
  • namespace: The target namespace. Combined with CreateNamespace=true in syncOptions, ArgoCD creates this namespace if it does not exist.

Controls how and when ArgoCD syncs. This is where the automation lives. Covered in full detail in the next section.


The syncPolicy block is what turns ArgoCD from a dashboard into an automated deployment tool.

Without this block, ArgoCD only reports drift. You have to click “Sync” in the UI or run argocd app sync to apply changes. Adding the automated block enables continuous deployment.

syncPolicy:
automated:
prune: true
selfHeal: true
allowEmpty: false

When a new commit lands in Git, ArgoCD automatically applies the changes to the cluster. No human approval, no manual trigger. The controller detects the diff and syncs immediately (within the polling interval or via webhook).

This is enabled for every application in this demo. The moment you push a commit that changes a manifest, ArgoCD picks it up and deploys it.

When selfHeal: true, ArgoCD watches for live drift, not just Git drift. If someone runs kubectl edit or kubectl scale to change a resource directly, ArgoCD detects the discrepancy and reverts the change to match Git.

This is powerful for enforcing GitOps discipline. It means the cluster cannot drift from Git, even if someone bypasses the Git workflow. It also means you cannot “hotfix” the cluster manually. Any manual change gets wiped. That is the point.

Default polling for self-heal is every 5 seconds.

When prune: true, ArgoCD deletes resources from the cluster that no longer exist in Git. Without this, removing a manifest from Git leaves the corresponding resource orphaned in the cluster.

Example: you delete demos/simple-app/manifests/service.yaml from Git. With prune enabled, ArgoCD deletes the Service from the cluster on the next sync. Without prune, the Service stays running with no manifest backing it.

When allowEmpty: false, ArgoCD refuses to sync if the source path yields zero manifests. This is a safety net. If your repo structure breaks or a path goes empty by accident, ArgoCD will not nuke all the resources in the target namespace.

These are flags that modify sync behavior:

  • Validate=true: Run server-side validation before applying. Catches schema errors before they hit the cluster.
  • CreateNamespace=true: Create the target namespace if it does not exist. Without this, syncing to a non-existent namespace fails.
  • PrunePropagationPolicy=foreground: When deleting resources, use foreground cascading deletion. This means the parent resource waits for all dependents to be deleted before it is removed. Prevents orphaned child resources.
  • PruneLast=true: Delete resources only after all other sync operations succeed. This prevents a situation where ArgoCD deletes the old resources but fails to create the new ones, leaving you with nothing.
  • Cascade=true (on the app-of-apps): Enable cascading deletion of child resources when the parent resource is deleted.

Every Application in this demo includes this finalizer:

metadata:
finalizers:
- resources-finalizer.argocd.argoproj.io/foreground

Kubernetes finalizers prevent an object from being deleted until a controller has finished its cleanup work. When you kubectl delete an Application that has a finalizer, Kubernetes marks it for deletion but does not actually remove it. ArgoCD’s controller sees this, deletes all the managed resources from the cluster, and then removes the finalizer. Only then does Kubernetes delete the Application object itself.

Why this matters for the App-of-Apps pattern

Section titled “Why this matters for the App-of-Apps pattern”

Without the finalizer, deleting the app-of-apps Application would remove only the Application resource itself. The child applications (simple-app, helm-app, kustomize-dev, kustomize-prod) would keep running, and all the resources they deployed would remain in the cluster.

With the finalizer, deleting app-of-apps triggers a cascade:

  1. ArgoCD deletes the child Application resources (because they are managed resources of the parent).
  2. Each child Application’s finalizer fires, causing ArgoCD to delete the resources those applications manage (Deployments, Services, etc.).
  3. Once all managed resources are gone, the finalizers are removed, and the Application objects are deleted.

The resources-finalizer.argocd.argoproj.io/foreground variant uses foreground cascading deletion. ArgoCD waits for child resources to be fully deleted before marking the parent as deleted. The alternative, resources-finalizer.argocd.argoproj.io (without /foreground), uses background deletion, which is faster but does not guarantee ordering.

If you remove the finalizer and delete an Application, ArgoCD removes the Application object but leaves all managed resources running in the cluster. This is sometimes intentional (you want to stop managing an app without destroying it), but usually it is a mistake.


The App-of-Apps pattern is an ArgoCD Application whose source path contains other Application manifests. Instead of managing Deployments and Services, it manages Application resources.

app-of-apps (Application)
|
+-- points to: demos/argocd/applications/
|
+-- 1-simple-app.yaml (Application)
+-- 2-helm-app.yaml (Application)
+-- 3a-kustomize-dev.yaml (Application)
+-- 3b-kustomize-prod.yaml (Application)

Here is the parent Application from this demo:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: app-of-apps
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io/foreground
spec:
project: default
source:
repoURL: https://github.com/savitojs/k8s-learn-by-doing.git
targetRevision: HEAD
path: demos/argocd/applications
destination:
server: https://kubernetes.default.svc
namespace: argocd
syncPolicy:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:
- Validate=true
- CreateNamespace=true
- PrunePropagationPolicy=foreground
- PruneLast=true
- Cascade=true

Notice the destination.namespace is argocd, not an application namespace. That is because the resources this Application manages are other Application CRDs, and those must live in the argocd namespace.

  • You have multiple applications and want a single kubectl apply to deploy all of them.
  • You want adding a new application to be as simple as adding a YAML file to a directory and pushing to Git.
  • You want a single delete to tear down everything cleanly (via finalizers).
  • You are managing environments where each environment gets its own set of applications.
  • You have only one or two applications. Just apply them directly.
  • You need fine-grained sync control per application (though you still get that, since each child Application has its own syncPolicy).
  • Your applications span multiple repos with different access controls. Consider ApplicationSets instead.
  1. You apply app-of-apps.yaml. ArgoCD creates one Application.
  2. ArgoCD syncs the parent. It reads demos/argocd/applications/ and finds four YAML files.
  3. It creates four child Application resources in the argocd namespace.
  4. Each child Application triggers its own sync loop, deploying resources to its respective namespace.
  5. Within seconds, you have four applications running across four namespaces, all managed from Git.

To add a fifth application, you create 5-new-app.yaml in the applications/ directory and push. The parent auto-syncs, creates the new child Application, and the child deploys its resources. No kubectl apply needed.


ArgoCD Projects are a boundary mechanism. They control which repositories an Application can pull from, which clusters and namespaces it can deploy to, and which Kubernetes resource types it can create.

Here is the development project from this demo:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
name: development
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io
spec:
description: Development environment project
sourceRepos:
- "https://github.com/savitojs/k8s-learn-by-doing.git"
- "https://helm.nginx.com/stable"
- "https://charts.bitnami.com/bitnami"
destinations:
- namespace: "dev-*"
server: https://kubernetes.default.svc
- namespace: "default"
server: https://kubernetes.default.svc
- namespace: "kustomize-example"
server: https://kubernetes.default.svc
clusterResourceWhitelist:
- group: ""
kind: Namespace
- group: rbac.authorization.k8s.io
kind: ClusterRole
- group: rbac.authorization.k8s.io
kind: ClusterRoleBinding
namespaceResourceWhitelist:
- group: ""
kind: Service
- group: ""
kind: ServiceAccount
- group: ""
kind: ConfigMap
- group: ""
kind: Secret
- group: apps
kind: Deployment
- group: apps
kind: ReplicaSet
- group: ""
kind: Pod
- group: networking.k8s.io
kind: Ingress
- group: networking.k8s.io
kind: NetworkPolicy

Restricts which Git repos or Helm chart repos applications in this project can use. An Application referencing a repo not in this list will be rejected. In this demo, the project allows the main demo repo plus two Helm chart repos (nginx and bitnami).

The default project allows all repos (*). Custom projects should always be explicit.

Restricts which cluster/namespace combinations are valid. Notice the wildcard dev-*, which allows any namespace prefixed with dev-. This is a common pattern: give each environment a namespace prefix, then use Projects to enforce boundaries.

An Application in this project targeting namespace: production on this cluster would be rejected. That is the point.

Cluster-scoped resources (Namespaces, ClusterRoles, ClusterRoleBindings) are powerful. This whitelist explicitly names which cluster-scoped types this project is allowed to create. Without this list, the project cannot create any cluster-scoped resources.

This project allows creating Namespaces and RBAC resources, but not things like CustomResourceDefinitions or PersistentVolumes. That limits the blast radius.

Same idea, but for namespace-scoped resources. This project can create Services, ConfigMaps, Secrets, Deployments, and a few other common types. It cannot create, say, a PodDisruptionBudget or a HorizontalPodAutoscaler, because those are not on the list.

This is defense in depth. Even if someone has access to create Applications in this project, they can only deploy a limited set of resource types.

To assign an Application to a project, set spec.project:

spec:
project: development # instead of "default"

The applications in this demo currently use project: default for simplicity. In a real environment, you would assign them to scoped projects like development or production.


8. How ArgoCD Handles Helm vs Kustomize vs Plain Manifests

Section titled “8. How ArgoCD Handles Helm vs Kustomize vs Plain Manifests”

ArgoCD’s repo server auto-detects the manifest format based on what it finds in the source path.

If the path contains .yaml or .json files without a Chart.yaml or kustomization.yaml, ArgoCD treats every file as a raw Kubernetes manifest. No rendering, no templating. What you see in Git is what gets applied.

This is what happens with simple-app:

source:
repoURL: https://github.com/savitojs/k8s-learn-by-doing.git
targetRevision: HEAD
path: demos/simple-app/manifests

ArgoCD reads all YAML files in that directory and applies them as-is.

If the path contains a Chart.yaml, ArgoCD treats it as a Helm chart. It runs the equivalent of helm template to render the manifests, then applies the result.

This is what happens with helm-app:

source:
repoURL: https://github.com/savitojs/k8s-learn-by-doing.git
targetRevision: HEAD
path: demos/helm/chart

You can override Helm values without modifying the chart by adding a helm block to the source:

source:
path: demos/helm/chart
helm:
valueFiles:
- values-production.yaml
parameters:
- name: replicaCount
value: "3"
values: |
ingress:
enabled: true

Key detail: ArgoCD does not use helm install or helm upgrade. It uses helm template to generate manifests, then manages those manifests itself. This means Helm hooks, Helm rollback, and helm ls do not apply. ArgoCD owns the lifecycle, not Helm.

If the path contains a kustomization.yaml, ArgoCD runs kustomize build and applies the result.

This is what happens with kustomize-dev and kustomize-prod:

# Dev overlay
source:
path: demos/kustomize/overlays/development
# Prod overlay
source:
path: demos/kustomize/overlays/production

Same base manifests, different overlays. Each overlay applies its own patches, labels, replica counts, or resource limits. ArgoCD deploys each to a separate namespace.

You can customize Kustomize behavior in the Application spec:

source:
path: demos/kustomize/overlays/development
kustomize:
namePrefix: dev-
images:
- myapp=myregistry/myapp:v2.0

The auto-detection means you do not need to tell ArgoCD what kind of source you are using. Drop a Helm chart in a directory, point an Application at it, and ArgoCD figures out the rest. Switch from plain manifests to Kustomize by adding a kustomization.yaml, and ArgoCD adapts on the next sync.

This demo shows all three approaches deployed side by side, managed by the same ArgoCD instance, through the same App-of-Apps pattern.


  1. Create a new Application YAML in applications/:
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: my-new-app
namespace: argocd
finalizers:
- resources-finalizer.argocd.argoproj.io/foreground
spec:
project: default
source:
repoURL: https://github.com/savitojs/k8s-learn-by-doing.git
targetRevision: HEAD
path: demos/my-new-app/manifests
destination:
server: https://kubernetes.default.svc
namespace: my-new-app
syncPolicy:
automated:
prune: true
selfHeal: true
allowEmpty: false
syncOptions:
- Validate=true
- CreateNamespace=true
- PrunePropagationPolicy=foreground
- PruneLast=true
  1. Commit and push. The app-of-apps parent auto-syncs, discovers the new file, creates the Application, and the child syncs its resources. Done.

Change the manifests in the source path (e.g., bump an image tag, change replica count, add an environment variable). Commit and push. ArgoCD detects the change and syncs automatically.

Terminal window
# Example: update image tag
sed -i 's|image: nginx:1.25|image: nginx:1.26|' demos/simple-app/manifests/deployment.yaml
git add . && git commit -m "bump nginx to 1.26" && git push

ArgoCD picks up the change within 3 minutes (or instantly with webhooks).

With Kustomize overlays, promotion is a Git operation. This demo has separate overlays for dev and prod, each deployed by its own Application:

  • kustomize-dev watches demos/kustomize/overlays/development
  • kustomize-prod watches demos/kustomize/overlays/production

To promote a change from dev to prod, update the production overlay to match what you tested in dev. Commit and push. ArgoCD syncs the production Application.

This can be a manual commit, a PR-based review process, or an automated promotion pipeline that copies values from one overlay to another after tests pass.

GitOps rollback is a Git revert.

Terminal window
git revert HEAD
git push

ArgoCD detects the revert commit and syncs the previous state. The cluster goes back to what it was before the bad change.

You can also roll back from the ArgoCD UI or CLI:

Terminal window
# Roll back to a specific Git revision
argocd app sync simple-app --revision abc123

But this is a temporary override. On the next auto-sync cycle, ArgoCD will re-sync to HEAD. For a persistent rollback, revert in Git.

Delete the Application YAML from the applications/ directory and push. The parent Application’s prune: true setting detects the missing file and deletes the child Application. The child’s finalizer kicks in and deletes all managed resources. Clean removal, end to end.


Symptoms: The application shows OutOfSync even after a manual sync.

Common causes:

  • Defaulted fields: Kubernetes adds default values to resources (e.g., strategy.rollingUpdate on Deployments). ArgoCD sees these as differences between the desired state (your manifest) and the live state (with defaults filled in). Fix: add the defaulted fields to your manifest, or configure resource customizations to ignore them.

  • Mutating webhooks: An admission webhook modifies resources after ArgoCD applies them. ArgoCD sees the modification as drift. Fix: configure diff customizations to ignore the mutated fields.

  • Helm hooks: If you have Helm hooks that create resources, those resources may show as OutOfSync since ArgoCD manages the lifecycle differently than Helm. Fix: annotate hook resources with argocd.argoproj.io/hook.

Check what ArgoCD sees as different:

Terminal window
argocd app diff simple-app

Symptoms: The sync operation starts but never completes.

Common causes:

  • Resource health check hanging: ArgoCD waits for resources to become “Healthy.” If a Deployment cannot pull its image or a Pod is crash-looping, the sync will not complete. Check the pod status:
Terminal window
kubectl get pods -n simple-app
kubectl describe pod <pod-name> -n simple-app
  • Resource dependencies: A resource depends on something that does not exist yet (e.g., a ConfigMap referenced by a Deployment). ArgoCD applies resources in waves, but sometimes the ordering is wrong. Fix: use sync waves with the argocd.argoproj.io/sync-wave annotation.

Symptoms: You delete an Application, but it hangs in a “Deleting” state.

Common causes:

  • Finalizer waiting for resources to be deleted: The finalizer tells ArgoCD to delete all managed resources first. If a managed resource is stuck (e.g., a namespace with a finalizer of its own, or a PVC waiting for a pod to unmount), the Application deletion hangs.

Check what is stuck:

Terminal window
kubectl get all -n simple-app
kubectl get pvc -n simple-app
  • Stuck namespace: If CreateNamespace=true was used and the namespace has resources that will not delete (e.g., resources from another controller), the entire chain stalls.

Nuclear option (use carefully): remove the finalizer from the Application to let it be deleted without cleaning up managed resources.

Terminal window
kubectl patch application simple-app -n argocd \
--type json -p='[{"op": "remove", "path": "/metadata/finalizers"}]'

Then manually clean up the orphaned resources.

Symptoms: The application shows a ComparisonError instead of Synced/OutOfSync.

Common causes:

  • Invalid manifests: A YAML file in the source path has syntax errors or references invalid API versions. Check the repo server logs:
Terminal window
kubectl logs -n argocd -l app.kubernetes.io/component=repo-server --tail=50
  • Repo access failure: ArgoCD cannot clone the repository. Check that the repo URL is correct and credentials are configured. In this demo, the repo is configured in terraform/values.yaml:
configs:
repositories:
github.com:
url: https://github.com/savitojs/k8s-learn-by-doing.git
insecure: "true"
insecureIgnoreHostKey: "true"

For private repos, you would need SSH keys or a personal access token configured here.

This is working as intended. Self-heal exists to prevent manual cluster drift. If you need to make a change, make it in Git. If you need to temporarily disable self-heal for debugging:

Terminal window
argocd app set simple-app --self-heal=false

Remember to re-enable it when you are done.

Symptoms: You deleted a manifest from Git, but the resource still exists in the cluster.

Common causes:

  • prune is not enabled: Check that automated.prune: true is set in the Application’s syncPolicy.
  • Resource has the argocd.argoproj.io/compare-options: IgnoreExtraneous annotation: This tells ArgoCD to ignore the resource during comparison. It will not be pruned.
  • Resource is not tracked by this Application: ArgoCD tracks resources using labels. If a resource was created outside of ArgoCD or by a different Application, this Application will not prune it.

ConceptWhat it doesWhere it is configured
Auto-syncSyncs on Git changes without manual triggersyncPolicy.automated
Self-healReverts manual cluster changes to match GitsyncPolicy.automated.selfHeal: true
Auto-pruneDeletes resources removed from GitsyncPolicy.automated.prune: true
FinalizerEnsures managed resources are deleted when the Application is deletedmetadata.finalizers
App-of-AppsApplication that manages other ApplicationsSource path contains Application YAMLs
ProjectRBAC boundary for repos, namespaces, and resource typesAppProject CRD
CreateNamespaceCreates target namespace automaticallysyncOptions: CreateNamespace=true
PruneLastDeletes old resources only after new ones are healthysyncOptions: PruneLast=true