Skip to main content

Command Palette

Search for a command to run...

Your GitOps Pipeline is a Lie Until You Prove Otherwise

Published
10 min read
Your GitOps Pipeline is a Lie Until You Prove Otherwise
A
DevOps Engineer at Devtron Inc. managing 100+ multi-cloud Kubernetes clusters. I write about practical DevOps (IaC, GitOps, K8s) and AI Ops at the intersection of DevOps and AI.

You use ArgoCD. Your manifests live in Git. Every change is a PR. You feel secure.

But here's the question nobody in the GitOps world wants to answer: how do you know the image running in your cluster is the one your pipeline built?

Not the one someone pushed to the registry with a similar name. Not the one a compromised CI runner injected. Not the one that passed through a dependency confusion attack. The actual artifact, built from the actual commit, by the actual pipeline.

If you can't answer that, your GitOps pipeline is a well-organized lie.

This is the supply chain trust problem. And it's bigger than you think.


The Attack You're Not Defending Against

Most Kubernetes security talks start with "harden your RBAC" or "scan your images." Important, sure. But they miss the fundamental issue: you're trusting a chain of systems, and you're verifying none of them.

Here's what a typical GitOps flow actually trusts:

  1. GitHub/GitLab — that the commit author is who they claim to be
  2. CI runner — that it built the right code, with the right dependencies
  3. Container registry — that the image wasn't tampered with after push
  4. ArgoCD/Flux — that it's deploying the right image tag
  5. Kubernetes — that the image reference in the manifest is what actually runs

Each of these is a trust boundary. Each one can be compromised. And in most clusters, none of them are cryptographically verified.

The SolarWinds attack wasn't about breaking encryption. It was about inserting malicious code into a trusted build system and letting the existing trust chain deliver it. The code was signed. The updates were verified. Everything looked legitimate because the compromise happened inside the trust boundary.

Now imagine that happening to your container images.


Enter SLSA: A Framework for Provenance

SLSA (pronounced "salsa") stands for Supply chain Levels for Software Artifacts. It's a framework not a tool that defines levels of supply chain integrity.

The Levels (SLSA v1.0 Build Track)

Level 1 Provenance Exists: The build process generates a provenance document showing how the artifact was built. It establishes a baseline of claims, even if unsigned. This is where most teams are today.

Level 2 Hosted Build Platform: Provenance is automatically generated and cryptographically signed by a hosted build service (GitHub Actions, Google Cloud Build) rather than user-controlled scripts. The build platform attests to what it built. This prevents tampering after the build, but the build platform itself is trusted.

Level 3 Hardened Build Platform: The build environment is isolated, ephemeral, and hardened against cross-build contamination. Secrets are completely isolated from user-defined build steps, preventing insider tampering. The provenance format is standardized and the attestation can be independently verified. This is where Sigstore comes in.

Note: Level 4 no longer exists in the current SLSA v1.0 specification. The old hermetic build and two-party review requirements were moved to future tracks beyond the Build Track.

Why This Matters for GitOps

In a GitOps world, your Git repo is the source of truth. But the artifacts that get deployed container images, Helm charts, OCI artifacts live outside Git. SLSA gives you a way to bridge that gap: cryptographic proof that the artifact in the registry matches the code in the repo.

Without this, your GitOps pipeline is only as secure as your weakest trust boundary.


Sigstore: The Missing Piece

If SLSA is the framework, Sigstore is the implementation. It's a set of open-source tools for software signing and verification, and it solves the hardest problem in software signing: key management.

Traditional signing (GPG, PGP) requires you to:

  1. Generate a key pair
  2. Securely store the private key
  3. Distribute the public key
  4. Handle key rotation
  5. Deal with the inevitable "my key expired" incidents

Sigstore eliminates all of this with keyless signing using OIDC identities.

How Keyless Signing Works

  1. You authenticate with an OIDC provider (GitHub, Google, your corporate IdP)
  2. You generate an ephemeral key pair and sign the artifact with the private key
  3. Sigstore's Fulcio CA issues a short-lived certificate (valid for 10 minutes) binding that public key to your OIDC identity
  4. The signature and certificate are recorded in Rekor a transparent, append-only log
  5. Verification checks: the signature is valid, the certificate was issued by Fulcio, and the entry exists in Rekor

No long-lived keys to manage. No key distribution problem. The trust root is the OIDC provider + Fulcio + Rekor.

The Tools

  • cosign: Sign and verify container images and OCI artifacts
  • fulcio: The short-lived certificate authority
  • rekor: The transparency log (think Certificate Transparency, but for software artifacts)
  • policy-controller: Kubernetes admission controller that verifies signatures before allowing deployments

Putting It Together: Sigstore + ArgoCD in Practice

Here's what a verified GitOps pipeline looks like:

Step 1: Sign Images in CI

In your GitHub Actions workflow:

- name: Install cosign
  uses: sigstore/cosign-installer@v3

- name: Sign container image
  run: |
    cosign sign --yes \
      --annotations "repo=${{ github.repository }}" \
      --annotations "commit=${{ github.sha }}" \
      --annotations "run-id=${{ github.run_id }}" \
      ghcr.io/\({{ github.repository }}:\){{ github.sha }}

This signs the image using your GitHub Actions OIDC token (keyless signing is default in cosign v2+). No keys to manage. The signature includes the repo, commit SHA, and workflow run ID as annotations.

Step 2: Generate SLSA Provenance

Use GitHub's native attest action (recommended as of 2025+):

- name: Attest build provenance
  uses: actions/attest-build-provenance@v1
  with:
    subject-name: ghcr.io/${{ github.repository }}
    subject-digest: ${{ steps.build.outputs.digest }}
    push-to-registry: true

This generates a signed SLSA v1 provenance attestation that says: "This image with digest X was built from commit Y by workflow Z on runner W." The attestation is pushed to the registry alongside the image and recorded in Rekor.

Alternatively, for the SLSA GitHub Generator (still maintained):

- name: Generate SLSA provenance
  uses: slsa-framework/slsa-github-generator/.github/workflows/generator_container_slsa3.yml@v2.1.0
  with:
    image: ghcr.io/${{ github.repository }}
    digest: ${{ steps.build.outputs.digest }}
    registry-username: ${{ github.actor }}
  secrets:
    registry-password: ${{ secrets.GITHUB_TOKEN }}

Step 3: Verify at Admission Time

Install Sigstore's policy-controller in your cluster:

helm install sigstore-policy-controller sigstore/policy-controller \
  --namespace sigstore-system \
  --create-namespace \
  --set webhook.configData.issuer="https://token.actions.githubusercontent.com" \
  --set webhook.configData.subject="repo:your-org/your-repo:ref:refs/heads/main"

Now create a ClusterImagePolicy:

apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: verify-ci-images
spec:
  images:
    - glob: "ghcr.io/your-org/*"
  authorities:
    - keyless:
        url: "https://fulcio.sigstore.dev"
        identities:
          - issuer: "https://token.actions.githubusercontent.com"
            subject: "https://github.com/your-org/your-repo/.github/workflows/build.yml@refs/heads/main"
      attestations:
        - name: slsa-provenance
          predicateType: "https://slsa.dev/provenance/v1"
          policy:
            type: cue
            data: |
              predicateType: "https://slsa.dev/provenance/v1"
              predicate: {
                buildDefinition: {
                  buildType: "https://actions.github.io/buildtypes/workflow/v1"
                }
                runDetails: {
                  builder: {
                    id: "https://github.com/actions/runner"
                  }
                }
              }

What this policy enforces:

  1. The image must be signed via keyless signing
  2. The signing identity must be your specific GitHub Actions workflow
  3. The image must have an SLSA provenance attestation
  4. The provenance must show it was built by the official SLSA generator

If any of these checks fail, Kubernetes rejects the pod creation. Not a warning. Not an alert. A hard rejection.

Step 4: Verify Provenance Content

You can go further verify that the provenance matches the exact commit.

The Fast Way (GitHub CLI):

Since actions/attest-build-provenance pushes attestations directly to the GitHub API alongside your container registry, you can verify locally without parsing base64 payloads:

gh attestation verify oci://ghcr.io/your-org/your-repo:tag --owner your-org

This single command checks the signature, verifies the Rekor transparency log, and validates the certificate identity clean, readable summary.

The Deep Dive (Cosign):

If you need to programmatically inspect specific fields in an automated script, use cosign to fetch and decode the raw JSON predicate:

cosign verify-attestation \
  --type slsaprovenance \
  --certificate-identity https://github.com/your-org/your-repo/.github/workflows/build.yml@refs/heads/main \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/your-org/your-repo:abc123 | jq '.payload | @base64d | fromjson | .predicate'

Note: SLSA v1 moved the builder ID to predicate.runDetails.builder.id (was predicate.builder.id in v0.2). Adjust your jq path accordingly.

Output shows the exact source repo, commit, builder ID, and build timestamp. If someone pushes a malicious image with the same tag, it won't have valid provenance and your admission controller will reject it.


The Hard Parts Nobody Talks About

1. Keyless Signing Has a Trust Assumption

You're trusting:

  • GitHub's OIDC token issuance (that the workflow identity is real)
  • Fulcio (that it correctly validates OIDC tokens before issuing certs)
  • Rekor (that the transparency log is append-only and consistent)

This is a different trust model than "I hold my own private key." It's arguably better for most teams, but it's not zero-trust. If GitHub's OIDC provider is compromised, the whole chain breaks.

2. Provenance Is Only as Good as the Build

SLSA Level 2 provenance says "GitHub Actions built this." It doesn't say "GitHub Actions built this in a hermetic, reproducible environment." A compromised GitHub Actions runner can produce valid provenance for malicious code.

SLSA Level 3+ requires hardened build platforms. This is why Google's Cloud Build and GitHub's hosted runners matter they provide the platform-level attestation that the build environment wasn't tampered with.

3. Verification Is Only Useful If You Enforce It

You can generate all the provenance you want. If your cluster doesn't verify it at admission time, it's just metadata. The policy-controller (or Kyverno with Sigstore verification) is what turns provenance from a nice-to-have into a security control.

4. Rollback Gets Complicated

If you enforce signature verification, you need signed images for every version you might roll back to. If you're using mutable tags (like :latest and if you are, stop), you need to switch to immutable tags (commit SHAs or semantic versions) and ensure every deployed version is signed.


What This Looks Like End-to-End

Developer pushes code → GitHub Actions triggers
    ↓
Build container image
    ↓
cosign sign (keyless, OIDC identity = GitHub Actions)
    ↓
Generate SLSA provenance (actions/attest-build-provenance)
    ↓
Push image + signature + provenance to registry
    ↓
ArgoCD detects new image, updates manifest
    ↓
Kubernetes creates pod → admission webhook intercepts
    ↓
policy-controller verifies:
  ✓ Signature valid?
  ✓ Signed by expected workflow?
  ✓ SLSA provenance present?
  ✓ Provenance matches expected builder?
    ↓
All checks pass → pod runs
Any check fails → pod rejected, alert fired

At no point in this chain does anyone "trust" a system. Every step is cryptographically verified. The Git commit, the build process, the artifact, the deployment all linked by signatures and attestations.

This is what "GitOps security" actually means. Not just "we use Git." But "we can prove that what's running matches what's in Git, end to end."


Where the Industry Is Heading

  • SLSA v1.0 was released in April 2023, and adoption is accelerating. Projects like Sigstore itself, Terraform, and the SLSA GitHub Generator publish SLSA provenance today. Kubernetes achieved SLSA Level 3 compliance starting with v1.26 (December 2022) and now signs all binary release artifacts .sig and .cert files, SBOMs, and signed build provenance ship alongside every core Kubernetes release.
  • SBOM attestations are being layered on top of provenance. Not just "what built this" but "what's inside this." Tools like Syft can generate SBOMs that are signed and attached as attestations.
  • In-toto (the framework behind SLSA provenance) is being extended to cover more of the supply chain dependency resolution, test execution, deployment approval.
  • Sigstore is becoming the default. Docker Hub supports cosign signatures. GitHub Container Registry displays verification status. AWS ECR, Google Artifact Registry all adding native support.

The direction is clear: within 2-3 years, unsigned artifacts will be treated the way unencrypted traffic is today technically possible, but socially unacceptable.


Monday Morning Task

Don't try to secure your entire pipeline at once. Start here:

  1. Pick one service. Add cosign sign to your CI workflow. That's it. Just start signing.
  2. Install cosign locally and verify:
cosign verify \
  --certificate-identity https://github.com/your-org/your-repo/.github/workflows/build.yml@refs/heads/main \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  ghcr.io/your-org/your-service:tag
  1. Look at the Rekor entry: rekor-cli search --artifact <image-digest>
  2. Once signing is routine, add the admission controller. Enforce verification. Then add provenance.

One step at a time. But start.

More from this blog

P

Ping to Production

8 posts

Deep dives into Kubernetes orchestration, GitOps workflows, and Cloud-Native security. From scaling EKS clusters to building custom Go operators, I document my journey of automating the world, one YAML file at a time. Expect production-ready insights about DevOps, Agentic AI and SRE best practices.