Skip to content
SARC

Pipeline showcase

This is the architect’s single-page view of the SARC pipeline: what each stage and action does, why it is built the way it is, and how the three control planes — CI/CD, the Fides evidence vault, and ServiceNow — exchange data as a change moves from commit to production.

A normal pipeline builds and ships software and throws its scan output away. SARC does two things at once: it builds and ships, and it produces tamper-evident evidence and a real change record an auditor can later query. The one-line summary:

Fides advises, ServiceNow decides, the pipeline enforces — every step writes immutable, hash-chained evidence keyed to a specific artifact digest and a specific change request.

Supply-chain maturity. Builds run on a hosted runner (not a workstation) and emit signed keyless provenance whose signature is recorded in the public Rekor transparency log, plus a SLSA in-toto provenance attestation — placing SARC at SLSA Build Level 2, reaching toward L3. GitLab CI is the source of truth; GitHub Actions and Azure DevOps run the same shape.

The pipeline, end to end

Section titled “The pipeline, end to end”

Thirteen stages. The diagram is interactive — use the on-canvas controls, the mouse wheel, or click-and-drag to zoom into any stage. Solid arrows are stage progression (a stage starts only when the previous one succeeds). Dotted arrows are the evidence and change-record side-channels the pipeline fills as it runs; Fides and ServiceNow are data planes, not stages.

%%{init: {'flowchart': {'useMaxWidth': false}}}%%
flowchart TD
  START(["Trigger<br/>push to main / merge request / schedule"]) --> V

  subgraph V["1. validate — fail fast, fail cheap"]
    direction TB
    V1["helm lint + template · prisma drift · route exports"]
    V2["cold-install smoke on a real kind cluster"]
    V3["portal unit tests (vitest)"]
  end

  subgraph B["2. build — produce immutable artifacts"]
    direction TB
    B1["docker buildx: portal + 6 podtato + mcp<br/>capture OCI digest (provenance foundation)"]
  end

  subgraph SEC["3. security — find problems, generate evidence"]
    direction TB
    SC1["SAST (semgrep) · secrets (gitleaks) · SCA (gemnasium)"]
    SC2["container CVEs x2 (Trivy + Grype) · IaC (KICS + checkov)"]
    SC3["SBOM (syft, CycloneDX): every image + every chart"]
  end

  subgraph AT["4. attest — turn output into evidence + a change record"]
    direction TB
    A0["fides begin trail (keyed by commit SHA)"]
    A1["attest artifact / security / SBOM to the Fides trail"]
    A2["control-coverage map: each scan -> a named control"]
    A3["cosign keyless sign -> Rekor log + SLSA provenance"]
    A4["ServiceNow CR create (risk-scored) + enrich + sync"]
  end

  subgraph DEV["5. deploy-dev — first running environment"]
    direction TB
    D1["helm upgrade karc-portal (no approval)"]
    D2["deployment attestation (dev)"]
  end

  subgraph DAST["6. dast — dynamic security"]
    direction TB
    Z1["OWASP ZAP against the live deploy (URL from the LB)"]
  end

  subgraph QA["7. deploy-qa — earned promotion"]
    direction TB
    Q1{"Fides change-gate<br/>approve/hold + 0-100 risk"}
    Q2{"ServiceNow CR approval (human)"}
    Q3["deploy-gate + deployment attestation (qa)"]
  end

  subgraph PRD["8. deploy-prod — humans own production"]
    direction TB
    P1{"Fides change-gate (prod)"}
    P2{"CAB approval — manual, always"}
    P3["deploy-gate + approval/deploy attestation (prod)"]
  end

  subgraph POST["9. post-deploy"]
    PO1["ServiceNow CR close (Implemented -> Closed)"]
  end

  subgraph GO["10. gitops"]
    G1["bump image tags / promote chart -> ArgoCD branch"]
  end

  subgraph REL["11. release (v* tags)"]
    R1["OCI chart publish · cosign-sign digests + chart<br/>SBOM as in-toto attestation · GitLab Release"]
  end

  subgraph MIR["12. mirror (scheduled)"]
    M1["push main -> GitHub + Azure DevOps"]
  end

  subgraph PG["13. pages"]
    PGS["build + publish this showcase site"]
  end

  V --> B --> SEC --> AT --> DEV --> DAST --> QA --> PRD --> POST --> GO --> REL --> MIR --> PG

  FIDES[("Fides<br/>evidence vault")]
  SNOW[("ServiceNow<br/>change workflow")]
  A1 -.evidence.-> FIDES
  A3 -.provenance.-> FIDES
  D2 -.evidence.-> FIDES
  Q1 -.reads verdict.-> FIDES
  Q3 -.evidence.-> FIDES
  P3 -.evidence.-> FIDES
  A4 -.change record.-> SNOW
  Q2 -.approval.-> SNOW
  P2 -.approval.-> SNOW
  PO1 -.close.-> SNOW

  class START trig
  class V1,V2,V3 validate
  class B1 build
  class SC1,SC2,SC3 security
  class A0,A1,A2,A3,A4 attest
  class D1,D2 dev
  class Z1 dast
  class Q3 qa
  class P3 prod
  class Q1,Q2,P1,P2 gate
  class PO1 post
  class G1 gitops
  class R1 release
  class M1 mirror
  class PGS pages
  class FIDES fides
  class SNOW snow

  classDef trig fill:#57606a,stroke:#32383f,color:#ffffff
  classDef validate fill:#1f6feb,stroke:#0b3d91,color:#ffffff
  classDef build fill:#6639ba,stroke:#3b1f7a,color:#ffffff
  classDef security fill:#cf222e,stroke:#82071e,color:#ffffff
  classDef attest fill:#2da44e,stroke:#116329,color:#ffffff
  classDef dev fill:#1b7c83,stroke:#0f4a4e,color:#ffffff
  classDef dast fill:#e8590c,stroke:#9a3b08,color:#ffffff
  classDef qa fill:#d29922,stroke:#8a6500,color:#ffffff
  classDef prod fill:#b62324,stroke:#6e0f10,color:#ffffff
  classDef gate fill:#9a6700,stroke:#5c3d00,color:#ffffff
  classDef post fill:#1f8f6b,stroke:#0d5a42,color:#ffffff
  classDef gitops fill:#8957e5,stroke:#512a8f,color:#ffffff
  classDef release fill:#0969da,stroke:#0a3069,color:#ffffff
  classDef mirror fill:#768390,stroke:#444c56,color:#ffffff
  classDef pages fill:#57606a,stroke:#32383f,color:#ffffff
  classDef fides fill:#2da44e,stroke:#116329,color:#ffffff
  classDef snow fill:#d29922,stroke:#8a6500,color:#ffffff

For the deep, per-job walkthrough of any stage, see the full pipeline action map; for the control mapping (each job to SLSA / NIST SSDF / SOC 2 / ISO 27001) see docs/ci/PIPELINE-CONTROL-MAP.md in the repository.

Stages and the actions they run

Section titled “Stages and the actions they run”
StageWhat it provesActions / toolsGate or advisory
validateSource + config integrityhelm lint/template, prisma migrate diff, route-export scan, kind cold-install smoke, vitestGate
buildReproducible artifacts, pinned by digestdocker buildx (portal + 6 podtato + mcp), OCI digest captureGate
securityVulnerabilities + policy violations surfaced as evidencesemgrep, gitleaks, gemnasium, Trivy + Grype, KICS + checkov, kubeconform, syft SBOMAdvisory
attestOutput becomes immutable evidence + a change recordFides REST attestations, control-coverage map, cosign keyless + Rekor + SLSA, ServiceNow CR create/enrich/syncAdvisory + CR create
deploy-devFirst running environment (no approval)helm upgrade (karc-portal), deployment attestationGate
dastRuntime behaviour, not just static codeOWASP ZAP baseline against the live deployAdvisory
deploy-qaPromotion is earnedFides change-gate, ServiceNow approval poll, helm deploy-gateGate
deploy-prodHumans own productionFides change-gate, CAB manual approval, helm deploy-gateGate (manual)
post-deployThe change lifecycle closesServiceNow CR close with outcomeAdvisory
gitopsDesired state lives in gitimage-tag bump / chart promote on the ArgoCD branchAdvisory
releaseSigned, versioned artifactsOCI chart publish, cosign-sign, SBOM in-toto attestation, GitLab ReleaseGate
mirrorDR replicas stay in syncforce-push to GitHub + Azure DevOps (scheduled)Advisory
pagesDocs ship from the same pipelinebuild + publish this siteAdvisory

“Advisory” means the job records evidence and surfaces findings on the Fides trail and the GitLab Vulnerability Dashboard but does not fail CI on its own — enforcement happens at the aggregated Fides policy / change-gate layer, not on a single scanner’s exit code.

Design decisions — what, why, how

Section titled “Design decisions — what, why, how”

The interesting choices, and the reasoning a reviewer will ask about.

1. Pin every artifact by its digest at build time

Section titled “1. Pin every artifact by its digest at build time”
  • What: the build captures the immutable OCI digest and everything downstream (scan, SBOM, signature, deploy) references the image by image@sha256:..., not by tag.
  • Why: it guarantees the bytes you scanned, signed, and attested are exactly the bytes that run in production — the foundation of SLSA provenance. Tags can be moved; digests cannot.
  • How: docker buildx --metadata-file writes the digest into a dotenv artifact that the scan, SBOM, and Fides-attest jobs import.

2. Disable buildx’s own provenance/SBOM attestations

Section titled “2. Disable buildx’s own provenance/SBOM attestations”
  • What: images are built with provenance=false sbom=false.
  • Why: buildx attestations change the pushed manifest digest so it no longer equals the running pod’s imageID — which would break Fides attestation keying (evidence keyed to a digest the cluster never reports). SARC generates SBOM (syft) and provenance (cosign + SLSA) itself, on the exact digest.
  • How: the flags are set on every build job; syft/cosign run separately.

3. podtato-head is deployed only by ArgoCD; karc-portal only by CI helm

Section titled “3. podtato-head is deployed only by ArgoCD; karc-portal only by CI helm”
  • What: the demo app (podtato-head) is GitOps-owned; the dashboard (karc-portal) is a dev-only helm singleton.
  • Why: when both CI helm and ArgoCD applied the same Deployment they fought for server-side-apply ownership of the pod template, forcing churn. Splitting the deploy path removes the conflict.
  • How: CI writes image-tag files on the gitops branch; ArgoCD ApplicationSets reconcile podtato across dev/qa/prod per cloud.

4. No terraform in CI — infrastructure is applied out of band

Section titled “4. No terraform in CI — infrastructure is applied out of band”
  • What: the pipeline never runs terraform apply or terraform validate.
  • Why: the demo clusters pre-exist; infra changes are rare and roll nodes, so running them inside a per-push pipeline adds risk and noise for no benefit.
  • How: ops applies infra/{aws,azure,gcp} in a maintenance window; the pipeline consumes the cluster via OIDC.

5. Keyless signing over long-lived keys

Section titled “5. Keyless signing over long-lived keys”
  • What: cosign signs images and the released chart keylessly.
  • Why: no signing key ever lives in CI to leak or rotate, and the signature is recorded in the public Rekor transparency log — non-repudiable proof of who built what, from where.
  • How: the CI job mints a short-lived OIDC token, Fulcio issues an ephemeral certificate, cosign signs by digest, and the entry lands in Rekor.

6. Fides advises, ServiceNow decides

Section titled “6. Fides advises, ServiceNow decides”
  • What: Fides computes the compliance verdict + risk score; ServiceNow holds the change request and the human approval.
  • Why: it separates the evidence/decision engine from the human change-authority and the segregation-of-duties record — the split auditors expect. Neither system is a single point of both truth and authority.
  • How: the change-gate reads the Fides trail and can write its verdict onto the ServiceNow CR natively; the CR approval poll is the human gate.

7. The change-gate is advisory per-push, hard at promotion

Section titled “7. The change-gate is advisory per-push, hard at promotion”
  • What: on the GitLab per-push pipeline a Fides “hold” is advisory — it records the verdict + risk on the trail as evidence and passes green. The hard block happens at qa/prod promotion time via scripts/ci/promote.sh, which rejects a non-compliant promotion and performs the human SoD sign-off. GitHub’s on-demand orchestrators can still hard-gate.
  • Why: GitLab main runs on every push, and the hard gate needs a human approver (segregation of duties) that CI — a service account — cannot provide; a hard per-push gate would block every routine deploy. Enforcement therefore lives where a human is actually in the loop: the promotion. The verdict + risk score are recorded either way.
  • How: a single CI variable flips GitLab from advisory to hard, matching the GitHub behaviour once trails are green.

8. Scanners are advisory; enforcement is at the policy layer

Section titled “8. Scanners are advisory; enforcement is at the policy layer”
  • What: SAST/SCA/container/IaC/DAST jobs do not fail CI on findings.
  • Why: failing every pipeline on every noisy or accepted finding trains people to ignore red. Instead each finding becomes evidence, and enforcement happens where risk is aggregated — the Fides change-gate and the GitLab Vulnerability Dashboard.
  • How: scanner jobs are allow_failure; their results are attested to the trail and mapped to named controls.

9. Two container scanners, not one

Section titled “9. Two container scanners, not one”
  • What: every image is scanned by both Trivy and Grype.
  • Why: they use different vulnerability databases and matching logic, so together they catch more real CVEs than either alone.
  • How: parallel jobs per image, each attested separately.

10. One self-hosted runner, concurrency-capped

Section titled “10. One self-hosted runner, concurrency-capped”
  • What: a single in-cluster GitLab runner runs every job, with a concurrency cap.
  • Why: an in-cluster runner gets OIDC + cluster access without long-lived credentials; the cap bounds how much node disk the parallel image builds consume (the node root volume is being raised to 100 GiB to lift the cap).
  • How: the runner is Helm-managed from apps/gitlab-runner/.

11. Selective pipelines and one multi-cloud shape

Section titled “11. Selective pipelines and one multi-cloud shape”
  • What: docs-only commits skip the demo compliance DAG; a single env var targets any cloud.
  • Why: the full run is ~90-190 jobs — running it on a docs typo wastes runner time; and one pipeline shape across AWS/Azure/GCP/k3d keeps the compliance story identical everywhere.
  • How: rules:changes path filters gate the app jobs; TARGET_CLOUD selects the kubectl context, the helm values overlay, and the Fides environment name.

Data flow: CI/CD, Fides, and ServiceNow

Section titled “Data flow: CI/CD, Fides, and ServiceNow”

The three control planes exchange data in a fixed order as a change moves from commit to production. Fides is the evidence plane (writes during build/attest, reads at the gate); ServiceNow is the workflow plane (the change record, the human approval, the closure).

sequenceDiagram
  autonumber
  actor Dev as Developer
  participant CI as CI/CD (GitLab)
  participant F as Fides (evidence)
  participant SN as ServiceNow (workflow)
  participant K as Cluster (Helm / ArgoCD)

  Dev->>CI: push to main / merge request
  Note over CI: validate -> build (digest pinned) -> scan

  CI->>F: begin trail (keyed by commit SHA)
  CI->>F: attest artifact, security, SBOM
  CI->>F: control-coverage map (scan -> named control)
  CI->>F: cosign keyless sign -> Rekor + SLSA provenance

  CI->>SN: create Change Request (risk-scored)
  CI->>SN: attach compliance report + SBOM bundle
  CI->>SN: sync Fides evidence -> u_fides_* tables

  Note over CI,K: deploy-dev (no approval)
  CI->>K: helm upgrade karc-portal (dev)
  CI->>F: attest deployment (dev)
  Note over CI: DAST (ZAP) against the live dev deploy

  Note over CI,SN: QA gate — Fides advises, ServiceNow decides
  CI->>F: change-gate: request verdict
  F-->>CI: recommendation (approve / hold) + 0-100 risk
  F->>SN: write verdict + risk onto the CR (native)
  CI->>SN: poll CR for human approval
  SN-->>CI: approved / rejected
  CI->>K: deploy-gate (qa) — podtato via ArgoCD
  CI->>F: attest deployment (qa)

  Note over CI,SN: prod gate — change-gate + CAB manual approval
  CI->>F: change-gate (prod)
  F-->>CI: verdict + risk
  CI->>SN: poll CR for CAB approval (manual, always)
  SN-->>CI: approved
  CI->>K: deploy-gate (prod)
  CI->>F: attest deployment + approval (prod)

  CI->>SN: close CR (Implemented -> Closed) with outcome

Reading it. During build + attest the pipeline writes evidence to the Fides trail (artifacts, scans, SBOMs, control coverage, signatures) and opens + enriches the ServiceNow change request. At the QA and prod gates it reads back: the Fides change-gate returns an approve/hold verdict and a 0-100 risk score computed from the trail, Fides writes that verdict onto the CR, and the pipeline blocks on the ServiceNow human approval. After the deploy it writes the deployment attestation to Fides and closes the CR in ServiceNow. Every arrow is a real job; the trail and the CR are the durable records an auditor queries later.

Why this shape

Section titled “Why this shape”
  • Evidence is written continuously, read once at the gate. Collecting evidence as a side effect of the work (not as a separate audit project) means the gate decision is made from what actually happened, not from a form someone filled in.
  • The risk score is computed server-side by Fides, from failed + missing controls and non-compliant attestations on the trail — so the number a CAB member sees on the CR is derived from evidence, not asserted by the pipeline.
  • ServiceNow remains the system of record for the change. Fides advises and writes its verdict onto the CR, but a human in ServiceNow decides, and the closed CR is the auditable artifact — satisfying change-management and segregation-of-duties controls without the pipeline granting itself approval authority.

Cross-platform parity

Section titled “Cross-platform parity”

The same shape runs on GitHub Actions (.github/workflows/main.yml + _reusable-*.yml) and Azure DevOps, producing the identical attestation shape on the Fides trail. The one enforcement difference is the change-gate default (hard on GitHub, advisory-toggle on GitLab, as above). See Multi-CI parity and Multi-cloud.

Where to go next

Section titled “Where to go next”