Someone opens a PR that changes four lines in values.yaml. CI goes green — the chart lints, the YAML parses, the schema validates. Two reviewers approve. It merges, deploys, and forty minutes later checkout starts OOM-killing.
Nothing in that pipeline was broken. It answered the question it was asked: is this valid YAML? Nobody asked the question that mattered: what does this do to what's currently running?
The source diff is not the change
Reviewing raw git diff on a Helm chart is reviewing the wrong artifact. One line in values.yaml can move forty rendered objects; a template edit can move all of them. The reviewer sees four lines and approves four lines' worth of risk.
So render both sides and diff the output:
Diff what actually gets appliedhelm template release ./chart -f values.yaml > /tmp/after.yaml
git stash push --quiet
helm template release ./chart -f values.yaml > /tmp/before.yaml
git stash pop --quiet
dyff between /tmp/before.yaml /tmp/after.yaml
dyff is worth the install over plain diff here — it's YAML-aware, so key reordering doesn't show up as a change and you get a structural report instead of a wall of red. Plain diff on rendered Kubernetes YAML produces enough noise that people stop reading it, which is the actual failure mode.
Diff against the cluster, not against main
Rendering both sides of the PR tells you what the PR changes. It does not tell you what changes in production, because production is not what's on main.
Someone scaled a deployment during an incident and never backported it. An HPA has been sitting at 12 replicas for a month while the manifest says 3. A hotfix went out through a different path. Drift is the normal state of a cluster, not an exception, and a PR that "changes nothing" relative to main can change plenty relative to reality.
Diff the rendered output against live statehelm template release ./chart -f values.yaml \
| kubectl diff --server-side -f -
--server-side matters. It sends the manifest to the API server, which applies defaulting, admission webhooks, and field ownership, and returns what would genuinely change. Client-side diff compares against your local idea of the object and reports differences that are pure defaulting noise.
Exit code 1 means there is a diff. That's information, not a failure — most PRs should have one.
The five regressions worth blocking on
Having a diff isn't a verdict. A verdict is a short list of things that, if present, mean don't merge. These five earn their place because each one has a direct path to a page:
| Regression | What it does in production |
|---|---|
| Memory limit removed or raised | The workload becomes best-effort or burstable and gets OOM-killed under load, or evicts its neighbours. The single most common version of the incident above. |
| Replicas dropped to 1 | Any node drain — including the ones your cloud provider does without asking — is now a full outage for that service. |
| Probe removed or timeout raised | Traffic routes to pods that aren't ready, or a wedged pod never gets restarted. Both present as intermittent 5xx nobody can reproduce. |
PDB removed or minAvailable: 0 | Removes the guardrail that makes cluster upgrades safe. Nothing breaks today; everything breaks during the next node rotation. |
| Image moved to a floating tag | :latest means the running image is now whatever the registry last pushed. You lose reproducibility and the ability to say what's deployed. |
Notice what these have in common: every one is a change that makes the manifest simpler. They pass review because a diff that removes a stanza reads as cleanup.
Make CI say it
You can catch a useful share of these against the rendered output with no cluster access at all. Missing memory limits, as a starting point:
Fail the build on a workload with no memory limitmissing=$(yq eval-all '
select(.kind == "Deployment" or .kind == "StatefulSet")
| select(.spec.template.spec.containers[].resources.limits.memory == null)
| .kind + " " + (.metadata.namespace // "default") + "/" + .metadata.name
' /tmp/after.yaml)
if [ -n "$missing" ]; then
echo "::error::workloads with no memory limit:"
echo "$missing"
exit 1
fi
Check that against your own yq — v4 handles multi-document streams differently from v3, and the invocation above assumes v4. Run it against a manifest you know is bad before you trust it in CI. A check that silently matches nothing is worse than no check, because it produces a green tick that means nothing.
The same shape extends to the other four: select the kind, assert the field, collect names, exit non-zero. It's not sophisticated. It doesn't need to be — the failure mode it prevents is somebody deleting three lines on a Friday.
What the verdict should actually look like
A PR comment that says diff detected, 47 lines gets collapsed and ignored. One that a reviewer reads answers four questions in the order they'd ask them:
- What materially changed — by workload, not by line. "checkout-api: memory limit 2Gi → unset" beats forty lines of context.
- Does it regress readiness — the five checks above, pass or fail, named.
- What's the blast radius — is this workload internet-facing, how many replicas, does it have a PDB. The same signals that decide any remediation ordering.
- Can we roll back — is the previous image tag still resolvable, is this change backward-compatible with data on disk.
Four short lines. The value isn't automation for its own sake — it's that the reviewer sees the behavioral consequence next to the textual change, at the moment they're deciding, instead of reconstructing it from memory or not at all.
Where this breaks down
- Server-side diff needs cluster credentials in CI. That's a real security decision, not a footnote. Scope a dedicated read-only ServiceAccount for it — the same shape we described for a read-only cluster agent — and never reuse a deploy credential for a diff step.
- Defaulting still produces noise. Server-side diff removes most of it, but managed fields, mutating webhooks, and injected sidecars will still show up. Expect to maintain an ignore list, and expect that list to be a place bugs hide.
- Rendering isn't running. None of this catches a value that is syntactically perfect and semantically wrong — a correct-looking connection string pointing at the wrong database renders clean and fails in production.
- Multi-cluster gets awkward fast. One PR usually targets several environments with different values files. Diffing against "the cluster" means picking one, and the one you pick is rarely the one that breaks.
Disclosure: we build Runtimez, which posts this verdict onto pull requests automatically — rendered diff against live state, the readiness regressions, cost delta, and blast radius. The helm template and kubectl diff --server-side pipeline above is the same idea assembled by hand, and it's a genuinely good afternoon's work if you'd rather own it yourself.
See the intersection on your own clusters
One read-only Helm install · first report in under an hour · secrets and images never leave your cluster.
Questions? hello@runtimez.io