Your PR check reviews the wrong thing.

Someone opens a PR that changes four lines in values.yaml. CI goes green — the chart lints, the YAML parses, the schema validates. Two reviewers approve. It merges, deploys, and forty minutes later checkout starts OOM-killing.

Nothing in that pipeline was broken. It answered the question it was asked: is this valid YAML? Nobody asked the question that mattered: what does this do to what's currently running?

If you would rather see that question answered on every pull request than build it: Runtimez's PR-time deployment check diffs the PR against the live cluster and returns one verdict. The rest of this post is what that check has to say to be worth reading.

The source diff is not the change

Reviewing raw git diff on a Helm chart is reviewing the wrong artifact. One line in values.yaml can move forty rendered objects; a template edit can move all of them. The reviewer sees four lines and approves four lines' worth of risk.

So render both sides and diff the output:

helm template release ./chart -f values.yaml > /tmp/after.yaml

git stash push --quiet
helm template release ./chart -f values.yaml > /tmp/before.yaml
git stash pop --quiet

dyff between /tmp/before.yaml /tmp/after.yaml

dyff is worth the install over plain diff here — it's YAML-aware, so key reordering doesn't show up as a change and you get a structural report instead of a wall of red. Plain diff on rendered Kubernetes YAML produces enough noise that people stop reading it, which is the actual failure mode.

Diff against the cluster, not against main

Rendering both sides of the PR tells you what the PR changes. It does not tell you what changes in production, because production is not what's on main.

Someone scaled a deployment during an incident and never backported it. An HPA has been sitting at 12 replicas for a month while the manifest says 3. A hotfix went out through a different path. Drift is the normal state of a cluster, not an exception, and a PR that "changes nothing" relative to main can change plenty relative to reality.

helm template release ./chart -f values.yaml \
  | kubectl diff --server-side -f -

--server-side matters. It sends the manifest to the API server, which applies defaulting, admission webhooks, and field ownership, and returns what would genuinely change. Client-side diff compares against your local idea of the object and reports differences that are pure defaulting noise.

Exit code 1 means there is a diff. That's information, not a failure — most PRs should have one.

The five regressions worth blocking on

Having a diff isn't a verdict. A verdict is a short list of things that, if present, mean don't merge. These five earn their place because each one has a direct path to a page:

Notice what these have in common: every one is a change that makes the manifest simpler. They pass review because a diff that removes a stanza reads as cleanup.

Make CI say it

You can catch a useful share of these against the rendered output with no cluster access at all. Missing memory limits, as a starting point:

missing=$(yq eval-all '
  select(.kind == "Deployment" or .kind == "StatefulSet")
  | select(.spec.template.spec.containers[].resources.limits.memory == null)
  | .kind + " " + (.metadata.namespace // "default") + "/" + .metadata.name
' /tmp/after.yaml)

if [ -n "$missing" ]; then
  echo "::error::workloads with no memory limit:"
  echo "$missing"
  exit 1
fi

Check that against your own yq — v4 handles multi-document streams differently from v3, and the invocation above assumes v4. Run it against a manifest you know is bad before you trust it in CI. A check that silently matches nothing is worse than no check, because it produces a green tick that means nothing.

The same shape extends to the other four: select the kind, assert the field, collect names, exit non-zero. It's not sophisticated. It doesn't need to be — the failure mode it prevents is somebody deleting three lines on a Friday.

What the verdict should actually look like

A PR comment that says diff detected, 47 lines gets collapsed and ignored. One that a reviewer reads answers four questions in the order they'd ask them:

1. What materially changed — by workload, not by line. "checkout-api: memory limit 2Gi → unset" beats forty lines of context.
2. Does it regress readiness — the five checks above, pass or fail, named.
3. What's the blast radius — is this workload internet-facing, how many replicas, does it have a PDB. The same signals that decide any remediation ordering.
4. Can we roll back — is the previous image tag still resolvable, is this change backward-compatible with data on disk.

Four short lines. The value isn't automation for its own sake — it's that the reviewer sees the behavioral consequence next to the textual change, at the moment they're deciding, instead of reconstructing it from memory or not at all.

Where this breaks down

• Server-side diff needs cluster credentials in CI. That's a real security decision, not a footnote. Scope a dedicated read-only ServiceAccount for it — the same shape we described for a read-only cluster agent — and never reuse a deploy credential for a diff step.
• Defaulting still produces noise. Server-side diff removes most of it, but managed fields, mutating webhooks, and injected sidecars will still show up. Expect to maintain an ignore list, and expect that list to be a place bugs hide.
• Rendering isn't running. None of this catches a value that is syntactically perfect and semantically wrong — a correct-looking connection string pointing at the wrong database renders clean and fails in production.
• Multi-cluster gets awkward fast. One PR usually targets several environments with different values files. Diffing against "the cluster" means picking one, and the one you pick is rarely the one that breaks.

Frequently asked questions

How do I see what a Helm values change actually does?

Render the chart on both sides of the PR and diff the output, not the source. Run helm template on the branch and on the stashed base, then compare the two files with dyff between. dyff is YAML-aware, so key reordering is not reported as a change and the report stays short enough that people read it.

How do I diff a Helm chart against the live cluster?

Pipe the rendered chart into a server-side diff: helm template release ./chart -f values.yaml | kubectl diff --server-side -f -. The API server applies defaulting, admission webhooks and field ownership before reporting what would change, which removes most client-side noise. Exit code 1 means a diff exists, which is information rather than a failure.

Which Kubernetes manifest changes should block a merge?

Five: a memory limit removed or raised, replicas dropped to 1, a probe removed or its timeout raised, a PodDisruptionBudget removed or set to minAvailable: 0, and an image moved to a floating tag such as :latest. Each makes the manifest simpler, so it reads as cleanup in review, and each has a direct path to a page.

Can CI catch a missing memory limit without cluster access?

Yes. Run a yq v4 eval-all query over the rendered manifest that selects every Deployment and StatefulSet whose containers have no resources.limits.memory, and exit non-zero if any name comes back. Test it against a manifest you know is bad first, because a check that silently matches nothing produces a green tick that means nothing.

What should a PR-time deployment verdict contain?

Four short answers: what materially changed, by workload rather than by line; whether it regresses readiness on the five blocking checks; the blast radius, meaning internet exposure, replica count and PDB; and whether you can roll back. The reviewer then sees the behavioural consequence next to the textual change at the moment they decide.