← All posts

Your PR check reviews the wrong thing.

A manifest diff shows you text. The review question is behavioral: what does this change do to something currently serving traffic? Those are different questions, and CI usually answers only the first one.

3 August 2026· 8 min read· Runtimez
kubernetes cicd devops sre

Someone opens a PR that changes four lines in values.yaml. CI goes green — the chart lints, the YAML parses, the schema validates. Two reviewers approve. It merges, deploys, and forty minutes later checkout starts OOM-killing.

Nothing in that pipeline was broken. It answered the question it was asked: is this valid YAML? Nobody asked the question that mattered: what does this do to what's currently running?

The source diff is not the change

Reviewing raw git diff on a Helm chart is reviewing the wrong artifact. One line in values.yaml can move forty rendered objects; a template edit can move all of them. The reviewer sees four lines and approves four lines' worth of risk.

So render both sides and diff the output:

Diff what actually gets applied
helm template release ./chart -f values.yaml > /tmp/after.yaml

git stash push --quiet
helm template release ./chart -f values.yaml > /tmp/before.yaml
git stash pop --quiet

dyff between /tmp/before.yaml /tmp/after.yaml

dyff is worth the install over plain diff here — it's YAML-aware, so key reordering doesn't show up as a change and you get a structural report instead of a wall of red. Plain diff on rendered Kubernetes YAML produces enough noise that people stop reading it, which is the actual failure mode.

Diff against the cluster, not against main

Rendering both sides of the PR tells you what the PR changes. It does not tell you what changes in production, because production is not what's on main.

Someone scaled a deployment during an incident and never backported it. An HPA has been sitting at 12 replicas for a month while the manifest says 3. A hotfix went out through a different path. Drift is the normal state of a cluster, not an exception, and a PR that "changes nothing" relative to main can change plenty relative to reality.

Diff the rendered output against live state
helm template release ./chart -f values.yaml \
  | kubectl diff --server-side -f -

--server-side matters. It sends the manifest to the API server, which applies defaulting, admission webhooks, and field ownership, and returns what would genuinely change. Client-side diff compares against your local idea of the object and reports differences that are pure defaulting noise.

Exit code 1 means there is a diff. That's information, not a failure — most PRs should have one.

The five regressions worth blocking on

Having a diff isn't a verdict. A verdict is a short list of things that, if present, mean don't merge. These five earn their place because each one has a direct path to a page:

RegressionWhat it does in production
Memory limit removed or raisedThe workload becomes best-effort or burstable and gets OOM-killed under load, or evicts its neighbours. The single most common version of the incident above.
Replicas dropped to 1Any node drain — including the ones your cloud provider does without asking — is now a full outage for that service.
Probe removed or timeout raisedTraffic routes to pods that aren't ready, or a wedged pod never gets restarted. Both present as intermittent 5xx nobody can reproduce.
PDB removed or minAvailable: 0Removes the guardrail that makes cluster upgrades safe. Nothing breaks today; everything breaks during the next node rotation.
Image moved to a floating tag:latest means the running image is now whatever the registry last pushed. You lose reproducibility and the ability to say what's deployed.

Notice what these have in common: every one is a change that makes the manifest simpler. They pass review because a diff that removes a stanza reads as cleanup.

Make CI say it

You can catch a useful share of these against the rendered output with no cluster access at all. Missing memory limits, as a starting point:

Fail the build on a workload with no memory limit
missing=$(yq eval-all '
  select(.kind == "Deployment" or .kind == "StatefulSet")
  | select(.spec.template.spec.containers[].resources.limits.memory == null)
  | .kind + " " + (.metadata.namespace // "default") + "/" + .metadata.name
' /tmp/after.yaml)

if [ -n "$missing" ]; then
  echo "::error::workloads with no memory limit:"
  echo "$missing"
  exit 1
fi

Check that against your own yq — v4 handles multi-document streams differently from v3, and the invocation above assumes v4. Run it against a manifest you know is bad before you trust it in CI. A check that silently matches nothing is worse than no check, because it produces a green tick that means nothing.

The same shape extends to the other four: select the kind, assert the field, collect names, exit non-zero. It's not sophisticated. It doesn't need to be — the failure mode it prevents is somebody deleting three lines on a Friday.

What the verdict should actually look like

A PR comment that says diff detected, 47 lines gets collapsed and ignored. One that a reviewer reads answers four questions in the order they'd ask them:

  1. What materially changed — by workload, not by line. "checkout-api: memory limit 2Gi → unset" beats forty lines of context.
  2. Does it regress readiness — the five checks above, pass or fail, named.
  3. What's the blast radius — is this workload internet-facing, how many replicas, does it have a PDB. The same signals that decide any remediation ordering.
  4. Can we roll back — is the previous image tag still resolvable, is this change backward-compatible with data on disk.

Four short lines. The value isn't automation for its own sake — it's that the reviewer sees the behavioral consequence next to the textual change, at the moment they're deciding, instead of reconstructing it from memory or not at all.

Where this breaks down

Disclosure: we build Runtimez, which posts this verdict onto pull requests automatically — rendered diff against live state, the readiness regressions, cost delta, and blast radius. The helm template and kubectl diff --server-side pipeline above is the same idea assembled by hand, and it's a genuinely good afternoon's work if you'd rather own it yourself.

See the intersection on your own clusters

One read-only Helm install · first report in under an hour · secrets and images never leave your cluster.

Questions? hello@runtimez.io