Post-Deploy Sanity Checks: Verify Production Without Re-Running Your Test Suite

A deployment pipeline can finish successfully while production is still broken.

The artifact may have uploaded correctly, the platform may have accepted the release, and every CI test may be green. None of those facts proves that users can reach the new version, that the application can talk to its dependencies, or that the expected configuration reached production.

That is the purpose of a post-deploy sanity check: a small, fast verification performed against the live environment immediately after deployment.

The useful version is deliberately narrow. Confirm the deployed identity, exercise a critical path, verify an essential dependency, inspect immediate health signals, and make the rollback decision obvious. Do not re-run the entire test suite in production.

Sanity Checks Answer a Different Question Than CI

Pre-deploy CI asks whether a change is safe enough to release. A post-deploy check asks whether the intended release became healthy in the real production environment.

A unit test can prove a parser handles a response correctly. It cannot prove production received the right API endpoint. An integration test can prove a migration works in a controlled environment. It cannot prove that migration actually ran in production. A build can prove an asset exists. It cannot prove the CDN is serving the new asset instead of a stale version.

GitHub Actions environments create deployment records and can apply protection rules around production jobs. Google Cloud Deploy separately models deployment verification inside a rollout. The durable principle is the same: deployment and verification are different states.

If the release process still needs stronger promotion and rollback boundaries, start with the RayLabs guide to executing a smooth technical rollout before adding more checks.

Keep the Check Small Enough to Trust

A useful sanity check should finish in seconds or a few minutes. The goal is high-signal evidence that the release crossed the production boundary correctly.

A practical baseline has five checks:

Check	What it proves
Release identity	Production is serving the expected commit, image, or version
Critical entry point	The application is reachable through the real public route
Core transaction	A representative user path can complete
Essential dependency	A database, API, queue, or storage dependency is usable
Health signal	Error rate, latency, or service health is not immediately abnormal

Each check should correspond to a production failure you would actually act on. A static site may only need the final commit, public routes, asset delivery, and headers. A transactional backend may need a read path, a safe synthetic write, dependency connectivity, and queue health.

Verify the Release Identity First

Before testing behavior, prove that you are testing the release you think you deployed.

Expose a non-sensitive build identifier through a health endpoint, response header, version file, or deployment metadata such as:

{
  "status": "ok",
  "version": "2026.09.24.1",
  "commit": "4f2a9c1"
}

Then compare the live identifier with the artifact that was just promoted:

expected_sha="$GITHUB_SHA"
live_sha="$(curl --fail --silent https://example.com/version.json | jq -r .commit)"

test "${expected_sha:0:7}" = "$live_sha"

This catches successful workflows that updated the wrong environment, platforms still serving the previous release, stale edge caches, and promotion jobs that referenced the wrong artifact.

Do not expose secrets, internal hostnames, environment variables, or detailed dependency versions. The identifier only needs enough information to bind verification to the deployed artifact.

Test the Real Route

A health endpoint is useful, but it can lie by omission.

If users reach a service through DNS, TLS termination, a CDN, an edge worker, ingress, and then the application, checking only localhost inside the container proves very little about the public path.

At least one check should use the same public hostname users use:

curl \
  --fail \
  --silent \
  --show-error \
  --max-time 10 \
  https://example.com/

For an API, verify a safe endpoint through the production gateway rather than calling the service directly by its private address.

This follows the same evidence-first principle used in reliable CI debugging: verify the layer that owns the failure. The RayLabs guide to cleaning build artifacts safely in CI applies that principle to workspace state and reproducibility.

Choose One Core Transaction Carefully

A homepage returning 200 is not enough for many applications. The strongest check usually performs one representative transaction that crosses important runtime boundaries.

Useful examples include loading a catalog item backed by a database, authenticating a dedicated synthetic account and fetching its profile, submitting a harmless test event to a queue, or rendering a public page and fetching its referenced asset.

The transaction should be safe, idempotent, and easy to identify. Never let a verification script create real orders, send real notifications, charge a payment method, or mutate customer data.

If a production write is too risky, choose a read path that still exercises meaningful dependencies. A weaker safe check is better than a powerful synthetic transaction that can cause damage during retries.

Classify Failures Before Deciding to Roll Back

These incidents should not produce the same response:

A. deployment failed before production changed
B. deployment completed, but release identity is wrong
C. release identity is correct, but the critical transaction fails

Case A is a delivery problem. Production may still be healthy on the previous release.

Case B suggests a promotion, caching, routing, or artifact-selection problem.

Case C says the intended release is live but unhealthy. That is the strongest rollback signal.

Record enough evidence to preserve the distinction: expected release ID, observed release ID, checked URL, response status, duration, failed checkpoint, and timestamp. Avoid logging tokens or sensitive payloads.

Define rollback criteria before the release. For example, a team might roll back when the live identity matches the new release and the critical transaction repeatedly fails, while treating a verification runner network outage as a separate incident. Thresholds should come from the system’s own risk model, not copied percentages.

Do Not Turn Production Into a Test Environment

A common failure mode is expanding the sanity stage until it becomes a second end-to-end suite.

That makes releases slower, creates synthetic production data, increases flakiness, requires broader permissions, and makes failures harder to classify. Eventually engineers stop trusting the signal.

Keep exhaustive behavior testing before deployment. Keep the post-deploy layer focused on facts that can only be proven after promotion.

A useful test for every proposed check is:

Could this failure only become visible, or become materially different, in the real deployed environment?

If the answer is no, the check probably belongs earlier in CI.

A Minimal GitHub Actions Pattern

Keep promotion and verification visibly separate:

jobs:
  deploy:
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Deploy release
        run: ./scripts/deploy.sh "$GITHUB_SHA"

  verify:
    needs: deploy
    runs-on: ubuntu-latest
    steps:
      - name: Verify live release
        env:
          EXPECTED_SHA: ${{ github.sha }}
        run: ./scripts/verify-production.sh

The verification script should return a non-zero exit code when a required checkpoint fails and print a concise diagnostic message for each checkpoint.

GitHub environments can add reviewers, branch restrictions, environment-specific secrets, and protection rules before deployment. Those controls govern whether a release may deploy. The post-deploy check governs whether the deployed release is healthy enough to keep.

Monitor After the Immediate Check Passes

A fast sanity check catches immediate breakage. It does not replace observability.

Memory growth, queue backlog, slow database queries, cache churn, regional routing problems, or elevated errors on uncommon paths may need real traffic or time to appear. Treat the first successful verification as the start of a short observation window.

Watch the service’s normal operational signals after promotion: request errors, latency on critical routes, resource pressure, dependency failures, queue backlog where relevant, and client failure signals for mobile-backed services.

If staged rollout is available, a small verified cohort gives telemetry time to reveal problems before the release reaches everyone.

A Reusable Post-Deploy Checklist

Before calling a production release complete, verify:

1. The live environment reports the expected release identity.
2. The public route resolves and negotiates TLS successfully.
3. A critical read or safe synthetic transaction completes.
4. Essential dependencies respond through the application path.
5. Static assets or API payloads come from the expected release where relevant.
6. Immediate error and latency signals are normal enough to continue.
7. Verification logs identify the failed checkpoint without exposing secrets.
8. Rollback criteria are already defined and executable.
9. The verification suite remains small and does not duplicate CI.
10. A short observation window follows the automated check.

The Practical Rule

A deployment is a change to infrastructure. A successful release is evidence that the intended software is live and usable.

Keep pre-deploy CI broad. Keep post-deploy verification narrow. Bind the check to the exact release, exercise the real production boundary, verify one meaningful transaction, inspect immediate health signals, and know what failure triggers rollback.

That gives the team something more useful than a green deploy job: proof that the release actually works where users will meet it.