Preview environments per pull request on Kubernetes: what is actually hard

A preview environment per pull request is one of the few platform features that developers, reviewers, product managers, and QA all ask for unprompted. It is also one of the easiest to build badly. Creating the environment is a week of work. Living with fifty of them is the project. This guide covers the options, the parts that are actually hard, and what we would do again.

The options

• A hosted product. Vercel-style previews for front ends, or services that spin up a full stack per branch. Fastest to adopt, least control, and they get expensive or impossible once the environment needs your database, your queues, or your VPC.
• A shared staging environment with feature flags. Cheap and fine for small teams. It stops working the day two people need to show different unmerged changes to different stakeholders at the same time.
• Your own, on Kubernetes. A namespace or a release per pull request, created by CI and torn down when the pull request closes. Full control, and every piece of it is yours to maintain.

If you already run Kubernetes, the third option is usually right, and the rest of this guide is about doing it well.

Decide what a stack is before you automate it

Write down the shape of one environment as a Helm chart, and make the chart the only definition. Ours puts the web container, queue workers, scheduler, and optional database and cache sidecars in one pod with localhost networking, so nothing crosses a network boundary and nothing needs a service mesh. Every connection string is derived by the chart from the mode you pick, and the chart refuses to render if you pick two.

The hard part is the database, and there are four answers, each right for a different pull request.

1. A throwaway database sidecar with a small volume, for changes that need a clean slate.
2. A pointer at the shared non-production database, for changes that do not touch the schema.
3. A copy-on-write clone of a production snapshot, provisioned in about a minute on a filesystem that stores only the blocks the pull request changes, for anything that has to be tested against real data shapes.
4. A clone cut from a streaming standby, for data platforms where the snapshot is too large or too stale.

Migrations run automatically before the web server starts, and workers wait for migrations to finish. Get that ordering wrong and every stack boots into a race.

Queues and dependent services

Do not provision cloud queues per pull request. An SQS-compatible sidecar with dead-letter queues, visibility timeouts, and a queue dashboard on the stack's own hostname gives you the behaviour without the cloud resources or the cleanup. Any dependent microservice runs as another container in the pod, and the chart switches the application's endpoint variables to localhost when it is enabled.

Cost controls belong in the template

Stacks should scale to zero outside working hours, prefer spot capacity, and spread across zones, and none of that should be a decision anyone makes per pull request. Put it in the chart. A pull request environment that costs money on a Sunday is a bug in the template.

The part everyone underestimates: lifecycle

Once environments exist, the second problem appears. An engineer without cluster access cannot tell whether their stack is healthy, tail its logs, restart it, or tear it down, so every one of those becomes a message to the platform team. Nobody can answer "how many stacks are live right now" without checking three or four tools. And cleanup is a cron job somebody wrote on a Friday.

The answer that worked for us was to make every stack a first-class Kubernetes object. A small operator watches Helm's own release records and emits a status object per stack: phase, replica readiness, chart and revision, the ingress URL, the pull request's state on GitHub, and events for every transition. Restarts are an annotation. One-time bootstrap commands are listed on the object and run once after the stack is ready, idempotent by name and never retried blindly, because a blindly retried migration can corrupt data.

Cleanup uses the same object. Orphaned stacks, where the pull request closed or lost its label, are reaped by dispatching the repository's own teardown workflow. Stacks whose release never came up are garbage-collected after a grace period. A teardown that keeps failing is marked stuck after three attempts and backed off, instead of hammering the CI system every few minutes. Idle stacks retire after a fixed number of days of pull request inactivity, with a protected label for QA work that must survive.

Then a dashboard, and keep it thin

A dashboard is what makes the environments self-service for people who do not run commands. Build it as a view over the operator's status object, not as a second source of truth. Ours started by reading Helm releases directly; we reversed that, wrote the operator first, and swapped the dashboard's data layer without a rewrite. Every live stack across every repository, live logs, restart, run a command, teardown, and an audit log of who did what. Put it behind single sign-on from the first day.

What did not go to plan

• Removing a pull request's label only triggers teardown while the pull request is open. Closed and merged ones need a direct workflow dispatch, and long-lived stacks have no pull request at all, so teardown is a three-way branch rather than one call.
• A pull request with merge conflicts never fires the deploy workflow at all, not even as skipped. It looks identical to a broken pipeline until you check mergeability.
• A live log stream that reconnected with stale credentials after the single sign-on session lapsed produced about 200 authorization cycles an hour against the identity provider. Handle session expiry in anything that streams.
• Running two operator replicas without leader election meant both reconciled everything. Fixed the same day; easy to miss in a first rollout.

What it delivered

Across thirteen repositories, more than 1,470 stacks have deployed since February, with a median of six minutes from push to a working URL and eleven at the 90th percentile. Volume roughly doubled after the operator and dashboard shipped, because engineers stopped treating a stack as a favour. Platform-team tickets to see, restart, or tear down a stack went to zero. The same chart and operator later grew two more kinds of environment, long-lived dev stacks and prototyping sandboxes, in days rather than months, because everything between "born" and "torn down" was already shared.

The full story

The chart, operator, and dashboard are described in detail, including the database clone design, in the pull request environments case study. The sandboxes built on the same operator are in the AI prototyping sandbox case study.

Want pull request environments your team runs themselves?

We'll show you the chart, the operator, and the dashboard on a live stack, and what it takes to onboard your first repository.

Schedule a call