An AI prototyping sandbox where product teams ship in minutes

Industry: Media and events
Company size: 200-500 employees
Services: Platform engineering, developer experience, Kubernetes, AI infrastructure

The Challenge

The client's product managers and analysts had started building with AI coding agents, and the agents were good enough to produce working services. What they couldn't do was run them anywhere. Every prototype meant a ticket to the platform team, a review of manifests nobody else could write, and a wait measured in days.

The workaround made things worse. AI-built applications were piling up on a second cloud provider, outside the company's GitOps, single sign-on, monitoring, and cost controls. The platform team could not let untrained users or autonomous agents near the repository that defines production, and it could not keep supporting a shadow platform either.

Our Approach

We gave them their own front door to the cluster, with the walls built into the cluster rather than into a review process.

• A separate GitOps repository that Flux reconciles into a single walled-off namespace. Product staff and their agents merge there; a merge is live in under five minutes, and a deletion is pruned. The production platform repository stays untouched, and the reconciler itself runs as a namespace-scoped identity, so nothing in that repo can reach outside.
• A Sandbox custom resource for throwaway prototypes. One short YAML object describes an app, its database, and its queue. The operator builds it as one pod with a URL behind single sign-on, with a second pod available for a database that needs its own lifecycle, and deletes everything when the time limit expires. Long-lived work sets the lifetime to zero and lives in Git like everything else.
• Guardrails the cluster enforces on every object, no human in the loop: spot capacity only, a namespace quota with per-container ceilings, baseline pod security, no public exposure except through the authenticated ingress, and no LoadBalancer or NodePort at all.
• Hardened pods by default. A prototype gets no Kubernetes API token, no raw sockets, no privilege escalation, and the runtime's seccomp profile. It gets no cloud credentials either, unless it opts into a named service account with a policy scoped to its own storage buckets. Secrets reach a container by reference, never by pasting them into the object.
• A read-only cloud identity for the agents themselves, so an agent can read infrastructure state across four accounts to reason about a problem while secrets and data reads are denied by policy.
• A rules file written for the coding agents, listing what will be rejected, at which of the three enforcement layers, with the exact error messages and what each one means, so an agent can fix its own manifest instead of asking a human.
• A self-service dashboard to create, edit, and delete sandboxes, choose which existing secret a prototype may read, and see which apps share the namespace.

What happened in the first ten days

The question of how product users should be allowed to use AI-assisted development was raised in a Monday standup. The namespace, its guardrails, and a dedicated user group were live that Thursday, and the first product-side application was registered the same day. The Sandbox resource shipped the following Sunday. Over the next three days the operator went through seven releases as real users found real edges: readiness probes so "Ready" means the app is actually answering, scratch volumes for databases, secret injection, and the hardened defaults above.

Adoption did not need a rollout plan. Product-side staff were granted cluster and repository access in the first week, three contributors from outside the platform team merged into the sandbox repository, and an AI-built event demo moved off the second cloud's serverless runtime into the sandbox in a single pull request. Two days later its author replaced the hand-written manifests with one Sandbox object and turned on the non-root, read-only hardening.

The sandbox is an on-ramp, not a destination. Two AI-built services that proved themselves as prototypes, a campaign data service and a client-facing reporting application, were promoted into the managed non-production platform through the normal GitOps path: sealed secrets, single sign-on, an internal load balancer, and three independent review rounds. Production code still needs human review and an accountable owner. What changed is that the prototype no longer needs one.

One lesson worth sharing: the first time the operator ran with two replicas, both reconciled every Sandbox, and each prototype churned through about eight pods before settling. Leader election fixed it the same day. Guardrails that are cheap to change get changed; guardrails that live in a review queue do not.

The Results

• Non-engineers can deploy a working service without learning Kubernetes
• The production repository is off limits by policy, not by trust
• Failed experiments cost nothing to clean up because the cluster does it after 72 hours by default
• The same guardrails apply whether a human or an agent wrote the manifest
• Prototypes that prove out graduate through the same review path as every other service
• The shadow platform on the second cloud is being emptied instead of supported

Key Takeaway

The fastest way to let more people build is to move the safety checks from the review queue into the platform. When the cluster rejects a bad manifest in seconds, you don't need a person to reject it in days. And when the sandbox sits on the same cluster as production, with the same GitOps and identity underneath, a prototype that works has somewhere to go.

Related guide: Preview environments per pull request on Kubernetes: what is actually hard

Want your product team shipping prototypes without a ticket?

We'll show you what a guardrailed sandbox looks like on your cluster, and how prototypes graduate out of it.

Schedule a call