Rebuilding our deployment engine for speed and resilience

Deploying a container on Bahriya looks like one action to you: apply a change, and moments later it is running in every region you have chosen. Behind that single action is the system that actually carries your change out to the clusters — and we recently rebuilt it from the ground up. This is the story of why, and what we learned.

Where we started

Our first deployment engine was built on a general-purpose CI/CD pipeline system. It was a reasonable place to start: pipelines are well understood, and they gave us a working path from "customer applied a change" to "container running in a cluster."

But a pipeline is fundamentally a linear thing. It runs a sequence of steps to completion. That shape fought us in two ways. First, multi-region rollouts happened region after region, so deploy time grew with your footprint — a multi-region change took roughly ten minutes. Second, a pipeline is not a natural fit for something that should be continuously true. A deployment is not a one-off job; it is a statement about how the world should look, and the system's job is to keep making that true, forever, through restarts and failures.

We were not happy with either property. So we spent about two weeks replacing the engine with something purpose-built.

The core idea: reconcile toward a desired state

The new engine treats every project as a desired state — a versioned description of exactly what should exist: which containers, which datastores, which secrets and configs, which network rules, in which regions. That description lives in version control, which turns out to be the most important design decision we made.

The engine's job is no longer to "run a deploy." It is to continuously reconcile the real world toward that desired state. You change the description; the engine notices and converges. Something drifts; the engine converges it back. A cluster comes back from an outage; the engine rebuilds it to match. Deployment stops being an event and becomes a property that is always being maintained.

Modular, and parallel by region

Each region runs its own deployer, and each converges independently. This is where the speed came from: your regions no longer wait in line behind one another — they roll out at the same time. A busy or briefly unreachable region isolates itself and catches up on its own, without holding the others hostage. Adding a region adds a deployer; it does not add to your rollout time.

Modularity also made the system far easier to reason about. Each deployer has one job — make my region match the desired state — and does it in isolation, rather than one monolith trying to coordinate everything at once.

Disaster recovery, by design

Because the desired state (and every teardown intent) lives in version control rather than inside a running process, recovery is almost free. Nothing critical exists only in memory. If a cluster is lost, we do not restore it from a backup and hope it matches — we point a fresh deployer at the versioned source of truth and let it rebuild the region to exactly the intended state. The same source of truth is an audit trail: every change to what should be running is a recorded, reviewable revision.

Two other properties fall out of this design almost for free. Operations are idempotent — applying the same desired state twice changes nothing the second time — so retries are always safe. And the engine is self-healing: workers watch their own health and recover automatically rather than sitting wedged.

The two weeks in the middle

Rebuilding the thing that touches every customer's production is not a weekend project, and we will not pretend it went perfectly. The architecture was sound quickly; making it trustworthy took the fortnight.

We ran the new engine against real, full multi-region deployments — repeatedly — and hunted down the edge cases that only surface under real conditions: a race between reading a project's intended state and that state being fully written, ordering hazards when tearing a project down, resource pressure in long-lived workers under sustained load. Each run surfaced something; we fixed it and ran again, until repeated end-to-end deployments and teardowns were boringly, reliably identical.

That boredom was the goal. A deployment engine should be the least dramatic part of your day.

The payoff

A change now reaches every region in under three minutes, in parallel, consistently — down from around ten. Just as importantly, the system underneath is one we can reason about, recover, and extend: independent per-region deployers, a versioned source of truth, idempotent reconciliation, and self-healing workers.

You will never see any of this. That is rather the point.