Case Study for “Viewcraft”
Introduction
You want to launch a g6e instance to run an AI image generation. You ask a region for one. The region says no, not right now, no capacity. This is the constraint that shaped everything we built for Viewcraft’s GPU infrastructure, and it’s the reason the system spans six AWS regions instead of living comfortably in one.
This post is about running scarce GPU workloads across multiple regions: why we did it, how we orchestrate it with AWS Step Functions, and the challenges we hit along the way; cold starts, warm-up times, and the ever-present hunt for g6e capacity. It’s written for engineers, so it leans on the tradeoffs and the dead ends rather than the highlight reel.
The real problem: g6e capacity is scarce
Viewcraft runs its AI image generation on g6e instances, which carry NVIDIA L40S GPUs. They’re the right hardware for the workload. The problem is that they’re hard to get. Capacity is limited, not every region has them available when you ask, and during GPU crunches a given region will simply refuse to launch one.
Once you accept that a single region cannot be relied on to hand you a GPU on demand, the architecture has to change. You stop thinking “launch an instance” and start thinking “find a region that will actually give me one.” That is fundamentally a capacity problem, and going multi-region is how we solve it.
Two levers, two different reasons
It’s worth being precise about this up front, because it’s easy to conflate the two and get the reasoning backwards:
Multi-region orchestration is about capacity. We spread across regions because g6e is scarce and we need somewhere (anywhere) that will give us a GPU.
Instance lifecycle (stopped or terminated) is about cost. We keep GPUs from running idle because idle GPU time is expensive. This has nothing to do with which region they’re in.
Keeping these separate in your head matters, because the right answer for capacity (scan widely across regions) pulls in a different direction from the right answer for cost (keep as few things running as possible). The interesting part of the design is where those two pressures meet.
Multi-region orchestration with Step Functions
Our approach to capacity scarcity is deliberately unglamorous: try regions one at a time, in priority order, until one gives us an instance. AWS Step Functions is an excellent fit here because the whole thing is really just a state machine with error handling and Step Functions makes that readable and debuggable instead of burying retry logic somewhere in application code.
The state machine tries six regions in priority order:
eu-central-1
us-east-1
us-east-2
eu-south-2
eu-north-1
eu-south-1
Each region attempt calls a Lambda function that handles the actual EC2 API call. The Lambda has its own resilience two attempts with exponential backoff so a transient hiccup doesn’t immediately burn a region. If the Lambda ultimately fails for that region (no capacity, a launch error, whatever), the state machine catches the error and falls through to the next region in the list.
The shape of it is simple enough to hold in your head:
Try Region → Check Result (Choice state) → Success, or fall through to the next region.
If all six regions fail, the state machine terminates with a NoCapacityAvailable error. That's rare, but it happens during the worst GPU crunches, and it's genuinely useful that it fails loudly with a named error rather than hanging or silently degrading.
Step Functions Diagram
Notably, we do this sequentially, not in parallel more on why in the lessons section, but it’s one of the decisions I’m most confident about in hindsight.
ECS for backend and frontend
Everything that isn’t the GPU fleet runs on ECS. Viewcraft’s backend API and frontend are both deployed as ECS services. This gives us container orchestration rolling deploys, health-managed tasks, straightforward scaling without taking on the operational weight of running Kubernetes ourselves. For a workload where the genuinely hard part is the GPU layer, we didn’t want to also be babysitting a control plane.
The request flow is straightforward: a user hits the frontend, the ECS-hosted backend receives the request, and when GPU work is needed the backend triggers a Step Functions execution. Step Functions finds a region, gets an instance serving, and the result flows back to the user.
The separation matters. The ECS layer is stateless, cheap, and elastic; the GPU layer is expensive, scarce, and slow to warm. Decoupling them means the easy-to-scale part scales freely and the orchestration logic only has to reason about the hard part.
Cost efficiency: don’t pay for idle GPUs
Multi-region solves capacity. It does nothing for cost if anything, spreading out makes it easier to accidentally leave expensive hardware running in places you’ve forgotten about. The cost lever is separate: instance lifecycle. An idle, running g6e is the most expensive way to have a GPU ready, so we don’t do that. Instead we choose between two cheaper states depending on the latency we can tolerate:
Terminated: you pay nothing while idle, but a cold start means launching a fresh instance from an AMI. Cheapest, slowest.
Stopped: you pay only for the EBS volume while idle, and starting the instance is far faster than a fresh launch. A little standing cost, much faster start.
Here’s where the two pressures meet, and it explains something that might otherwise look inconsistent. Our capacity-hunting path scans all six regions, because launching fresh from an AMI costs nothing until it succeeds, so you can afford to look everywhere. But stopped instances carry a standing EBS cost, and you can’t afford to keep those warm in every region. So we keep stopped-and-ready instances in a smaller subset the state machine that starts stopped instances fails over across three regions (us-east-1 → us-east-2 → us-west-2), not six. Wide net for capacity, narrow footprint for cost. The two region counts aren’t a contradiction; they’re the two levers pulling in their natural directions.
Challenges
The reframe above is clean in hindsight. Getting there was not. Here are the challenges that actually shaped the system.
G6e capacity scarcity
This is the root of everything. L40S-backed g6e instances are in genuinely short supply, and availability shifts region to region and hour to hour. There’s no clever trick that makes a region give you a GPU it doesn’t have the only real answer is to try more places, which is exactly what the six-region state machine does. Every other design choice downstream is a consequence of this one hard fact.
Cold start, the three iterations it took
Getting a GPU is only half the battle. Once you have the instance, it has to become ready to serve, and that took us three iterations to get right.
Phase: the naive approach (~10–12 minutes). The first version launched a fresh g6e from a base AMI (~2–3 min), downloaded the ~24 GB model onto the instance (~3–5 min and unreliable S3 bandwidth varies, cross-region downloads are slow, and sometimes it failed outright), then loaded the model into GPU VRAM (~45 s). Ten to twelve minutes from request to ready. The download step was the villain: not just slow, but the single most likely thing to fail and send the whole attempt back to square one.
Phase 2: bake the model into the AMI (~4–6 minutes). If downloading 24 GB every cold start is the worst part, stop downloading it. We built custom AMIs with the model weights pre-baked onto the EBS volume, so the instance boots with the model already on disk and the download step disappears. Cold start dropped to roughly four to six minutes just launch plus VRAM load. The tradeoff: the AMI now has to exist in every region, so a model update means rebuild the AMI and copy it out before anyone can serve the new version.
Phase 3: stopped instances with health checks (~3–4 minutes). The insight: starting a stopped instance (~30–60 s) is far faster than launching a fresh one (~2–3 min). So we keep pre-configured g6e instances stopped-but-ready, AMI baked, EBS volume warm and start them on demand. The VRAM load adds only about 45 seconds, so getting a stopped instance from start to serving is quick, and the total sits comfortably in the three-to-four-minute range. This is the state machine that fails over across the three regions above.
The evolution in one line: naive launch-and-download → AMI-baked launch → stopped instances plus health checks.
Cold-start times and the first-generation wait
Cold-start time is not the same as the time the first user actually waits. Once an instance reports ready, the first generation on a freshly warmed GPU still has to run, and that first request tends to be the slowest one the instance ever serves. So the real user-perceived latency on a cold path is cold start plus first generation:
~3–4 minutes cold start + ~1–6 minutes for the first generation (depending on image quality and output specs) = roughly 4–10 minutes total for the first user.
This is the number that actually matters to a person staring at a loading spinner, and it’s worth measuring and quoting separately from the infrastructure cold start, because they get conflated constantly.
Knowing when “started” means “ready”
A started instance is not a ready instance. EC2 will happily report the instance as running while the inference server is still loading the model into VRAM and can’t accept a single request. Our answer is a health-check Lambda that polls port 8000 where the inference server listens and the state machine refuses to report success until the server actually responds. The poll runs with about a 9.5-minute timeout. We don’t expect a healthy start to take anywhere near that; the timeout is generous precisely because occasionally things are slow, and its size is an honest measure of how much uncertainty is still in the system.
Cross-region AMI management
Baking the model into the AMI fixed the download problem but created an operational one that grows with every region: each model update means rebuilding the AMI and copying it to all the regions that need it before any of them can serve the new version. It’s the kind of overhead that’s easy to underestimate and quietly compounds as the fleet spreads.
Lessons learned
Separate your capacity strategy from your cost strategy
The single most useful mental shift was refusing to conflate the two. Multi-region exists for capacity; instance lifecycle exists for cost. Once we stopped treating “which regions” and “stopped vs terminated” as one decision, both got clearer and the six-vs-three region footprint stopped looking like an inconsistency and started looking like the correct answer to two different questions.
Sequential failover is simpler than parallel and cheaper
Trying all regions at once feels faster on paper. Sequential won on every axis that mattered. It’s far easier to reason about and debug at any moment the state machine is trying exactly one region, and the execution history reads like a story. And you don’t pay for failed parallel launches: fire six launches at once and you may start several GPU instances before you can cancel the extras, which is not where you want surprise spend.
Health checks are non-negotiable
Every layer of infrastructure will cheerfully report success before your application can do its job. Without the health-check Lambda gating success on a real response from port 8000, we’d routinely route users to instances that aren’t serving yet. The health check is what makes “success” mean something.
Model size drives every architectural decision
Every phase of the cold-start journey is a response to one fact: the model is 24 GB. The download was slow because it’s big. The AMI copy overhead exists because it’s big. The unavoidable VRAM load exists because it’s big. Measure your model early and let that number drive the design, because it will whether you plan for it or not.
Be honest about the tradeoffs
None of this is free. Stopped-but-ready instances mean paying for idle EBS. The baked-AMI strategy means ongoing pain managing images across regions on every model change. The 9.5-minute health-check timeout is a standing admission that we can’t fully predict how long a start will take. This is a set of deliberate tradeoffs, not a trick that made the costs vanish.
What’s next
The obvious next frontier is keeping some capacity genuinely warm instances running with the model already in VRAM so the fastest path skips the load entirely, balanced against the cost of paying for idle GPUs. We’re also weighing a container-based orchestration layer for the GPU fleet for finer-grained control than the stopped-instance model gives us. Both come back to the same tension that runs through this whole system: how much are you willing to pay to keep a scarce, expensive resource ready for a request that hasn’t arrived yet? We don’t think that question has a permanent answer just a current best one.
Check out our medium page: Clerion Medium