A fintech startup came to us with a problem that had nothing to do with their product and everything to do with their survival: their AI infrastructure was costing $100,000 per year, and they were a seed-stage company with 18 months of runway. At that burn rate, infrastructure alone was eating nearly six months of their life as a company.
By the time we were done, they were spending $7,000 per year. Same performance. Same models. Same user experience. The difference was architecture — and the five decisions that led to it.
I can't name the company due to our agreement, but I can share every architectural lesson. Here's exactly what we found, what we changed, and what you can take from this whether you're running a fintech startup or any company where AI infrastructure costs feel out of control.
93%
$100K/year reduced to $7K/year — same model performance, improved latency, and months of runway returned.
What we walked into
The startup had hired a development agency to build their initial AI system. The agency was competent at web development but had limited ML infrastructure experience. They made a series of decisions that are individually defensible but collectively catastrophic for costs.
Always-on GPU instances. The agency provisioned dedicated GPU instances running 24/7 for model inference. The startup's actual traffic pattern? Concentrated between 9 AM and 6 PM Eastern, with virtually zero overnight usage. They were paying for 24 hours of GPU compute to serve 9 hours of traffic.
Monolithic model serving. Every model — from their core prediction model to a simple text classifier — was served through the same heavyweight infrastructure. A model that needed 50ms to respond was sitting behind the same load balancer as a model that needed 500ms. The infrastructure was sized for the slowest, most resource-hungry model, even when 80% of requests hit the lightweight ones.
No caching layer. The same predictions were being computed repeatedly for identical inputs. In fintech, many queries are repetitive — the same market data, the same risk calculations, the same customer segments. Without a caching layer, every request was a full model inference, even when the answer hadn't changed in hours.
Oversized data pipeline. The training pipeline was running on the same class of machines as serving. Training happened once a week. The machines ran continuously. That's like renting a moving truck for your daily commute because you moved apartments once last year.
Managed services for everything. The agency defaulted to the most expensive managed service for every component — managed Kubernetes, managed model serving, managed monitoring — without evaluating whether the startup's scale justified the cost. At 10,000 daily active users, it didn't.
This isn't a story about incompetence. Every one of these decisions made sense in isolation, to a team optimizing for speed of delivery rather than cost of operation. The agency's job was to get the product working. They did. But nobody asked the question: "What does this cost to run at this scale?" until the bills started arriving.
The five changes that mattered
We didn't rewrite the product. We didn't change the models. We changed five infrastructure decisions — each one individually significant, together transformative.
1. Time-based compute scaling
The most impactful single change. We moved from always-on GPU instances to a time-aware scaling policy. The system scales up during business hours, scales to minimal capacity overnight, and spins up additional instances only when request volume exceeds thresholds.
For this specific startup, 70% of their compute spend was happening during hours when virtually no one was using the product. The scaling policy alone cut costs by nearly half.
The key insight isn't "use auto-scaling" — every cloud tutorial tells you that. The key insight is that you need to understand your actual traffic pattern deeply before designing your scaling strategy. We spent two days analyzing their request logs before writing a single line of infrastructure code. Those two days of analysis saved more money than anything else we did.
2. Tiered model serving
Not every model needs the same infrastructure. We split their serving into three tiers:
The lightweight tier handles simple classifiers and rule-based models on CPU instances. No GPU required. These models respond in under 10ms and handle 80% of incoming requests.
The standard tier handles their core prediction models on modest GPU instances. These are the models that actually need GPU acceleration, but they don't need the largest instance type available.
The heavy tier handles their most complex models, served on-demand with aggressive caching. These models run infrequently enough that cold-start latency is acceptable.
The previous architecture treated all models identically. The tiered approach means each model gets exactly the resources it needs — no more, no less.
3. Intelligent caching
We added a prediction cache with a time-based invalidation strategy. For their use case, many predictions are valid for minutes or hours — market conditions don't change millisecond by millisecond for the types of analysis their users perform.
The cache intercepts incoming requests, checks whether a valid prediction exists for those inputs within the staleness window, and returns the cached result if so. Cache hit rate stabilized around 60%, which means 60% of requests never touch the model at all.
The important nuance: caching predictions isn't the same as caching API responses. We cache at the model output level, keyed on the feature vector, not the raw request. This means differently-formatted requests that produce the same feature vector share a cache entry. That subtlety doubled our effective hit rate.
4. Right-sized data pipeline
Training happens weekly. The training pipeline now runs on spot instances that spin up for the training job and terminate when it's complete. Training takes approximately four hours. They were previously paying for 168 hours of compute per week (24/7) to run a 4-hour job.
We also moved their feature engineering from a real-time streaming architecture to a batch process. The agency had built a Kafka-based streaming pipeline for feature computation — impressive engineering, but wildly over-built for weekly model retraining. A scheduled batch job that runs an hour before training produces identical features at a fraction of the cost.
5. Managed services audit
We replaced three managed services with self-hosted alternatives that are appropriate for their scale:
Managed Kubernetes was replaced with a simpler container orchestration approach. At their scale, they didn't need Kubernetes at all — a basic container service handled their workload.
The managed model serving platform was replaced with a custom FastAPI-based serving layer. At their volume, a framework that costs $0 and takes two days to set up outperforms a managed platform that costs thousands per month.
Managed monitoring was replaced with open-source alternatives. At their scale, the open-source tools provide everything they need.
The general principle: managed services are worth their premium when you're operating at a scale where managing the infrastructure yourself would require dedicated personnel. At seed-stage volumes, you're paying enterprise prices for startup-scale problems.
The result
| After | After | |
|---|---|---|
| Annual infrastructure cost | $100,000 | $7,000 |
| Model performance | Baseline | Same or better |
| Average latency | Baseline | Improved (caching) |
| Time to implement | — | ~4 weeks |
| Runway impact | — | Extended by months |
The $93,000 in annual savings didn't just reduce their burn rate. It fundamentally changed their fundraising position. They could now demonstrate capital efficiency that investors notice — not just a good product, but a team that knows how to build without burning cash.
How to know if you have this problem
If you answer "yes" to two or more of these, your infrastructure is almost certainly oversized:
- Your AI infrastructure costs more than $5K/month and you have fewer than 100K monthly active users
- You're running GPU instances 24/7 but your traffic has clear peaks and valleys
- You couldn't explain to an investor exactly why each line item in your cloud bill exists
- Your infrastructure was set up by a web dev agency, a full-stack team, or someone following a tutorial
- You haven't had an ML infrastructure specialist review your architecture since launch
If that sounds familiar, the savings are almost always there. Sometimes it's a 30% reduction. Sometimes it's 93%. But I've never reviewed a startup's AI infrastructure and found nothing to optimize.
Burning cash on infrastructure?
We'll do a quick assessment of your AI infrastructure and tell you where the waste is — even if we're not the ones who fix it.
Talk to Manmeet
Manmeet Singh
Founder & CEO, AIshar Labs · Ex-Apple, Ex-Instacart · 15 AI Patents
Built ML systems at Apple (Search: Maps, Safari, Spotlight) and Instacart (Search, Recommendations, Ranking). Writes about production AI tradeoffs and system design.
Follow on LinkedIn →