Over the past few weeks, we’ve been talking a lot about LLMs, cost, and real-world
practicality.
From the hidden costs of foundational model providers to the diminishing returns of scaling large language models, the message has been clear: raw power inference power is only the one dimension to consider. And inference power certainly isn’t cheap.
But there’s good news. Thoughtful design choices can dramatically reduce the cost of deploying GenAI at scale.
In this three-part series, we’ll explore practical ways to build GenAI agents that deliver results without draining your budget, including:
Matching the right model to the right task
Training smaller models with data generated by larger ones
Using model caching to avoid redundant inference
Part 1: Right Model, Right Task
If you’re building anything agentic — copilots, assistants, planners, explainers — one of the best ways to control costs is by not using your most powerful model for everything.
Let’s walk through a real example.
At Fuse, we’re building an agentic data analyst that allows users to query their data warehouse using natural language. The system needs to understand vague or ambiguous user questions and turn them into a set of SQL queries that can be executed in parallel.
Take a question like:
“How’s the business doing out west?”
This is vague. There’s no metric. No time frame. The system needs to:
Identify missing context using the metadata in the warehouse (e.g. what "west" might mean geographically)
Break the question into several precise sub-questions (e.g. sales, margin, customer churn)
Generate optimized SQL for each sub-question
Run the queries in parallel
Synthesize the results into a single narrative that answers the original prompt
To do this, we use a tool-calling LLM to orchestrate the process. That orchestration step requires advanced reasoning, so we use Gemini Pro.
But synthesis? That’s a different job.
Once we have the answers, the task becomes: "write a coherent summary that references these values and aligns to the original intent."
That’s a great use case for a smaller model. In our case, we use Gemini Flash, which:
Has much lower latency
Costs significantly less per 1M tokens
Performs very well on summarization and synthesis
So we get the best of both worlds:
High reasoning accuracy when it matters most (Gemini Pro)
Faster, cheaper inference when the task is well-defined (Gemini Flash)
This pattern shows up everywhere in agent design. Consider:
Using a large model to plan a task, and a smaller one to execute it
Validating inputs with a smaller model before escalating to a larger one
Reserving premium models for ambiguous or high-impact flows only
In short?
Don’t use a sledgehammer when a screwdriver will do.
Thoughtful model selection is one of the fastest ways to cut GenAI costs without degrading your user experience or accuracy.
In the next post, we’ll look at how to use large models to train smaller ones, so you can build lean, high-performing agents without paying premium inference prices forever.
At Fuse, we believe a great data strategy only matters if it leads to action.
If you’re ready to move from planning to execution — and build solutions your team will actually use — let’s talk.
