Local LLM or hosted AI API: when self-hosting pays

Every AI feature has the same bill hiding in it. With a hosted AI API, you pay each time the model reads something. That is fine at ten requests a day and painful at ten million. Running a model on your own machine changes the shape of the bill, but it adds work that an API quietly does for you.

This guide is for founders choosing between a local LLM and an API (the "local LLM vs API" question). It is based on two projects where I ran a small open model locally: an event-data platform and a sales-lead pipeline. It does not quote prices, because they change often and depend on the model, so it explains the trade-off instead.

When is a hosted AI API the cheaper choice?

At low or uneven volume, almost always. A hosted API costs nothing when nobody uses it, needs no servers, and gets better when the provider ships a new model. You are paying a small amount per request for someone else to run the hardware.

That makes an API the right default for a first version, for any feature you are still testing, and for work that arrives in bursts. A self-hosted model costs money whether or not anyone is using it, so a quiet feature on your own machine is the most expensive way to run it. Here is the trade-off in plain terms:

An API also tends to give you a stronger model than anything you would run yourself on a modest machine. If your task needs real judgment, such as reading a long contract and flagging risks, the quality gap can matter more than the bill.

When does a local model pay off?

A local model pays off when you run the same kind of job at high, steady volume, or when privacy, data residency or fine-tuning rules out sending data to a provider. The first is about money. The second is about what you are allowed to do.

On money, the logic is simple. An API bill rises with every request, while a machine costs about the same whether it handles a little or a lot, up to its limit. So there is a volume where the two lines cross. Here is the idea with made-up units:

Example figures in made-up units, to show the shape of the trade-off. They are not real prices.

Where your crossover sits depends on the model and your workload, so work it out with your own numbers before you commit. The point is the shape: one line climbs with volume, the other is nearly flat.

Then there are the reasons that are not about money. Some data cannot leave your systems: patient records, a client's financial files, anything under a data residency rule that says it must stay in a given country. Some teams also need a model fine-tuned on their own examples, which a local model makes easier to control. In those cases self-hosting is a requirement, not an optimisation.

The event platform I built for Conference Finder is a good example of the first case. It collects event pages from thousands of inconsistent sites, and paying a hosted AI for every page would have been far too expensive at that volume. Microsoft Phi-3 runs locally through llama.cpp instead, so there is no per-page API bill as volume grows. The case study has the full build.

Which jobs suit a small local model?

Narrow, repetitive jobs with a clear right answer: pulling dates and a location out of an event page, sorting messages into categories, or turning messy text into a fixed set of fields. A small model does these well when you tell it exactly what to return.

Jobs that need broad knowledge, long chains of reasoning or careful writing are a poor fit. That is where a larger hosted model earns its price. A useful test is whether you can describe the correct output in one sentence and check it mechanically. If you can, a small model is worth trying.

The best use of a small model is often as a fallback, not the first step. In the Conference Finder build, the model is called only when a page's structured data (the machine-readable markup many sites publish, called JSON-LD) lacks dates or a location. Rules and structured data go first, and the model fills the gaps. The result is saved as a plain record you can search and filter:

The same idea shaped the Legendrings lead pipeline, which monitors 130+ sources. Rule-based parsing is accepted when it is confident, and only the remaining pages go to a local Qwen 2.5 7B model, run through Ollama. Rules are fast and free, so the model only sees the hard cases. That keeps the expensive step small, whichever way you run it.

How do you make a small model's output reliable?

You never trust it directly. You ask for structured output, repair and validate what comes back, and save only the answers that pass your checks. A small model will sometimes return broken or invented answers, so the checking is part of the product.

The Conference Finder pipeline treats every answer that way:

• Repair. The model's JSON (a fixed, machine-readable format) is repaired when it is slightly broken, such as a missing bracket.
• Validate. The result must have the right fields and sensible values before it counts as a record.
• Score. Each record gets a confidence score, and a record is saved only above a threshold.
• Filter early. A relevance pre-filter rejects junk pages before the expensive step, so the model never spends time on them.
• Separate the work. Fast collection and slow extraction run on separate queues, each with its own rate limit, so one cannot starve the other.

None of this is specific to local models. It is what you would want around a hosted API too, since those also make mistakes. It matters more locally because a smaller model makes more of them, and because you have no provider to blame.

How do you avoid paying for a GPU all month?

Start the GPU for the job and stop it when the job ends. A rented GPU bills while it runs, so a batch workload does not need one switched on around the clock.

This is the hidden cost people miss when they compare a local LLM with an API. Self-hosting is not only the model. It is also the GPU, the serving software that makes the model answer requests, the monitoring that tells you when it has stopped, and the staff time to keep all of it running. A machine left on all month for work that takes a few hours is a bill for the other days.

In the Legendrings pipeline, the local model runs on a GPU cloud instance that is started for the run and stopped after it. The dashboard has a panel for the instance, and in the screenshot it shows as stopped.

Batch work suits this well, because nobody is waiting on an answer at 3 a.m. Live features that need an instant reply are harder, since the machine has to be ready when a user shows up. If your feature is live and steady, the cost of an always-on machine is part of the comparison. If it is a nightly or weekly job, it should not be.

A short checklist before you self-host:

• Is the volume high and steady, or could a hosted API do it for less?
• Is there a privacy, residency or fine-tuning reason that makes a provider impossible?
• Can rules or structured data handle most inputs, leaving the model the hard cases?
• Who will watch the machine, and what happens when it fails at night?
• Can the work run in batches, so the GPU is only on when it is needed?

If you are weighing this for a real product, see how AI and automation work is scoped, or read the lead pipeline case study for the hybrid approach in full. For budgets in general, read what an MVP really costs. A Build Plan starts from $500, is ready in 2 to 3 days, and is credited in full toward the build, so you can settle this choice on paper before any code.

Questions founders ask about local LLMs and APIs

Written bySikandar Javid

Technical partner for real-estate and fintech founders: decides with you what to build, launches it and stays on after launch. Top Rated Plus on Upwork with 100% Job Success. Builds MVPs, rescues stalled builds and hardens AI prototypes.

See client work →