Private LLMs: when to run your own model

What private LLMs actually are

Run your own model only when a rule forbids sending data out, when steady volume keeps GPUs busy, or when you need to change the model itself. Otherwise a contract change or a managed model in your own cloud account is usually enough. Private LLMs are language models you run where you control the data path, so prompts and answers stay inside boundaries you set. The word "private" covers several setups with very different effort attached.

Most of the confusion in this topic comes from people using one phrase for all of them. A vendor's API with a zero retention promise is called private. So is a model running on a server in your own rack.

What they share is the question they answer: who can see the text you send? A clean way to frame it is by asking where the prompt travels, who stores it, and who can read the logs. If you can answer all three with names and contracts, you have a private setup. If any answer is "the vendor, probably", you do not yet.

The four ways to make private LLMs real

The options run from a contract change to buying hardware. Each step buys more control and costs more engineering. Pick the lightest one that satisfies your actual obligation, which is often a contract clause and not a server.

• Vendor API with retention limits. The model runs in the vendor's cloud. You control the contract terms and carry almost nothing else.
• Managed model in your cloud account. The model runs in your AWS, Azure or similar tenant. You control network, access and region, and you carry the cloud configuration.
• Self hosted LLM on rented GPUs. The model runs in your cloud on servers you manage. You control the model, weights and logs, and you carry serving, scaling and updates.
• On premise LLM. The model runs in your own data centre. You control everything, including hardware, and you carry all of the above plus procurement.

The first two options are where most companies should start. AWS documents that Bedrock does not use your prompts to train models, and OpenAI states that business data is not used for training by default. If you are weighing these against a custom build, our LLM development work covers both the managed and the self hosted routes.

When running your own model is justified

Run your own model when a rule forbids sending the data out, when volume is high and steady enough to keep GPUs busy, or when you need to change the model itself. Convenience and curiosity are weak reasons.

The three real reasons, in the order most teams meet them:

• A hard rule. Some contracts, health data regimes and defence work bar third party processing outright, and no vendor promise changes that. A private LLM deployment is then a requirement and the discussion ends.
• High, steady volume. If one workload sends millions of similar requests a day, a rented GPU kept busy can beat per token billing. Check this against your own numbers, because the break even point depends on model size and prices.
• Control over the model. This means fine tuning on proprietary data, pinning a version so behaviour never shifts under you, or running offline.

Pinning matters more than people expect. A hosted model can be retired or updated, and prompts that worked last quarter may quietly behave differently.

When self hosting private LLMs is the wrong call

Skip self hosting if your volume is low or spiky, your data is not regulated, or nobody will be on call for the GPU servers at 2 a.m. The costs sit in people and operations, and the hardware is only the visible part.

Three warning signs, any of which should slow you down:

• Low utilisation. A GPU bills by the hour whether it answers one question or ten thousand, so an idle machine is money spent on nothing.
• Spiky traffic. You either overprovision for the peaks or accept slow queues when they hit.
• No operator. If no named person will watch quality, capacity and updates, the setup will drift.

There is also the leak that self hosting does not fix. In 2023, Bloomberg reported that Samsung restricted staff use of generative AI tools after employees uploaded sensitive code. The cause was people pasting into a public tool.

Hosting a model inside your walls does little if staff still paste secrets into whatever chatbot is open in another tab. Give them an approved internal option, and write the policy down.

What open source LLM hosting takes

Open source LLM hosting needs three things: enough GPU memory to hold the weights, a serving engine that batches requests well, and monitoring so you notice when quality or speed drops. Memory is the constraint that surprises people first.

Model weights at 16-bit precision take about two bytes per parameter. A 70 billion parameter model therefore needs roughly 140 GB just for the weights, before any working memory. Quantising to 4-bit cuts that to around 35 GB at some cost in accuracy that you should measure, not assume.

The serving engine matters as much as the card. The vLLM paper reports 2 to 4 times the throughput of earlier serving systems at similar latency, by managing memory for in-flight requests more carefully. Pick an engine that does this, because the naive approach wastes most of an expensive GPU.

Meta's Llama 3 paper is a useful reference for the sizes open models come in, from small to very large.

Checking whether the open model is good enough

Do not decide on a leaderboard. Build a test set from your own real requests, run each candidate model over it, and have a person who knows the work grade the answers. Public benchmarks measure someone else's tasks.

Start with 100 to 200 real examples pulled from actual usage, including the awkward ones. Grade them blind, so the grader does not know which model wrote which answer. Count how often each model is acceptable, and look hard at the failures, because the failures are where your users will lose trust.

Expect the gap to depend on the job. Extraction, classification and answering from your own documents tend to close well on smaller open models. Long, open ended reasoning is where the largest paid models usually keep an edge, so test that specifically if it is your use case.

What tends to break first in private LLM deployments

Teams that move from an API to their own model most often hit the same few problems, and each one is measurable before launch. Record the number for each during your trial, so you know what "normal" looks like later.

• Latency. Time to first token and total response time both change when you serve the model yourself. Measure them at your real prompt lengths, not on short demos.
• Quality gap. The smaller or quantised model may handle most requests and fail on a specific class. Your graded test set shows which class.
• Cost per thousand requests. Include idle hours, not just busy ones, or the figure will flatter the self hosted option.
• Data leaks around the model. Logs, caches and debugging dumps can hold prompts. Decide who reads them and how long they live.

Keep each figure next to the sentence that explains it, with the period it covers. A cost number without a time window cannot be compared with next month's bill.

A one week test before you buy anything

Spend a week measuring before you commit. Log your real prompt volume, run a candidate open model on one rented GPU, and compare quality and cost per thousand requests against your current API. That gives you evidence in place of opinions.

A workable schedule:

• Day one: count requests per day and tokens per request from your existing logs.
• Days two and three: stand up one open model on a single rented GPU with a modern serving engine.
• Days four and five: replay a sample of real prompts and grade the results.

Then work out utilisation: what share of the hours that GPU was busy. If it sits idle most of the day, a per token API wins. If it is busy most of the day, the self hosted case is real.

Keep security in the same week. The US NIST AI Risk Management Framework is a reasonable checklist for who can query the model, what gets logged and how long logs live. If the review turns up gaps in how your wider systems handle data, custom software development can close them around the model, not only inside it.

Who should own your private LLMs afterwards

Name one owner before launch. A private model needs someone to apply updates, watch quality, manage GPU capacity and retire old versions. Without a named person, a deployment can decay quietly, because nothing fails loudly when quality slips.

That is the part most plans skip. Newer open models appear often, so the one you picked will eventually look dated. Decide up front how you will evaluate a replacement and how you will swap it without breaking the applications built on it.

A simple way to hold this together is to treat the model as a service with an owner and a short runbook. The runbook covers how to restart it, how to roll back a version, and who gets paged. If you do not have engineering leadership in house to own that, a fractional technical lead can cover the gap while you decide.

Where to start

Start with the cheapest private option that satisfies your actual rule, and prove the case for self hosting with a week of numbers. Often the answer is a contract clause or a managed model in your cloud account, and the GPU question never needs to be asked.

If you cannot yet say which of your workflows even needs a model, begin there. Book a workflow audit and we will map where text processing sits in your operations, which of it touches regulated data, and whether the answer is a vendor contract, a managed model in your cloud, or a GPU in your own rack. When you are ready to build, our LLM development team works on both managed and self hosted setups.