EN ▾
Get API key

LLM Inference APIGuide

LLM Inference Framework Comparison vs Hosted API

This llm inference framework comparison covers the engines you can run yourself (vLLM, llama.cpp, Ollama, TensorRT-LLM, SGLang) and sets them against a hosted OpenAI-compatible API. The right pick depends on your hardware, your traffic and how much operations work you want.

Updated:

Illustration of several server towers and one cloud node, comparing self-hosted inference engines with a hosted API

Key points

  1. vLLM, SGLang and TensorRT-LLM are built for GPU servers with many simultaneous users; llama.cpp and Ollama are built for a single machine, including laptops.
  2. Self-hosting gives control over weights and data path, but you pay for the hardware and the upkeep even when idle.
  3. A hosted API costs nothing at rest and is billed per token, which suits uneven or early-stage traffic.
  4. Most of these tools speak the OpenAI API format, so moving between them usually means changing a base URL.

The two questions that decide most of it

Before comparing engines, answer two questions. First, do you need to choose the model yourself? If you must run one specific set of weights, a hosted service that offers a single model cannot help you, and a framework is the only way. Second, how steady is your traffic? A GPU that sits busy twenty-four hours a day is a good use of money. A GPU that handles a burst in the evening and idles the rest of the time is not.

If the answer to the first is "any capable model that does not refuse ordinary adult requests is fine" and the answer to the second is "uneven", a hosted API is likely the simpler choice. If you need a particular fine-tune, have steady heavy load or must keep every byte on your own machines, look at the frameworks below.

The frameworks, one by one

vLLM

An open-source serving engine for GPUs. It is known for continuous batching, which keeps the GPU busy by adding new requests to a running batch, and for a paged approach to the KV cache that wastes less memory. It ships an OpenAI-compatible HTTP server. It is a strong default when you have a datacenter-class GPU and many concurrent users. Setup involves CUDA drivers, matching versions and model memory planning.

SGLang

Another open-source serving framework aimed at high-throughput GPU serving, with features for structured output and prefix reuse. It sits in the same category as vLLM, and teams often benchmark both on their own workload before picking one. Treat any published benchmark with caution: results change with model, hardware, prompt length and version.

TensorRT-LLM

NVIDIA's library for compiling and running models on NVIDIA GPUs. It can extract a lot of performance from that hardware, at the price of a heavier build and deployment process and a tie to one vendor's stack. It makes sense when you run a fixed model at scale on NVIDIA cards and have engineers to maintain it.

llama.cpp

A C/C++ implementation that runs quantized models in the GGUF format on CPUs, Apple Silicon and many GPUs. It includes llama-server, which exposes an OpenAI-style endpoint. Quantization shrinks the memory needed, at some cost in output quality that depends on how aggressive it is. It is the practical choice for a laptop, a home server or an offline tool.

Ollama

A friendly wrapper for running models locally: you pull a model by name and it handles downloads, storage and a local server, including an OpenAI-compatible endpoint. It is excellent for experiments and personal use. It is aimed at one user or a few, not at serving a busy public app.

Side by side

OptionBest fitYou manageCost model
vLLM / SGLangMany users on server GPUsDrivers, GPUs, scaling, updatesHardware, whether busy or idle
TensorRT-LLMFixed model at scale on NVIDIA cardsBuilds, deployment, tuningHardware plus engineering time
llama.cppLaptops, small servers, offline useModel files, quantization choiceYour own machine
OllamaLocal experiments, personal toolsModel downloads, disk spaceYour own machine
Hosted APIUneven or early traffic, no GPU opsAn API key and your codePer token, nothing at rest

The table leaves out speed figures on purpose. Throughput depends on the model, the quantization, the prompt length and the batch size, so a number from someone else's setup rarely carries over to yours.

What a hosted API gives up, and what it gives

The hosted option described on this site is a text-only, OpenAI-compatible API running one open-weight model, operated by an independent provider and tuned to answer without content refusals for lawful adult use. The model ID is uncensored. It supports streaming, tool calling and JSON mode, with a 100,000-token context window.

What you give up is clear. You cannot pick or swap the weights. You cannot fine-tune. There are no embeddings, no image, audio or video generation, and the service is bound by rate limits: a full key allows 8 concurrent requests and 300 requests per minute, and a trial key allows 2 and 60. It is also not the official API of any model vendor.

What you get in return is zero infrastructure. There is no GPU to buy, no engine to update and no idle cost. You pay $0.15 per 1M input tokens and $0.50 per 1M output tokens from prepaid credit that never expires, and requests that fail or are refused are not charged. The pricing page has the top-up amounts and bonuses.

Costs people forget when they self-host

The price of a GPU is only the first line of the budget. Teams that move from an API to their own servers tend to be surprised by the second and third lines.

  • Memory headroom. The weights are not the only thing that needs GPU memory. The KV cache grows with every concurrent request and every token of context, so a card that loads the model comfortably can still run out of room at 20 simultaneous users with long prompts.
  • Idle time. Overnight and weekend hours cost the same as peak hours. Autoscaling helps, but starting a GPU node and loading large weights takes minutes, not seconds.
  • Upgrades and breakage. Serving engines release often. A driver update or a new engine version can change behavior, and someone has to test it before production sees it.
  • Safety and abuse handling. With your own deployment, filtering policy, logging and abuse response are yours to design and to answer for.

None of this means self-hosting is a bad idea. It means the comparison should include staff time, not only the hourly price of a machine.

A cost check you can run on your own numbers

The honest way to compare is to price your actual workload. Say an app uses 100M input tokens and 20M output tokens in a month.

  • Input: 100 x $0.15 = $15.
  • Output: 20 x $0.50 = $10.
  • Total: $25 for the month.

Now put the monthly cost of your alternative next to that: the server or GPU rental, power if it is your own hardware, and the hours somebody spends keeping it running. Check current prices for hardware or cloud GPUs yourself, since they change often. If your alternative costs less than the API bill at your volume, and you can keep the machine busy, self-hosting wins on money. If the API bill is lower, or if your load is spiky, the hosted route wins.

The reverse case matters too. At 10 times that volume, the same maths gives $250 a month, and a dedicated GPU running a model you chose may start to look better, particularly if you need a model this service does not offer. Re-run the calculation when your usage changes.

Same client code on either side

A useful property of this category is that vLLM, llama.cpp, Ollama and most hosted services all expose the OpenAI request format. Your application code does not need to know which one is behind the URL. Only the base URL, the key and the model name change. Here is the client for the hosted endpoint:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.llminferenceapi.com/v1",
    api_key="YOUR_API_KEY",
)

resp = client.chat.completions.create(
    model="uncensored",
    messages=[
        {"role": "system", "content": "You are a helpful writing assistant."},
        {"role": "user", "content": "Give me three opening lines for a noir short story."},
    ],
    temperature=0.8,
)
print(resp.choices[0].message.content)

If you later move to your own server, you edit the base URL and the model name and leave the rest alone. That makes it low-risk to start on the hosted side, measure your real token volumes, and decide about hardware with data. The quickstart covers streaming, tools and JSON mode, and a trial key is available from the key page.

Which one should you pick

  • You want to try ideas this week: use a hosted API, or Ollama if you have a capable computer and want everything local.
  • You need an exact model or a private fine-tune: vLLM or SGLang on rented or owned GPUs.
  • You run a fixed model at scale on NVIDIA hardware with a platform team: TensorRT-LLM is worth testing.
  • You need offline operation or a tiny footprint: llama.cpp with a quantized model.
  • You need a text model with few refusals and no hardware work: the hosted endpoint here, with the limits above in mind.

Questions and answers

Is vLLM better than llama.cpp?

They target different setups. vLLM is built for serving many users on server GPUs, while llama.cpp is built for running quantized models on a single machine, including CPUs and laptops.

Can I use the OpenAI SDK with these frameworks?

Generally yes. vLLM, llama.cpp's server and Ollama all offer OpenAI-compatible endpoints, and so does this hosted API. You change the base URL, key and model name.

When does self-hosting become cheaper than an API?

When the monthly cost of your hardware and upkeep is below what your token volume would cost on the API, and you can keep the hardware busy. Price your own workload at $0.15 and $0.50 per 1M tokens to find out.

Does the hosted API let me choose the model?

No. It serves one model under the ID uncensored. If you need specific weights, you need to run them yourself.

Your key is one form away

Create an account, copy the key, change the base URL. That is the whole setup.

Get API key