EN ▾
Get API key

LLM Inference APIGuide

What Is LLM Inference? Tokens, Latency and Cost

If you are asking what is LLM inference, the short version is this: it is the step where a trained language model reads your prompt and writes a reply, one token at a time. This page explains how that works and what it costs.

Updated:

Isometric illustration of tokens flowing into and out of a processing chamber, representing LLM inference

Key points

  1. Inference is running a finished model to produce output; training is the earlier, far more expensive step that created the model.
  2. A request has two phases: reading the prompt (prefill) and writing the answer token by token (decoding).
  3. Cost and speed both follow token counts, so input length, output length and context window matter more than anything else.
  4. With a hosted API you pay per token and skip the GPUs; with your own hardware you pay for the machine whether it is busy or idle.

Inference in one paragraph

A large language model is a file full of numbers, called weights. Training is the process that sets those numbers by showing the model enormous amounts of text. Inference is everything that happens afterwards, when you use the finished weights to get an answer. Every chat message you send to a model, every autocomplete suggestion and every API call that returns text is an inference request.

The distinction matters because the two jobs have different economics. Training happens once (or a few times) and needs a large cluster. Inference happens every time someone sends a prompt, so it is the part that scales with your users. When people talk about the cost of running an AI product, they almost always mean the cost of inference.

The model does not "look up" an answer. It computes a probability for every possible next token, picks one, appends it to the text, and repeats. That loop is the whole mechanism, and the rest of this page is a closer look at it.

What happens during one request

Take a single chat request and follow it from your code to the final character.

  1. Tokenization. Your text is split into tokens, which are chunks of characters. A token is often a short word or a piece of a longer one. A common rule of thumb for English is about four characters per token, but it varies by language and by content such as code.
  2. Prefill. The model processes all prompt tokens in one pass. This is parallel work and it builds the internal state the model needs to continue the text.
  3. Decoding. The model produces the first output token, then the second, and so on. Each new token depends on all the ones before it, so this phase is sequential. It is the slow part of most requests.
  4. Sampling. At each step the model has a score for every token in its vocabulary. Settings such as temperature and top_p decide how the next token is chosen from those scores. A low temperature picks the likeliest token almost every time; a higher one allows more variety.
  5. Stopping. Generation ends when the model emits an end-of-text token, when it hits a stop sequence you defined, or when it reaches the token limit.

Prefill and decoding stress hardware in different ways. Prefill is limited mostly by raw compute. Decoding is limited mostly by how fast the GPU can read the model weights from memory, because every single token needs to touch them. That is why a long answer takes visibly longer than a long question of the same size.

Tokens, context window and the KV cache

Everything in inference is measured in tokens. Your prompt, the system message, earlier turns of the conversation and the reply all count. The context window is the maximum number of tokens the model can hold at once, prompt and completion together. On this service the context window is 100,000 tokens, and the completion can use whatever the prompt leaves free.

To avoid recomputing the whole conversation for every new token, inference engines keep a KV cache: the intermediate keys and values for tokens already processed. The cache grows with the length of the context and takes GPU memory. This is the main reason long contexts are costly to serve, and the reason a server can only handle a limited number of long requests at once.

A practical consequence for you as a developer: a chat that resends the full history on every turn pays for that history again each time. If a conversation grows to 20,000 tokens, every new message carries those 20,000 input tokens with it. Trimming or summarizing old turns is the cheapest speed and cost optimization available to you, and it needs no changes on the server side.

Latency: why there are two numbers

People often ask how fast a model is, but there are two separate measurements.

  • Time to first token. How long you wait before any text appears. It includes network time, queueing at the server and prefill. Long prompts push it up.
  • Generation speed. How quickly tokens arrive after the first one. It depends on the model size, the hardware and how many other requests the same GPU is serving at that moment.

Streaming exists because of this split. With stream: true the server sends tokens as they are produced, so the user starts reading after the first token instead of waiting for the full answer. The total time is the same, but the interface feels much faster. For chat, roleplay and writing tools, streaming is nearly always the right default.

One detail to remember when you plan concurrency: a streamed request holds its slot until the stream ends, not until the first token arrives. If your key allows 8 concurrent requests, eight long streams in flight means the ninth has to wait or gets a 429 response.

What inference costs, with real numbers

Hosted inference is billed per token, with separate prices for input and output because output tokens take more work. This service charges $0.15 per 1M input tokens and $0.50 per 1M output tokens, from prepaid credit with no subscription. Here is how that works out for a few typical requests.

RequestInput tokensOutput tokensCost
Short chat message500300$0.000075 + $0.00015 = $0.000225
Chat with history2,000500$0.0003 + $0.00025 = $0.00055
Long document summary30,0001,000$0.0045 + $0.0005 = $0.005
Full 100,000-token prompt, short reply100,000500$0.015 + $0.00025 = $0.01525

The math is simple: divide the token count by one million and multiply by the price. For the chat-with-history row, 2,000 / 1,000,000 x $0.15 = $0.0003 and 500 / 1,000,000 x $0.50 = $0.00025, which adds up to $0.00055. A thousand requests like that cost $0.55.

Two billing details help when you budget. Failed or refused requests are not charged, and credit does not expire, so a top-up is not a clock that runs down. Every response carries an X-Balance-USD header, which lets your code watch the remaining balance without a separate call.

Hosted inference or your own GPU

You have two ways to get inference. You can rent or buy GPUs, install a serving engine and run a model yourself. Or you can call an API and let someone else operate the hardware.

Running it yourself gives you full control over the model files, the settings and the data path. The price is operational work: choosing hardware that fits the weights plus the KV cache, keeping drivers and engines updated, handling traffic spikes and paying for the machine during quiet hours. A hosted API turns that into a per-token bill. It is usually the quicker route when your traffic is uneven or you are still testing an idea.

The service described here is a hosted, OpenAI-compatible, text-only API. It runs one open-weight model, operated by an independent provider and tuned to answer without content refusals for lawful adult use. It is not the official API of any model vendor, and it does not generate images, audio or video, create embeddings or fine-tune models. If you need any of those, you need a different tool. If you need plain text generation with few refusals, the quickstart shows the setup, and the pricing page lists the top-up options.

Try an inference call yourself

The fastest way to make the concept concrete is to send one request and read the numbers it returns. After creating a key, this curl command sends a chat completion to the endpoint:

curl https://api.llminferenceapi.com/v1/chat/completions \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "uncensored",
    "messages": [{"role": "user", "content": "Explain the KV cache in two sentences."}],
    "max_tokens": 120
  }'

The JSON response includes a usage object with prompt_tokens, completion_tokens and total_tokens. Multiply those by the prices above and you have the exact cost of the call. The same thing in Python with the OpenAI SDK:

from openai import OpenAI

client = OpenAI(
    base_url="https://api.llminferenceapi.com/v1",
    api_key="YOUR_API_KEY",
)

resp = client.chat.completions.create(
    model="uncensored",
    messages=[{"role": "user", "content": "Explain the KV cache in two sentences."}],
    max_tokens=120,
)

u = resp.usage
cost = u.prompt_tokens / 1e6 * 0.15 + u.completion_tokens / 1e6 * 0.50
print(resp.choices[0].message.content)
print(u.prompt_tokens, u.completion_tokens, round(cost, 6))

New users can claim $0.50 of trial credit with one click on the key page after sign-up. It needs no card, lasts 7 days, and at the prices above it covers roughly two thousand short chat messages. If something returns an error you do not recognize, the troubleshooting article lists the usual causes.

Questions and answers

Is inference the same as running a model?

Yes. Running a trained model on new input to produce output is inference. Training, which adjusts the weights, is a different and earlier step.

Why is output more expensive than input?

Input tokens are processed together in one parallel pass, while output tokens are produced one at a time. Providers price the slower, sequential part higher, and this service charges $0.50 versus $0.15 per 1M tokens.

Does a longer context window make requests slower?

Usually yes. A longer prompt means more prefill work and a larger KV cache, so time to first token rises. You only pay for the tokens you actually send and receive, not for the unused part of the 100,000-token window.

Do I need a GPU to use an inference API?

No. The provider runs the hardware. You send HTTPS requests from any machine, including a laptop or a small server.

Your key is one form away

Create an account, copy the key, change the base URL. That is the whole setup.

Get API key