← AI Knowledge Home
01 · APPLICATION 02 · API 03 · SERVING 04 · RUNTIME 05 · MODEL + MEMORY 06 · GENERATION 07 · API RESPONSE 08 · USER
10 · MODEL SERVING & LOCAL AI
How Does a Model Become a Running AI Service?
A model file cannot serve users by itself. It needs a runtime, memory, hardware, and an interface that applications can call.
MODEL+RUNTIME+HARDWARE+SERVING INTERFACE=RUNNING AI SERVICE
THE PROBLEM THIS MODULE SOLVES
QUESTION
“I downloaded a Qwen model. Why can’t my app use it directly?”
BECAUSE A MODEL FILE IS NOT A SERVICE
A model file does not by itself:
- listen for requests
- manage memory
- run inference
- expose an API
- handle concurrent users
- return application responses
The serving stack turns model weights into a usable service.
THE BIG PICTURE
Serving Stack
RESOURCE CONTROLS
Quantization
Context size
KV cache
Model size
PERFORMANCE
Latency
Tokens per second
USER / APPLICATIONSends a request
↓
APIReceives application calls
↓
MODEL SERVINGMakes the running model available
↓
RUNTIMEOllama / vLLM execute inference
↓
MODELLearned model weights
↓
HARDWARE / MEMORYGPU · VRAM · Unified Memory
DEPLOYMENT ENVIRONMENTLOCAL vs CLOUD
ONE REQUEST THROUGH THE STACK
“Explain RAG.”
Receives the request.
Routes it to a running model.
Executes inference.
Processes prompt tokens.
Produces output tokens.
Returns the result.
Sees the answer.
LATENCY
How long the full request feels.
TPS
How fast tokens are generated.
WHY HARDWARE MATTERS
Why can’t every model run on every computer?
RUNTIME MEMORY=MODEL WEIGHTS+KV CACHE+TEMPORARY ACTIVATIONS+CONTEXT MEMORY+RUNTIME OVERHEAD
MODEL FILE SIZE
What is stored on disk.
RUNTIME MEMORY
What is needed while the model is actually running.
A model that fits on disk may still exceed available memory.
QUANTIZATION
Lower precision, different resource trade-offs
FULLER PRECISION
↓Larger memory requirement
QUANTIZATION
↓Reduces numerical precision
LOWER PRECISION REPRESENTATION
Smaller resource requirement
Possible trade-offs include quality, speed, and compatibility. Quantization is not simply “compression.”
DEPLOYMENT ENVIRONMENT
Local vs Cloud
LOCAL
Runs on your hardware.
Good fit whenLocal control matters, hardware is available, or offline operation is useful.
ConstraintsMemory, setup, and maintenance.
CLOUD
Runs on provider infrastructure.
Good fit whenLarge managed infrastructure, scaling, or easy API access is preferred.
ConstraintsNetwork, usage cost, and governance considerations.
Neither is universally better.
UNDERSTAND EACH LAYER
Four roles in a running AI service
ACCESS
API · Model Serving
Lets applications reach the running model. Example: an app sends a chat request.
RUN
Runtime · Ollama · vLLM
Executes inference using model weights.
RESOURCE
Quantization · GPU / VRAM
Determines what can fit and run.
PERFORMANCE
Latency · Tokens per Second
Describes waiting time and generation speed.
KEY DISTINCTIONS
What not to confuse
MODEL vs RUNTIME
MODELLearned weights.
RUNTIMESoftware that executes them.
RUNTIME vs MODEL SERVING
RUNTIMERuns inference.
SERVINGMakes it available to applications.
QUANTIZATION vs FINE-TUNING
QUANTIZATIONChanges numerical precision.
FINE-TUNINGChanges learned behavior through training.
LATENCY vs TPS
LATENCYHow long you wait.
TPSHow fast output is generated.
REMEMBER THIS
A model becomes usable when a runtime serves it through an API on real hardware, while memory, quantization, and performance constraints determine how well it runs.