Skip to content
Copyright © 2022-2026 basebox GmbH, all rights reserved.
Licensed to be used in conjunction with basebox, only.

// installation

Inference

Applies to

Product: Server · Audience: Platform Operator

Inference is the language model – LLM, VLM or reasoning model – answering the user. It is the layer that determines a Server's GPU requirement, and it is separate from basebox itself.

How basebox talks to inference

AISRV calls the model over an OpenAI-compatible API (/v1/models, /v1/chat/completions). That boundary is what makes the layer separable: basebox does not care whether the endpoint is

  • the bundled vLLM deployed by the Helm chart on the same node,
  • a separate GPU server in the customer's network,
  • another Kubernetes cluster, or
  • an approved external provider.

Users and browsers never talk to inference directly – only AISRV does. The endpoint must require an API credential and use TLS when traffic crosses hosts. The worked example with Secrets, Helm values and validation steps is on Deployment topologies; the step in the self-installer path is Connect inference.

What drives sizing

Factor Effect
Selected model and size VRAM for weights; larger models need more GPUs or more aggressive quantization
Quantization Trades VRAM and speed against quality
Context length KV-cache grows with context; long-context use is memory-hungry
Expected concurrency More simultaneous users → more KV-cache and compute
Required latency Tighter latency → less batching headroom → more hardware per user
Usage pattern Short Q&A vs. long document analysis vs. tool-heavy agent loops

This is a complex topic with no single answer. This documentation does not primarily answer "what hardware should I buy?" – separate sizing material exists for that. Here we answer: given this infrastructure, how do I deploy and operate inference correctly?

Multi-GPU and beyond

One model can span several GPUs (tensor parallelism – e.g. two H100 as a pair in the 4 × H100 configuration); several models or instances can run side by side; inference can move to separate nodes. These layouts are collected under Advanced architectures and are documented as DevOps confirms them.

Failure behaviour

basebox does not silently fall back to another model when the configured endpoint is offline. It exposes the failure and recovers when inference is ready again. Plan monitoring and alerting for the inference endpoint accordingly – see Operations → Monitoring.

Where to continue

Next step: Resources & scaling