Licensed to be used in conjunction with basebox, only.
// installation
Service models
Applies to
Product: Server · Audience: Platform Operator
Service models are the AI services that basebox features rely on besides the language model. They are the middle layer: heavier than the platform, lighter and far more predictable than inference.
Which services
| Service | Used by | Component |
|---|---|---|
| Embeddings | Knowledge bases, RAG search | ragsrv |
| RAG processing | Document ingestion, chunking, retrieval | ragsrv, ragsrv-support |
| Document extraction | Reading PDF, Office, e-mail, images | ragsrv-support |
| OCR | Scans, photos, image-only documents | ragsrv-support |
| Speech-to-text | Dictation, audio transcription | ragsrv-support |
| TTS, long-term memory, others | As features are added | — |
Resource profile
As a general architecture principle, basebox platform + service models should have dedicated GPU capacity, separate from inference. Reasons:
- An upload, an OCR job or a one-hour transcription is bursty. On a shared GPU it competes with inference for VRAM and compute, and users see latency spikes in chat.
- Service models have a stable, predictable footprint; inference does not. Separating them makes sizing tractable.
Typical useful capacity: about 48 GB GPU memory, preferably around 96 GB for larger usage scenarios. This depends on which services are enabled and how heavily documents and audio are processed.
Architecture guidance, not a minimum requirement
Do not read these figures as a supported minimum. Exact values are pending DevOps confirmation and appear only on reference configuration pages once confirmed.
Dedicated GPU vs. MIG vs. CPU mode
Three ways to give service models their capacity:
- A dedicated physical GPU – e.g. the RTX PRO 6000 in the 2 × H200 + RTX PRO 6000 configuration. See Dedicated service GPU.
- MIG slices of a large GPU – e.g. one H200 split into four
1g.35gbinstances, one per service. This is the validated layout of the 2 × H200 reference configuration. - CPU mode –
ragsrvandragsrv-supportrun without GPU. Functional for RAG and OCR, noticeably slower; suitable for a CPU-only application server whose inference lives elsewhere. See Deployment topologies.
Placement
Service models can sit on the same node as inference (typical single-node Server) or on the application server when inference is on a separate GPU host. Either way, they should not share the inference GPU's memory.
Next step: Inference