Self-Hosting Decision Models: Serving, Quantisation, Latency and Operations Training Course

Deployment Strategies

• Trade-offs between hosted APIs and self-hosted solutions.
• Patterns for on-premises, edge, and air-gapped environments.
• Single model vs. model portfolio approaches.
• Comparing costs per decision.

Hardware Sizing and Capacity Planning

• Memory footprint analysis by parameter count.
• CPU-only vs. GPU acceleration strategies.
• Planning for concurrency and throughput.
• Reserving headroom for increasing task complexity.

Serving Stacks

• Evaluating vLLM and SGLang options.
• Running llama.cpp and GGUF formats.
• Using Ollama for local development.
• Selecting stacks based on decision workload characteristics.

Quantization While Preserving Calibration

• Precision formats and their impact on memory usage.
• Assessing quality and calibration loss.
• Per-tensor and importance-aware quantization variants.
• Conducting acceptance tests after quantization.

Latency Optimization

• Distinguishing median from tail latency.
• Managing latency growth with longer contexts.
• Implementing warm starts and model caching.
• Batching multiple decision queries.

Edge and Constrained Environments

• Running small decision models on CPUs.
• Deployment on Apple Silicon and embedded systems.
• Offline operation and update strategies.
• Distributing weights in air-gapped networks.

Containerization and Scaling

• Designing images for model serving.
• Managing volumes and caches.
• Horizontal scaling and traffic routing.
• Setting resource limits and sharing GPUs.

Observability

• Tracking request, latency, and error metrics.
• Monitoring confidence distributions.
• Following decision outcomes and escalations.
• Alerting on calibration drift.

Versioning and Rollouts

• Managing model and adapter versions.
• Executing shadow and canary deployments.
• Using pinned model files for auditing.
• Implementing rollback procedures.

Security and Multi-Tenancy

• Implementing authentication and rate limiting.
• Ensuring tenant isolation for prompts and logs.
• Configuring network isolation and egress controls.
• Auditing decision logs.