r/mlops
How we built LRU model eviction on top of SGLang to serve 139 open models across 23 agent tasks on one Kubernetes cluster (Apache-2.0)
- upvotes
- 3
- comments
- 5
Post
Highlighted: the lines this signal was extracted from
When you run an agent pipeline in production, the main LLM isn't the only thing eating VRAM. A single request often hits a prompt guard (Qwen3Guard-4B), an encoder (Qwen3-Embedding-4B), an entity/intent extractor (GLiNER2.5), a generator (Qwen3.8-27B-FP8), and a verifier (Granite-Guardian-2B). Deploying those the standard way means running vLLM for guards, N separate TEI containers for embeddings, a custom FastAPI service for GLiNER, and SGLang for generation, each locking down its own GPU memory slice even when idle. We built the Superlinked Inference Engine (SIE) under Apache-2.0 to collapse that fleet into one Helm chart and a single endpoint: - The 27B LLM stays pinned on its own GPU pool via SGLang so it never gets evicted. - Guards share one GPU pool and encoders share another. - Models load on demand, batch concurrent requests, and use Least Recently Used (LRU) eviction when VRAM gets tight. - If an individual SGLang process crashes, SIE unloads it and reloads cleanly on the next request. Moving high-frequency retrieval and extraction off per-token APIs onto shared GPUs reduces the cost of self-hosted embeddings to ~$0.50 per 1B tokens (vs ~$20 on managed APIs) and the cost of reranking to ~$8.50 per 1B tokens (vs ~$87 on Cohere Rerank).