The problem
The client sells a security analytics platform that groups related alerts and writes a short plain-English summary of each group for analysts. Summaries were generated through a hosted model API. Volume had grown to around 2.3 million summaries a day, and inference had become one of the largest lines in their cost of goods sold. Several enterprise prospects also asked that log data never be sent to a third-party model provider, which the hosted approach couldn't satisfy.
What we actually did
Deciding what to move, and what not to
We analyzed their traffic and found that alert summarization was high volume, narrow, and formulaic, with a fixed input structure and a short output, while a much smaller set of features (an analyst chat assistant, report drafting) needed the strongest available model. We moved only summarization. The analyst assistant stayed on the hosted API.
Choosing and adapting the model
We evaluated several open-weight models against 2,000 alert groups with summaries rated by the client's own analysts. A mid-sized open-weight model, lightly fine-tuned with LoRA on 15,000 examples of approved summaries, matched the hosted model's quality ratings on this specific task. We quantized it to reduce memory use, and validated on the same set that quality didn't drop.
Serving it efficiently
The model runs on vLLM, which uses continuous batching and paged attention memory management to serve many concurrent requests per GPU. It's deployed on Kubernetes in the client's own cloud account, with GPU node pools that autoscale on queue depth rather than CPU, scale down overnight when alert volume drops, and mix reserved capacity for the steady baseline with on-demand capacity for peaks.
Observability and a safe fallback
We added per-request metrics for latency, tokens, GPU utilization, and cost, plus a weekly quality sample scored against the analyst-rated set. If the self-hosted service is degraded, requests fail over automatically to the hosted API so analysts never see missing summaries.
Challenges and tradeoffs
- GPU availability. On-demand GPU capacity isn't always available at peak. Reserved baseline capacity plus the hosted-API fallback meant a capacity shortage degraded cost, not availability.
- Model upkeep is now the client's job. We set up a repeatable fine-tuning and evaluation pipeline and trained the client's platform team to run it, so updating the model is a routine release rather than a project.
- Cold starts. Loading model weights onto a new node takes time, so the autoscaler keeps a warm minimum and scales ahead of known daily peaks.
Results
Cost per summary fell 52% compared with the hosted API, including GPU, storage, and an allowance for the platform team's time. p95 latency at peak load is 1.8 seconds, and analyst quality ratings on the weekly sample are unchanged. The client can now offer enterprise customers a deployment where log data never leaves their own cloud account, which removed a blocker in several sales cycles.
What we'd do differently
We would set up the analyst-rated evaluation set before any infrastructure work. It was the single artifact that made every later decision, from model choice to quantization, a measurement instead of a debate.