SentinelLM
An LLM proxy that refuses to pass along what it can't vouch for.
Problem
Every team wiring an LLM into production rebuilds the same safety layer badly: a regex for PII, a vibes-based prompt-injection check, no audit trail, and no idea what the model actually returned to users last Tuesday. Meanwhile the safety layer itself becomes a single point of failure: when it breaks, the product breaks.
Approach
- Sit in the request path as a proxy so existing code changes one base URL, not its architecture.
- Run evaluations concurrently rather than in series, so safety costs latency once instead of seven times.
- Treat every decision as an auditable event, because 'the model said something bad' is only actionable with the record attached.
Architecture
Proxy
A FastAPI service fronting OpenAI, Anthropic, Gemini and Ollama behind one interface, routing every request through the evaluation pipeline.
Evaluators
Seven concurrent stages covering prompt-injection detection, PII redaction, hallucination and faithfulness scoring and toxicity filtering, with Redis caching on repeated content.
Observability
PostgreSQL-backed audit logging, WebSocket streaming for live monitoring, and a Next.js dashboard surfacing risk scores, flagged outputs and a structured human review queue.
Decisions
Fail-open, deliberately
A safety layer that takes the product down with it gets removed within a week. SentinelLM degrades to pass-through and logs loudly rather than blocking the request path. The alarm is worth more than the gate.
Concurrent evaluation
Seven sequential model calls would have made the proxy unusable. Running evaluators in parallel keeps the added latency close to the slowest single check rather than their sum.
Humans in the loop, on purpose
Automated scoring flags; it doesn't adjudicate. The review workflow exists because the interesting failures are the ones the classifier was unsure about.
Outcome
- One integration point covering four model providers.
- Full request-level audit history, streamed live rather than reconstructed after an incident.