BELLS-O (Benchmark for the Evaluation of LLM Supervision Systems – Operational) is CeSIA's latest installment in the BELLS line of work, a family of independent benchmarks the team has built since 2024 to test how well safety systems catch harmful or manipulated AI outputs. This installment was developed with funding from the UK AI Security Institute. It builds on our earlier paper, The Bitter Lesson of Misuse Detection (arXiv:2507.06282), and was accepted at the Trustworthy AI for Good workshop at ICML 2026, presented in Seoul on July 10.
LLM supervision systems (input/output moderation filters and jailbreak detectors) are the primary safeguard against misuse in deployed AI applications, yet existing benchmarks are often biased towards the vendor, they omit cost and latency, and they rarely compare specialized guardrails against repurposed generalist LLMs. BELLS-O, meanwhile, is the first independent operational benchmark of these systems, evaluating 28 systems from 17 providers, including every major specialized guardrail (LlamaGuard-4, ShieldGemma-2, Lakera Guard, among others) and frontier generalist models repurposed as supervisors, jointly on detection rate, false-positive rate, latency, and monetary cost.
The benchmark covers input/output moderation across 11 harm categories and jailbreak detection across 13 attack techniques. Mapping the Pareto frontier reveals tradeoffs that depend heavily on use case. On content moderation, specialized supervisors are operationally dominant: top systems match frontier LLMs on detection (about 95% versus 94%) at comparably low false-positive rates (≤2%), while running 5 to 10 times faster and roughly 10 times cheaper. On jailbreak detection, the tradeoff shifts: frontier LLMs achieve higher detection and lower false-positive rates, but at 10 to 50 times the cost and 5 to 10 times the latency. The benchmark, framework, leaderboard, and datasets are released as the first vendor-neutral basis for choosing safeguards under real deployment constraints.