Anthropic has published a guide to robust safety evaluations of AI models. According to the report from ad-hoc-news.de (editorial team boerse-global.de), published safety figures can differ sharply depending on testing method and deployment environment. The percentage figures common in the industry are often more marketing material than reliable benchmarks, because standardized protocols are lacking and providers use their own definitions and test sets.
Large gap between app and API
Anthropic illustrates the variation with its own systems: for Claude Fable 5.1, the rate of appropriate answers in test conversations on self-harm inside the Claude app was 94 percent after hidden system instructions were added. Via the application programming interface (API), the same model reached only 60 percent under identical test conditions, a difference of 34 percentage points. According to Anthropic, the cause lies not in the base model itself but in test conditions and the product layer such as system prompts and upstream safety filters.
Limits of single queries
Single queries are of limited significance. All recent models from the provider reached aggregated rates of harmless answers between 94 and 98 percent in single-question tests via the API. For test series, Anthropic recommends 16 policy areas in seven languages and at least 1,000 examples; for multi-turn dialogues at least 100 scenarios. In multi-turn conversations, according to Anthropic, confidence intervals can span over ten percentage points. Claude Opus 5 and Opus 5.5 scored 98 and 94 percent respectively in romance-scam tests on claude.ai, a difference the company described as not statistically significant.
New benchmarks for sensitive topics
Anthropic argues that for sensitive topics such as suicide, the mere refusal rate is a misleading criterion. A simple block can itself be a failure in crisis situations; what is needed is supportive feedback with pointers to human help services. In multi-turn tests, a single inappropriate response leads to failing the entire conversation. Open versus multiple-choice formats and the order of answer options also influence results. Anthropic further recommends isolated test environments without network access, after it temporarily suspended internet-based cybersecurity tests following incidents.
Assessment
For companies integrating models into their own applications via the API, it may be relevant that the protection level of the end-user app does not automatically carry over. Anyone comparing models should check under which conditions and with what sample size a figure was measured.
Sources
Editorially owned by Ideal Syka. Sources and method: Newsroom & method. Tips and corrections: ai@i6eal.de.
