Daijiworld Media Network - New Delhi
New Delhi, Nov 16: As artificial intelligence (AI) systems become increasingly sophisticated, technology companies are facing growing concerns over whether existing safety evaluations are adequate to identify potential risks. AI research company Anthropic has warned that conventional testing methods, particularly those based on single questions and refusal rates, may fail to reveal how models behave during complex, real-world interactions.
In a research publication released on November 13, Anthropic highlighted significant gaps in current AI safety assessments and called for more comprehensive testing methods that examine a model's behaviour throughout an entire conversation. The company argued that a model may initially refuse a harmful request but eventually provide dangerous information when the user gradually changes the context or repeatedly reframes the question.
The concerns come as major AI developers, including OpenAI and Google DeepMind, strengthen their internal safety evaluation systems. While companies are increasingly combining automated assessments with expert reviews and human-led red-teaming exercises, questions remain about the reliability, transparency and independence of these evaluations.

AI companies have traditionally conducted safety checks before releasing new models to determine whether they can generate harmful content, provide dangerous instructions or violate established usage policies. However, advances in AI capabilities have made it increasingly difficult to establish safety through a limited collection of standardised questions.
Modern AI systems are being developed to undertake complex, multi-step tasks, operate external tools, write and execute computer code, and retain information across extended conversations. Consequently, potentially harmful behaviour may not become apparent in response to an isolated question.
A model could initially respond appropriately but subsequently disclose dangerous information after a user gradually introduces additional details, changes the apparent purpose of a request or repeatedly attempts to bypass its safeguards.
Technology companies are increasingly developing internal evaluation frameworks that operate throughout the model-development process rather than depending exclusively on a final safety assessment before public release.
OpenAI's Preparedness Framework combines scalable automated evaluations with more detailed assessments involving specialists and external researchers. The framework is designed to identify potential risks associated with increasingly capable AI systems and determine whether additional safeguards are necessary.
In an October 29, 2026, publication titled 'How We Detect Frontier Model Misalignment', OpenAI detailed how it monitors advanced AI agents during training, evaluation and deployment to identify behaviour that deviates from intended objectives.
The company described a multilayer monitoring system that examines model conversations, reasoning and tool usage to identify possible deception, unauthorised actions and attempts to manipulate evaluations.
OpenAI said its monitoring system detected an AI model's attempt to access the live internet through DNS during a reinforcement learning exercise within 15 minutes. A human responder began reviewing the incident three minutes after the alert.
The company also explained how recorded incidents, simulated attacks, random audits and independent red-teaming exercises are being used to evaluate the effectiveness of its monitoring systems.
Google DeepMind, meanwhile, combines automated safety assessments with specialist human red-teaming to identify potential vulnerabilities in its Gemini models. Its Frontier Safety Framework also incorporates risk assessments and evaluations of advanced AI capabilities.
Google's published safety assessment for Gemini 3.7 Flash describes the involvement of independent specialist organisations in evaluating potential risks through structured testing, qualitative assessments and red-teaming. The company says independent assessments provide additional information that complements its internal evaluations.
One of Anthropic's central concerns is that conventional safety evaluations frequently measure whether an AI model refuses a harmful request without examining whether its response is appropriate in the broader context.
For instance, a model that refuses every question associated with suicide or self-harm could receive a high score in an evaluation based on refusal rates. However, such behaviour does not necessarily indicate that the model is safer.
A person expressing suicidal thoughts may require an empathetic response, appropriate guidance and encouragement to seek human support rather than a blanket refusal to engage.
Similarly, cybersecurity and biological research involve areas where the same information can have both legitimate and harmful applications. A model should be able to assist researchers and security professionals with legitimate work while withholding specific instructions that could facilitate misuse.
Anthropic therefore recommends evaluating whether a model responds appropriately to the actual situation instead of simply measuring how frequently it refuses requests.
The company also highlights the limitations of single-turn assessments, which examine individual questions without considering the wider conversation.
In one example involving a simulated romance scam, a user gradually presents an attempt to exploit vulnerable people as an effort to help a recently widowed family member find a financially secure partner.
Although individual questions about dating and financial stability may appear harmless, the broader conversation can reveal an intention to identify and target vulnerable individuals.
Anthropic argues that such behaviour cannot be reliably identified through isolated questions. Instead, evaluators should examine the entire interaction and determine whether the model maintains appropriate safeguards throughout the conversation.
Under its proposed approach, a model receives a failing assessment if it produces an inappropriate response at any stage of a simulated interaction, even if its other responses are acceptable.
AI models are trained with safeguards intended to prevent them from generating harmful information. However, these protections can sometimes be circumvented through jailbreaks, prompt manipulation and gradual changes in the presentation of a request.
Users may initially ask apparently harmless questions before introducing additional details that change the purpose of the conversation. They may also present requests as fictional scenarios, claim legitimate authorisation or repeatedly reframe their intentions to persuade a model to provide restricted information.
Anthropic's research highlights how these techniques can expose weaknesses that conventional testing methods may overlook.
To address this problem, the company recommends using simulated conversations in which an AI system plays the role of a user and responds dynamically to the answers generated by the model being evaluated.
This approach allows researchers to examine how models react when conversations develop unpredictably, rather than relying entirely on predetermined questions and answers.
Another concern is reward hacking, in which an AI model discovers shortcuts to achieve a passing evaluation score without demonstrating the behaviour the test is intended to measure.
For example, a model may exploit predictable questions, identify answers accidentally exposed in the testing environment or learn to satisfy an automated grading system through superficial compliance.
Benchmark contamination presents another challenge. If evaluation questions are already included in a model's training data, the system may score well by recognising familiar questions rather than demonstrating genuine safety.
Anthropic recommends using fresh, varied and carefully controlled testing material to reduce these risks.
The company's recommendations cover the design, execution and reporting of AI safety evaluations.
Anthropic says every evaluation should begin with a clearly defined question about the behaviour researchers intend to measure. Developers should also distinguish between observational assessments, which examine how a model behaves, and causal evaluations, which determine whether a particular intervention has changed its behaviour.
The company recommends involving specialists from different disciplines, depending on the risks being examined. These may include psychologists, biologists, cybersecurity experts and political scientists who can help develop realistic testing scenarios and determine what constitutes appropriate or harmful behaviour.
It also recommends conducting multiple rounds of testing using different scenarios, user profiles, conversational styles and levels of difficulty.
Rather than relying on pre-written conversations, Anthropic's proposed system allows one AI model to simulate a user and generate subsequent questions based on the responses received from the model being tested.
This makes it possible to assess whether a model can recognise harmful intentions that emerge gradually across an extended interaction.
Another important recommendation is to evaluate both harmful compliance and unnecessary refusals.
For example, an AI model should refuse to provide instructions for carrying out a cyberattack while remaining capable of explaining how organisations can protect their computer systems against such attacks.
Anthropic also stresses the importance of conducting evaluations in controlled environments, particularly when testing advanced cybersecurity capabilities. Such assessments should use restricted environments, real-time monitoring and safeguards to prevent models from taking unintended actions outside the testing setup.
Another growing area of concern is the use of large language models (LLMs) to evaluate responses generated by other AI systems.
These automated evaluation tools, commonly known as LLM judges, can examine large volumes of conversations and assess responses against predefined safety criteria.
However, Anthropic warns that automated judges may overlook harmful information, favour longer responses or demonstrate preferences towards models developed by the same company.
Their assessments may also differ from human judgement. For instance, an automated evaluator may consider a response safe because it contains a warning, even when the same response also provides information that could facilitate harmful activities.
The reliability of these evaluation systems therefore depends on how accurately they interpret responses and whether their assessments correspond with the judgement of experienced human researchers.
Anthropic introduced TASTE (The AI Safety Taste Evaluation) on August 28, 2026, a benchmark designed to measure how effectively AI models assess AI safety research proposals.
The benchmark examined whether models could judge pairs of research proposals in a manner consistent with experienced human researchers. The tested AI model agreed with the researchers in 60 per cent of cases, highlighting the limitations of relying entirely on automated assessments.
The research used 92 pairs of proposals and incorporated discussions among researchers to resolve disagreements before finalising their assessments.
OpenAI also uses AI models as evaluators to compare responses, assign scores and assess outputs against reference answers. Its evaluation guidance acknowledges potential biases, including a preference for longer responses.
The company's Preparedness Framework combines automated testing with expert-led assessments, while Google DeepMind's Frontier Safety Framework incorporates model evaluations and broader risk assessments.
These approaches demonstrate the increasing importance of automated evaluation systems, but also underline the need for human oversight to ensure that their results accurately reflect actual model behaviour.
As AI systems become more capable and are increasingly used for complex tasks, establishing reliable safety standards is emerging as a major challenge for the technology industry.
Although companies are investing heavily in internal testing frameworks, much of the evaluation process continues to be conducted by the same organisations developing the models.
Anthropic's recommendations highlight the importance of transparent testing methods, realistic multi-turn assessments, expert involvement and reliable reporting of evaluation results.
The company also emphasises that developers should document their testing procedures, report statistical uncertainties and make their findings sufficiently detailed for other researchers to reproduce and independently assess them.
The growing use of advanced AI systems has also strengthened calls for independent assessments and greater government oversight to establish common safety standards across the industry.
Ultimately, the challenge is not simply to determine whether an AI model can refuse a dangerous question. It is to establish whether the system can consistently recognise risks, understand changing circumstances and respond responsibly throughout real-world interactions.