Anthropic Gives AI Safety Evaluators Insider Access

Artificial Intelligence & Machine Learning , -Generation Technologies & Secure Development

Frontier artificial intelligence lab Anthropic will more deeply integrate independent model evaluators in a bid to quiet mounting concerns that AI firms haven't prioritized safety.

See Also: Webinar | Verifying AI Agent Activity Against Your Data Policies

The AI firm said in a blog post that personnel from independent evaluator Metr will obtain "employee-like access" to its models before deployment. The post follows Metr publishing guidelines on how it believes evaluators should assess unreleased frontier models.

Anthropic said six Metr employees have access to company hardware, office space and "access to internal tools and systems that are roughly on par" with its own risk assessment teams. Anthropic said it encouraged employees to information with the Metr team, except for "a small number of clearly defined categories," such as customer data.

Metr's team within Anthropic will stay small and be isolated from other Metr employees. The evaluator company will publish its own analysis of the risk levels and practices of Anthropic's models. Anthropic cannot veto the publication and only request redaction of some information.

Metr said alignment evaluations will take on two phases: surveys of observed incidents and a test to see how the models behave in simulated what-if scenarios to know if the developer's training process might be suppressing misalignment evidence.

The evaluation firm said it's important that assessors have access to saved states of model training to fully understand which part of training contributed to different model behaviors. It also needs codebase and infrastructure access "comparable to what an employee studying similar questions would typically have."

To do this, Metr said its evaluators need access to transcripts and other relevant datasets, the ability to interview lab staff and API access to the frontier models. Metr said that when it investigated the Hugging Face-OpenAI incident , it spent $400,000 in API credits in just six days.

"We estimate that a rigorous two-month investigation with this scope would have spent over $10 million in API credits," Metr said.

Anthropic said it plans to work with other independent evaluators, but because many of these partnerships are still in very early stages, the company will provide "these evaluators with more limited information, and they have more limited publication rights." The company is developing a tiered access approach for evaluators. The first tier gives access to conduct "narrowly scoped risk investigations," and the AI lab decides what they can tap or publish. The second tier expands information access, but Anthropic will still maintain the right to remove some categories of information. In this level, the independent labs can "publish their views on whether our redactions were justified." The third tier, the tier Metr currently sits, has employee-like access.

Executives like Anthropic's Dario Amodei and OpenAI's Sam Altman have committed to embedding third-party evaluators to assess the safety and alignment of their very capable models. A recent pledge signed by Anthropic, OpenAI at the White House commits the firms to implementing internal and external security processes, including possible partnerships with independent safety researchers.

One such lab is Metr, which became one of the more prominent third-party evaluators after Hugging Face and OpenAI tapped it to investigate how OpenAI's agents hacked the model repository. At the time, Metr and its partner, Redwood Research, acknowledged that while there was an overwhelming amount of data to review, the evaluators did not have direct access to OpenAI's infrastructure.