Model Evaluation & Threat Research (METR)

Model Evaluation & Threat Research (METR) is a nonprofit research institute headquartered in Berkeley, California, dedicated to scientifically measuring whether and when AI systems might threaten catastrophic harm to society. The organization develops evaluation methodologies, benchmarks, and open-source tools used by AI developers, governments, and other safety organizations worldwide. METR originated as ARC Evals, a team incubated within Paul Christiano's Alignment Research Center in 2022 when Beth Barnes joined from OpenAI to lead exploratory work on independent evaluations of cutting-edge AI models. After conducting some of the first independent evaluations of frontier models including GPT-4 and Claude, the team grew to become a majority of ARC's headcount. ARC Evals spun out as an independent organization in September 2023 and was formally renamed METR in December 2023, becoming a standalone 501(c)(3) nonprofit. The name METR references metrology, the science of measurement and its application. METR's research focuses on several core areas: evaluating broad autonomous capabilities of frontier AI systems, assessing AI's ability to accelerate AI research and development, studying behaviors that threaten evaluation integrity such as reward hacking and sandbagging, and measuring the potential for AI misuse in areas like cyberattacks. The organization has developed key benchmarks including RE-Bench for measuring AI performance on machine learning research engineering tasks, HCAST for testing autonomous software task completion, and the MALT dataset for studying evaluation integrity. METR also maintains Vivaria, an open-source evaluation platform used by organizations including the UK AI Safety Institute. METR has worked with leading AI companies to conduct pre-deployment model evaluations and contribute to system cards for models including OpenAI's GPT-4, GPT-4o, GPT-4.5, GPT-5, o3, and o4-mini, as well as Anthropic's Claude models and DeepSeek models. The organization has partnered with the UK AI Safety Institute, the US AI Safety Institute Consortium, and the European AI Office. Notably, METR does not accept compensation from AI labs for this evaluation work, maintaining independence through philanthropic funding. In October 2024, METR received approximately $17 million through The Audacious Project as part of Project Canary, a $38 million collaboration with RAND focused on developing and deploying evaluations to monitor AI systems for dangerous capabilities. The organization is also supported by the Sijbrandij Foundation, The Pew Charitable Trusts, Schmidt Sciences, the Survival and Flourishing Fund, Longview Philanthropy, and many individual donors. METR's research on AI capability timelines has been widely cited. Their time horizon analysis measures the task duration at which AI agents can reliably succeed, finding that frontier AI task completion ability has been doubling approximately every four to seven months. The organization has also conducted notable studies on AI's impact on developer productivity.