Center for AI Risk Management & Alignment (CARMA)

The Center for AI Risk Management & Alignment (CARMA) is a research and policy think tank dedicated to helping society navigate the complex and potentially catastrophic risks arising from increasingly powerful AI systems. Its mission is to lower the risks to humanity and the biosphere from transformative AI by grounding risk management in rigorous analysis, developing policy frameworks that squarely address AGI, advancing technical safety approaches, and fostering global perspectives on durable safety. CARMA was founded in November 2022 by Richard Mallah, who serves as Executive Director. Mallah brings over two decades of machine learning and AI experience, has focused on advanced AI safety since 2010, and previously served as Director of AI Projects at the Future of Life Institute from 2014 to 2022. He continues to serve as Principal AI Safety Strategist at FLI on a part-time basis. The organization is virtual-first, with staff clusters in Berkeley, Cambridge (UK), Washington DC, and London. CARMA operates five distinct research programs. The Comprehensive Risk Assessment program develops frameworks and tools for top-down threat surface scanning and risk propagation modeling across society, including the Probabilistic Risk Assessment (PRA) for AI framework published in 2024. The Public Security Policy program conducts targeted advocacy, policy development, and stakeholder engagement to address AI-related threats to critical infrastructure and national preparedness. The Geostrategic Dynamics Mechanisms program uses game theory and mechanism design to propose frameworks for multiscale international cooperation on AI safety. The Offense/Defense Dynamics program investigates how AI capabilities affect the balance between harmful and protective uses. The Alignment of Dynamical Cognitive Systems program explores architectural strategies to make powerful AI systems safer while respecting stakeholder needs and contextual tradeoffs. Notable outputs include the PRA for AI framework and accompanying workbook (released November 2024), a comprehensive technical paper on probabilistic risk assessment methodology (June 2025), an interim policy brief on national preparedness in the age of AI (July 2025), and a paper on platform-mediated coordination for AI agents (November 2025). CARMA is a fiscally sponsored project of Social & Environmental Entrepreneurs, Inc. (SEE), a California 501(c)(3) nonprofit public benefit corporation.