Oxford-based AI safety company developing concept extrapolation technology to enable AI systems to generalize human values and intent beyond their training data.
Oxford-based AI safety company developing concept extrapolation technology to enable AI systems to generalize human values and intent beyond their training data.
People
Updated 05/18/26Founder, CEO and Chief Technologist
Co-founder and Chief Mathematician
Funding Details
Updated 05/18/26- Annual Budget
- -
- Current Runway
- -
- Funding Goal
- -
- Funding Raised to Date
- $730,000
Org Details
Updated 05/18/26Aligned AI is an Oxford-area AI safety company incorporated in the United Kingdom in December 2021 (registered as ALIGNED AI LIMITED, company number 13798390, at 30 Upper High Street, Thame, Oxfordshire). The company was co-founded by Rebecca Gorman, a serial entrepreneur and AI researcher who has advised the EU, UN, OECD, and UK Parliament on responsible AI regulation, and Dr. Stuart Armstrong, who spent a decade as the Alexander Tamas Fellow in Artificial Intelligence and Machine Learning at the Future of Humanity Institute at the University of Oxford before co-founding the company in 2022. The company's central technical focus is concept extrapolation: solving the problem of how AI systems can understand and apply human-like concepts even when encountering scenarios outside their training distribution. This addresses what the team calls misgeneralization, where AI systems learn spurious correlations from training data and fail when presented with new situations. Their flagship algorithm, ACE (Algorithm for Concept Extrapolation), was demonstrated to surpass the CoinRun benchmark in September 2023, achieving a 72% success rate compared to the previous best of 59%. This work was widely covered as a significant milestone in applied alignment research. Additional products developed by the team include EquitAI (a gender bias mitigation tool for large language models), ClassifAI, and faAIr (an algorithm for measuring and ranking gender bias in foundation models). The faAIr and EquitAI tools won the CogX prize for Best Innovation in Mitigating Algorithm Bias in September 2023. In 2022, Aligned AI also released what was described as the first open-source ChatGPT jailbreak prevention prompt evaluator. Aligned AI is structured as a for-profit benefit corporation rather than a nonprofit, operating on a model that combines foundational research with commercial product development. The company has been funded through bootstrapping, angel investments, and UK government R&D tax credits. In September 2023, the company closed a $730,000 angel round at a $24 million valuation. The team is small (2-10 employees per LinkedIn), with known members including Gorman, Armstrong, Oliver Daniels-Koch, and Sophia Metz.
Theory of Change
Updated 05/18/26Aligned AI's theory of change centers on the premise that solving value extrapolation — the ability for AI to correctly extend human values to novel contexts not present in training data — is necessary and nearly sufficient for solving the alignment problem broadly. Their approach is explicitly aimed at scalability and distribution: rather than focusing solely on frontier AI labs (which may develop internal safety measures), Aligned AI develops alignment tools intended for widespread adoption across the AI industry, analogous to how CAD software improved engineering safety broadly. By making alignment solutions commercially viable and accessible to all AI developers, the company aims to raise the safety baseline for AI systems deployed in high-stakes scenarios such as autonomous vehicles, robotics, and content moderation.
Grants Received
Updated 05/18/26Projects
Updated 05/18/26Canvas is a family digital ecosystem from Aligned AI that embeds user-designed AI assistance in a personalised, offline environment, providing a powerful, safe and controlled AI experience that puts the user in the driver’s seat.
Discussion
No comments yet. Be the first to share your thoughts.