Building a technical mechanism to assess risks, evaluate safeguards, and identify control gaps in agentic AI systems, enabling verifiable human oversight.
Building a technical mechanism to assess risks, evaluate safeguards, and identify control gaps in agentic AI systems, enabling verifiable human oversight.
People
Updated 06/10/26By grantmaking.aicreator
Funding Details
- Start Date
- -
- End Date
- -
- Expected Duration
- -
- Funding Raised to Date
- -
- Annual Budget
- -
- Monthly Burn Rate
- -
- Current Runway
- -
- Funding Stage
- -
- Fiscal Sponsor
- -
Project Details
Updated 06/10/26By grantmaking.aiProject summary
This project forms the first phase of a larger effort to develop technical control mechanisms that ensure agentic AI systems operate within human-specified constraints.
I (Francesca) recently founded Wiser Human, an AI Safety org, through the Catalyze incubator at Lisa (AI Safety office in London). Our mission is to combine our experience in risk management with technical AI control and scalable oversight techniques to drive practical, rapid progress to in AI control mechanisms in a world where timelines to advanced AI are short.
We believe the first step in achieving this is to identify the pathways most likely to lead to harm in emerging agentic AI systems, evaluate how effective existing safeguards are in mitigating these risks, and identifying which new control mechanisms and protocols we need to develop most urgently. This project addresses these challenges.
This project aims to develop a technical framework for human control mechanisms in agentic AI systems by creating a structured, repeatable methodology for risk and threat modeling. Our approach will produce both a methodology and a technical mechanism to apply it in practice which takes an agentic AI system use case description as input and outputs:
- Threat models based upon the agent’s level of autonomy, action surfaces, tool access, affordances, capabilities, propensities, and deployment scale.
- Existing safeguards that could be applied to address these threat models e.g. OWASP prevention and mitigation strategies, estimating their potential effectiveness, weaknesses and potential for subversion.
- Critical safeguard gaps, identifying areas where new control mechanisms are required (serving as candidates for blue teams to develop control protocols via control evaluations, as introduced by Redwood Research).
Our aim is for this project to accelerate the AI control research agenda by making it faster to AI Safety researchers to understand the most critical threats to agentic AI use cases, where gaps exist against existing safeguards, and which existing safeguards would likely be most vulnerable to subversion.
We will publish our findings, release open-source code, and engage with the broader AI safety community to drive adoption.\
What are this project's goals? How will you achieve them?
This project aims to develop a technical framework for human control mechanisms in agentic AI systems by creating a structured, repeatable methodology for risk and threat modeling. Our approach will produce both a methodology and a technical mechanism to apply it in practice which takes an agentic AI system use case description as input and outputs:
- Threat models based upon the agent’s level of autonomy, action surfaces, tool access, affordances, capabilities, propensities, and deployment scale.
- Existing safeguards that could be applied to address these threat models e.g. OWASP prevention and mitigation strategies, estimating their potential effectiveness, weaknesses and potential for subversion.
- Critical safeguard gaps, identifying areas where new control mechanisms are required (serving as candidates for blue teams to develop control protocols via control evaluations, as introduced by Redwood Research).
Our aim is for this project to accelerate the AI control research agenda by making it faster to AI Safety researchers to understand the most critical threats to agentic AI use cases, where gaps exist against existing safeguards, and which existing safeguards would likely be most vulnerable to subversion.
We will publish our findings, release open-source code, and engage with the broader AI safety community to drive adoption.
How will this funding be used?
This funding will directly support:
- Technical R&D: Designing, developing, and testing the risk and threat modeling methodology and mechanism to run this against agentic AI use cases.
- Empirical validation: Running case studies on real-world agentic AI use cases to test the methodology’s effectiveness.
- Publication and dissemination: Producing a research paper, a LessWrong blog post, and making the codebase openly available to AI safety researchers.
This funding will lay the foundation for future work, for us and other AI Safety researchers to conduct targeted control evaluations, and develop new control mechanisms resistant to agent subversion.\
Who is on your team? What's your track record on similar projects?
Francesca Gomez
Francesca has over 10 years of experience in operational and technology risk management, including her role as Head of Operational Risk at Tandem, a digital bank. She holds a BSc in Artificial Intelligence and an MSc in Human-Centered Computer Systems (Sussex University, UK). Francesca co-authored research on Safety Cases for Frontier AI with Arcadia Impact, published in December 2024, contributing to the AI safety space. She is interested in the threat model of losing control to AI and has published a project on ‘A Monte Carlo Simulation for estimating the risk of loss of control to AI’, and in the summer worked on a game built using the ArcWeave platform for people to explore choices and power structures in a world shaped by advanced Artificial Intelligence technologies.
Seb has 15+ years of software engineering experience, including developing risk monitoring systems for financial institutions like JP Morgan and HSBC. Additionally, he has 4 years of product management experience at Silicon Valley startups, combining technical expertise with strategic execution. He has led teams of 20+ software engineers on complex systems developments.
Seb and Francesca previously co-founded a fintech company, which participated in the Techstars and Barclays Accelerator programs. \
What are the most likely causes and outcomes if this project fails?
We believe the most likely causes of this project failing are:
- It proves unfeasible to develop a methodology that is effective for all agentic AI use cases i.e. it does not fully account for specific deployment scenarios, such as multi-agent systems or highly open-ended environments where an agent operates with significant autonomy across a broad range of domains.
- Lack of adoption, if AI safety researchers and the broader community does not engage with or apply our methodology.
If the project fails we expect the outcomes to be:
- Gaps in existing control mechanisms for agentic AI systems are more likely to go unnoticed, increasing systemic AI risks, particularly in high-autonomy and multi-agent deployment scenarios.
- Slower iterative testing and development of AI control protocols, plus a reduced ability to prioritise the most critical areas where safeguards are lacking.\
How much money have you raised in the last 12 months, and from where?
5,000 from a private donor which has allowed us to start initial work on the design of the methodology.
Achieving our minimum funding allows us to develop the methodology for threat modelling but limits our ability to develop a technical means of running this. Full funding allows us to deliver the set of deliverables outlined above.\
Grants Received
Updated 06/10/26By grantmaking.aiDiscussion
Approving this grant as in line with our mission for funding AI safety technical research!
Apologies for the delay here; we were waiting for Ryan or others with context to weigh in on the grant rationale, which we still hope to have; but we want to unblock Francesca in the meantime.
Thank you! Ryan was introduced to the project while he was advising on the Catalyze Impact AI Safety incubator. Looking forward to kicking off now!
I submitted the info to withdraw funds at the end of April but not sure how to track progress of this. I imagine being based in UK there are some extra checks, but let me know if you need any additional info from me to process.
@wiserhuman apologies for the delay on this; I've just put in the wire!
Good stuff ill be watching too!
[Progress update]
What progress have you made since your last update?
We began by drafting a methodology for agent-centred threat assessments that considered an agent’s autonomy, action surfaces, tool access, affordances, capabilities, propensities, and deployment scale. The goal was to recommend safeguards based on a theoretical analysis.
However, the release of Anthropic’s Agentic Misalignment study (Lynch et al., 2025) highlighted two key points:
- Misaligned behaviour only emerged when agents experienced goal conflicts or autonomy threats - our draft methodology did not incorporate this key consideration of 'triggers'.
- Without empirical testing, recommended safeguards might not actually effectively prevent such behaviours.
In response, we updated our approach. Rather than only theorising about safeguards, we first designed a set of preventative mitigations for agentic misalignment, inspired by insider risk controls, and tested them on the original Agentic Misalignment scenario to understand their real-world effectiveness.
Part 1 – Empirical Study We conducted an empirical evaluation of five mitigation types across 10 models and 66,000 trials, using Anthropic’s original framework. We are now finalising the written report, which will be published in the next 2–3 weeks.
Part 2 – Practical Outputs To make the results directly useful to AI safety researchers and developers and honour the original aims of this project we will also release:
- A repeatable methodology for identifying threat models and likely pathways to agentic misalignment harm in agentic AI systems.
- A mapping of existing safeguards for both preventative (tested empirically) and detective (monitoring) controls.
- An analysis of safeguard gaps and potential harm areas given today’s available control portfolio.
What are your next steps?
Publish empirical testing results (in research paper) in next 2-3 weeks. We will also release our code so others can replicate or test different mitigations.
The part 2 outputs will follow after and will likely be in a blog post format.
Is there anything others could help you with?
Feedback once published and sharing with others interested in this topic.
I’m excited to share that Part 1 of our project is now complete: our empirical study testing mitigations for agentic misalignment across 10 models and 66,600 trials, using Anthropic’s Agentic Misalignment scenario framework (Lynch et al., 2025).
We designed and tested controls, adapted from insider-risk management, that steer AI agents toward escalation under stressors such as autonomy threats, substantially reducing blackmail across ten models without retraining or fine-tuning.
Key findings:
- Controls adapted from insider-risk management significantly reduced blackmail rates across all ten models, though not entirely.
- Escalation channels and compliance cues steered agents toward safe, compliant actions without altering base model weights.
- Because these mitigations generalised across model families, they may form a low-cost, model-agnostic defence that reduces the number of harmful actions needing to be caught by monitoring.
- The study also surfaced new failure modes and biases detectable only through cross-model and counterfactual analysis.
We believe that environment shaping, where agents act to preserve autonomy or goal achievement over longer time horizons, is a credible threat model requiring deeper study.
📄 Research page: https://www.wiserhuman.ai/research ✍️ Blog summary: https://blog.wiserhuman.ai/p/can-we-steer-ai-models-toward-safer 💻 Code and dataset: https://github.com/wiser-human-experimental/agentic-misalignment-mitigations/tree/public-mitigations-v1 📘 Paper (preprint): https://arxiv.org/abs/2510.05192
This is an early proof of concept, and we hope to explore further how steering controls can form part of a layered defence-in-depth approach.
[Final report]
Description of subprojects and results, including major changes from the original proposal
We have now completed our project, with a paper focusing on implications for designing escalation channels for agents to follow at the point of task-rule conflict added to Arxiv: https://arxiv.org/abs/2510.05192.
Code from our research is available here: https://github.com/wiser-human-experimental/agentic-misalignment-mitigations
Wiser Human will continue to do further empirical work in this area, which can be followed here: wiserhuman.ai,
This will include building on tour understanding of threats from agents with strong preferences for task completion, even when requiring unsanctioned behaviour.
Spending breakdown
All spend was used on compute: mainly APIs for the 60,000 tests.
I hope you'll have enough money to launch your project, it looks promising!