No description available.
People
Updated 06/11/26By grantmaking.aicreator
Funding Details
- Start Date
- -
- End Date
- -
- Expected Duration
- -
- Funding Raised to Date
- -
- Annual Budget
- -
- Monthly Burn Rate
- -
- Current Runway
- -
- Funding Goal
- -
- Funding Stage
- -
- Fiscal Sponsor
- -
Project Details
Updated 06/11/26By grantmaking.aiProject summary
At some point programmers will be able to spin up a fleet of thousands of world class automated researchers—the resulting progress accomplished by this researcher cohort could be immense. Future poorly controlled self-improvement is probably the biggest issue in AI development and has the least amount of work on thinking about how to control it. This research project will use agent-based models to model the process of automated AI R&D / uncontrolled self-improvement (USI). The goal is to better understand potential runaway AI R&D processes and how to make these processes more controllable. Paul Salmon is a highly-cited expert on complex systems, human factors, and sociotechnical systems. He has also previously published work on how human factors can be used to manage the risks associated with AGI. Basically for this problem we don’t need extreme ML knowledge; instead we need knowledge of how to model systems and their dynamics and ways they can go awry.
What are this project's goals and how they be achieved?
The goal is to better understand potential runaway AI R&D processes and how to make them more controllable. It will create basic models of the dynamics and interactions between interventions that attempt to make this process more controllable.
How will this funding be used?
This funding will be used to pay graduate students who will help with the modeling.
Who is on the team and what's their track record on similar projects?
Paul Salmon is a professor at the University of the Sunshine Coast. He is a highly-cited expert on complex systems, human factors, and sociotechnical systems (google scholar). He has become interested in AGI risk and has previously published work on how human factors can be used to manage the risks associated with AGI.
What are the most likely causes and outcomes if this project fails? (premortem)
There is a chance that modeling runaway AI R&D processes requires a different skillset or is more complicated/intractable than initially expected. If that happens, the model may produce less interesting results or explain less than one might hope.
What other funding is this person or project getting?
None that I am aware of.
Grants Received– no grants recorded
Updated 06/11/26By grantmaking.aiDiscussion
Hey Paul, unfortunately Manifund won’t be able to fulfill this grant at this time. An unexpected influx of year-end regrants spent down the total pot of regrantor funding, meaning that we don't have enough left to fund a few of the last projects (like this one).
We’re so sorry if this created false expectations. Best of luck applying for funding elsewhere—hopefully Dan’s enthusiasm and support for your project will be of help, even if he couldn’t give you a grant directly.
Main points in favor of this grant
Despite AI capabilities quickly progressing towards human or superhuman level, the dynamics of an intelligence explosion or automated AI R&D haven’t been very thoroughly explored. If an intelligence explosion were to happen, humans would likely quickly lose control of the process, by default, unless precautions had been setup beforehand.
Paul Salmon has previously published highly impactful work in safety engineering and is familiar with the type of systems analysis needed to do this research. Paul is also interested in AI safety, having previously published on the topic of AGI risks.
Donor's main reservations
Whether agent-based models is the right approach remains to be seen.
Process for deciding amount
The amount regranted was comparable to other grants in the field.
Conflicts of interest
I will be helping with this project as well.