Project summary

Arkose is an AI safety fieldbuilding organization that supports mid-career ML professionals to become involved in research mitigating large-scale risks from AI.

Arkose’s main offerings are one-on-one calls, where we introduce AI safety and discuss ways to get involved, and a public website (with key AI safety papers and opportunities to get involved, e.g. funding and jobs). After calls, we facilitate researchers’ connections in the field by direct introductions to experts and through our public database of AI safety researchers.

Arkose has conducted calls with 223 researchers and engineers since December 2023. Of those who gave feedback on the call (n=115), 77% said Arkose accelerated their involvement in AI safety, most commonly by 1-3 months (n=42). 3 researchers said there was a “significant trajectory change” following the call such that they were unable to assess the magnitude of the acceleration. You can view additional impact analysis as graphs in this doc.

Over the next year, we’re interested in careful growth to increase our call volume and quality, and expanding the scope of our outreach (e.g. outreach to mid-career cybersecurity professionals).

For potential research leads, we think these are encouraging per-call statistics, especially with the volume we’re able to achieve through our one-call approach.

What are this project's goals? How will you achieve them?

We hope to expedite the engagement of mid-career ML professionals with research on large-scale risks from advanced AI, particularly as research leads. We think this is a neglected niche in the fieldbuilding landscape: we’re targeting people who can contribute quickly to the literature, and focussing on outreach to those with low prior experience in AI safety to maximise the counterfactual impact of each call.

To this end, we plan to continue and refine our current programs:

• Calls – 66% of our calls are from direct email outreach to PhDs, postdoctoral researchers, professors, and industry professionals (the remainder are referrals from individuals and organizations interested in AI safety). On calls, we discuss the reasons to work on catastrophic risk, discuss research relevant to the researcher’s interests, and support researchers to take next steps in the space by connecting them with tailored lists of opportunities or introductions to other researchers. We then set goals on call and provide ongoing support towards contributing to the field.
• Website – we think our website is the best public resource for people with machine learning experience who are interested in starting work in AI safety. We maintain a list of key papers with regular input from our advisors and experts, a list of top opportunities to get involved (funding, jobs, compute, etc.), and a database of researchers, recently adapted from Sören Mindermann’s now-deprecated database.
• Advisors – we have a group of 38 experts in AI safety who we connect people with after calls, when there’s a good fit. The goal of these connections is for the experts to provide a more technical overview of the AI safety literature or discuss specific research projects in more depth. We match advisors to people we speak with based on research interests and career context to increase the likelihood of a productive conversation.

We think a further year’s funding would allow us to conduct at least 300 calls with top researchers worldwide while enabling us to improve the quality and efficiency of our programs. To maintain this pace, we plan to improve our internal systems and source new leads for cold outreach.

We’re not excited about rapid growth at present. We think there’s a relatively small population to reach out to here, and we want to avoid portraying AI safety as junior or inexpert to experienced professionals. Therefore, we’re prioritising high call quality over scaling volume rapidly. By providing a single high-quality call with someone familiar with both the AI safety literature and the opportunities in the space, we think we strike a good balance of providing a professional and relatively in-depth introduction to AI safety while retaining a high-volume approach.

Instead of rapid growth, our strategy over the next 18 months focuses on improving our programs to be the very best resources available for mid-career professionals, and expanding into new areas while ensuring every call is a high-quality and professional experience. In particular, we’re interested in:

• Alternative outreach methods. We’ve experimented with increasing referrals from other places in the AI safety community, but concluded that much of our counterfactual comes from direct outreach. We’re interested in other methods of reaching mid-career professionals who would not otherwise interact with the AI safety community, like attending ML conferences (and perhaps running an event or stall), setting up referral incentives, or advertising through the CS departments of top universities.
• Outreach to other populations that might be able to contribute to AI safety. We’ve experimented with reaching out to cybersecurity professionals, and we’re interested in expanding this. We also think some astrophysics academics may be able to transition to AI safety and contribute relatively quickly due to their experience with ML and very large datasets. Notably, we do not plan on expanding into governance-focused work, as we think other organizations are better placed to do this outreach (e.g. the Horizon Institute for Public Service).
• Increasing calls capacity:
  • Dedicated call specialist. With sufficient funding, we may hire a dedicated call specialist to increase volume while retaining quality. However, we’re unsure how easy it will be to find a strong candidate. We plan to hire cautiously, and only if we’re particularly excited about somebody.  We think a level of technical understanding of the AI safety literature is key to conducting high-quality calls with this population, and this skill is in high demand at the moment. We think this may allow us to run up to a maximum of 800 calls per year.
  • Automation and offloading. If we are unable to find a strong calls specialist, we may explore ways to free up Victoria’s time to enable her to run more calls. We think the maximum number of calls we could run in this scenario is around 450.

Between these three focusses, and supported by the skills of the current team to automate more of our processes and expand our operational capacity, we think we can increase our call volume while building on and improving the quality of our programs. We think the main bottlenecks to this outcome are finding the right call specialist and sourcing new leads in a sustainable manner.

How will this funding be used?

Running Arkose at its current scope, without major growth in costs, requires around $25,000 per month. Our minimum funding bar of $75,000 represents a 3-month funding runway, which we think is the minimum time we would need to find additional funding.

With $300,000, we can run Arkose for a full year at its current size, costing around $1,000 per call in total. Funding above this bar will go towards growing the organization, which we think would reduce cost per call. We don’t think we can make use of more than $383,100 within 12 months without losing focus on our core programs.

Given the counterfactual acceleration reported after calls, by senior machine learning professionals, we think $1,000 per call represents a high-impact proposition, particularly when our other programs are taken into account.

A more detailed budget is available on request.

Who is on your team? What's your track record on similar projects?

Team

• Executive Director: Victoria Brook. Victoria brings experience from a variety of roles in small AI safety fieldbuilding organizations including aisafety.info, AI Safety Quest, and AI Safety Edinburgh. Prior to this, her background was in medicine and neuroscience.
• Operations Lead: Maris Sala. Maris has a background in operations and security, having helped set up a variety of programs for AI safety, including ARENA and the London extension of SERI MATS.
• Advisor (founder): Vael Gates. Vael, now Head of Community at FAR.AI, was a postdoctoral researcher at Stanford prior to founding Arkose.

Call Feedback

Arkose has conducted 223 calls since December 2023. Of those who gave feedback on the call (n=115), 77% said Arkose accelerated their involvement in AI safety, most commonly by 1-3 months (n=42). Of 12 responses to our 6-month feedback form, 5 researchers had applied to new jobs due to their call with Arkose and 3 had conducted new research they would not otherwise have done. You can view some additional impact analysis statistics as graphs in this doc.

Transition of Executive Director

Importantly, 153 of these calls were conducted by Vael Gates, who has now moved to an advisory capacity at Arkose. Calls since late June have been conducted by Victoria Brook (n=81). Call quality appears to be comparable: of 38 people who gave feedback after a call with Victoria, 76% stated the call accelerated their involvement in AI safety (vs 79% with Vael). Vael has reviewed a number of calls Victoria has conducted, and from what they’ve seen they are enthusiastic about Arkose in its present form.

Testimonials

• “The call with Victoria was incredibly helpful in supporting me with weighing up my potential options for career steps as well as highlighting some exciting and relevant opportunities that I hadn't come across before.” -Lead Machine Learning Researcher at a company spun out of one of the world's top 20 ML departments
• “The call was incredibly informative. The website offered practical upskilling resources that are particularly valuable from a post-PhD perspective, especially for someone with a theoretical background. I think Arkose has curated a fantastic set of resources.” - Senior Researcher, Microsoft Research
• “AI Safety/Alignment is an active field with diverse perspectives. The Arkose website has a resourceful collection of some of the most relevant papers, jobs, companies and grants, which I found very handy. I gained much clarity when I interacted with the Arkose team over a call and debated on some of the safety issues. I would highly recommend the supportive Arkose team for anyone hoping to enter the field.” - Postdoctoral Researcher, University of Cambridge

Pairing Program

Of 14 people who gave feedback on the connections we made following their call, 9 had spoken with the person we introduced them to when asked 6 weeks after the introduction was made. Of those, 6 rated the connection as “valuable” and 3 as “extremely valuable”. Of 9 experts who gave feedback and had spoken with the other researcher, 7 gave positive feedback.

Negative Examples

Examples of people who found Arkose calls actively unhelpful are available on request. We think these are rare, and have taken a variety of steps to ensure the best possible experience for everybody we contact.

What are the most likely causes and outcomes if this project fails?

We think there are 3 top situations where this project could fail:

Population for outreach is too small. Cold outreach is a key part of the Arkose model: it allows us to maintain a high call volume, and to reach people who may otherwise not have become involved in AI safety.

However, the response rate to cold outreach has dropped since we transitioned Executive Director (2.9% -> 1.7%). We think there are two primary causes:

First, we ranked our leads initially in late 2023, and are now reaching out to people we are less certain will be interested in AI safety. We are presently in the process of sourcing new leads to remedy this concern.

Secondly, Victoria is in GMT, whereas Vael was in PT. As many ML researchers are based in the US, this means they have to book calls in the morning. We are still exploring ways to improve this, though hiring a second call specialist in the US would help considerably.

We think response rate will improve when we find new leads who we think are more likely to be interested in AI safety. We’re also working hard to optimize our initial outreach communications, our public image, and Victoria’s working hours. Alongside this, we’re exploring other outreach methods to supplement our cold email outreach, as mentioned above (in "Project Goals").

Though the population of mid-career ML researchers is small, it grows every year. We’re investigating the rate at which people join the populations we’re interested in reaching out to, even as we look into which other populations might be able to contribute to technical AI safety.

Model is below funding bar, even when well executed. The Arkose model is ambitious. We aim for a high counterfactual impact by reaching out to senior researchers who are unlikely to have prior knowledge of AI safety, and trying to support them through to applying for jobs or conducting new research on the topic, with minimal support (most often we have only one call with researchers, though we provide ongoing email support and occasionally have a second call where appropriate).

We’ve presented the evidence we have for this model being effective, but it could be that this is just too hard to do – changing research direction mid-career can be challenging, and it’s a big ask for people who have relatively little experience with large-scale risks from AI, even if they are interested. We’re exploring ways to track impact better, particularly for long-term outcomes, and we are constantly looking for ways to improve call quality and tailor our programming to our audience.

AI safety is not bottlenecked on researchers. It could be the case that, either now or within the next 18 months, the AI safety community is not in a position where it is bottlenecked on researchers. In this case, the Arkose model makes less sense. Many of the jobs and funding opportunities we’re most excited about are already very competitive, though we think experienced ML researchers and engineers are well-placed to upskill in large-scale risks and apply (relative to more junior researchers or those in other fields). In particular, it seems plausible that AI safety is presently bottlenecked onhighly talented researchers, much more so than just “researchers”. In this scenario, reaching out to senior researchers still seems like a promising approach.

If we feel AI safety is not bottlenecked enough on researchers for the Arkose model to make sense, we may focus on academics, who are uniquely well-placed to do research on large-scale risks with little or no support from the AI safety community, though this is, again, a hard ask without dedicated funding.

How much money have you raised in the last 12 months, and from where?

In the last 12 months, a $10,000 speculation grant from the Survival and Flourishing Fund.

In mid 2023 we received $683,000 from the Survival and Flourishing Fund, which we expect will last us until May or June 2025. We are also currently exploring a number of other funding avenues.