The world really needs more and better places/programs where bright people can try to grow into serious AI alignment researchers. I would guess that the 1-year fellowship proposed above would become the best existing thing of this kind. The 1-month seminar would also probably be the best in its reference class.

Imo, almost all other alignment upskilling programs are mostly creating coding minions for ML labs and printers of ML conference slop, with little emphasis on creating people that can do novel interesting thinking about the AI problems we face. I expect that the proposed program would emphasize getting people to grow into serious alignment thinkers much more than almost any other existing program. (Among existing programs, the main exception that comes to mind is PIBBSS. PIBBSS is good.)

My main uncertainties are about stuff like whether the project will be run decently competently and whether finding applicants goes well — I don't know the organizing team etc. well enough to voice strong views on these sorts of things.

All things considered, I think supporting this is a great use of money.

Overall: I recommend funding this to at least ~$240K, the level needed for the Seminar + 1-year fellowship.

I researched AGI alignment at MIRI for about 7 years; in my judgement, the field is generally not well set-up to appropriately push newcomers to work on the important difficult core problems of alignment. Personally my guess is that AGI alignment is too hard for humans to solve at all any time soon. But, if I were wrong about that, I would probably still think that novel deep technical philosophy about minds would be a prerequisite. I'm not up to date, so this impression might be partly incorrect, but broadly my belief is that most AI safety training programs are not able to create a context where people have the space, and are spurred, to think about those core problems.

Since this program is new, it's hard to judge. I've worked with Mateusz on alignment research, and I think he gets the problem, and the description of the program seems around as promising as any I've seen. Because the space hasn't found great traction yet, trying new things is especially valuable. So, IF you want to fund AGI alignment research, this should probably be among your top investments.

Further, if you want to fund this program, I'd strongly recommend funding it at least to the minimum bar to continue it with the 1-year fellowship. The reason is that learning to approach the actual AGI alignment problem is a slow process that probably needs multiple years, with sparse but non-zero feedback; so the foundations laid down in the month-long seminar might tend to somewhat go to waste without longer-lasting scaffolding.

stable working pods of three to five people

I would suggest creating space for even smaller groups (the standard in Yeshiva, I gather, is pair study, and personally I need substantial time/space set aside for solo thinking). The area is very strongly inside-view-perspective thirsty, so an admixture of space for those to grow is needed, even given the opportunity cost. You could try to offload that to before and after the program, but I'd suggest also making space for it during. E.g. a "Schelling" time for 2 hour solo walks / thinks, or whatever.

We actually consider it very likely that the project "fails" in the sense that it will complete with none of the Fellows producing any clearly promising research outputs or directions at building pieces of a solution. The reason/cause of this would be that the problem being tackled is one of great difficulty, very slippery, and with difficult feedback loops with reality.

This is an unbelievably based statement, which on the object level would hopefully contribute to making an environment where actual new perspectives (rather than just the Outside the Box Box https://www.lesswrong.com/posts/qu95AwSrKqQSo4fCY/the-outside-the-box-box ) can grow, and furthermore indicates some degree of hopeworthiness of the organizers on that dimension.

participants will share their learning with each other through structured showcases and peer instruction

Sounds cool, but do keep in mind that this could also create a social pressure to "publish or perish" so to speak, leading to goodharting. A not-great solution is to make it optional or whatever; it's not great because it's sort of just lowering standards, and presumably you do want to have people aiming to work hard and do the thing. Maybe there are better solutions, such as somehow explicitly and in common knowledge making it "count for full points" to present on "here's how I have a really basic/fundamental question, and here's how I kept staring at that question even though it's awkward to keep staring at one thing and not have publishable technical results from that, and here's my thoughts in orienting to that question, and here's specifically why I'm not satisfied with some obvious answers you might give". Or something. In other words, alter the shape of the landscape, rather than making it less steep.

Selection criteria for the fellows:

I would suggest somewhat upweighting something like "security mindset", or (in the same blob), something like "really gets that you can have a plausible hypothesis, but it's wrong, and you could have quickly figured out that it's wrong by actually trying to falsify it / find flaws in it, but you probably wouldn't have quickly figured out that it's wrong just by bopping around by default". And/or trying to bop people on the head to notice that this is a thing, though IDK how to do that. This is especially needed because, since we don't get exogenous feedback about the objects in question, we have to construct our own feedback (i.e. logical reasoning about strong minds).

This seems like an overall good idea, and I strongly recommend funding this to at least the 1-month Seminar level.

A few people have floated this kind of program to widen the funnel for people that might want to work on AI safety research with the hopes of kickstarting the involvement of people who otherwise might not know how to get working, or what concepts that existing researchers - even marginal ones - would find very basic.

A 1-month version would likely be best-in-class due to a relative lack of comparable programs - itself a problem! - though I think that a 1-year version might be overambitious and risk burning out or disengaging scholars if not done extremely well and carefully.

All the same, I've worked with Mateusz for a period of time and been part of a what turned into a very small category theory reading group with him, and I think he's very well-suited to this approach. AI safety - especially the kind of AI safety that looks like attempts to find a solution to alignment rather than a dozen ad-hoc patches to existing LLMs - suffers badly from a lack of serious research groups, and this project looks to me like it would be at least half as promising per person as MATS is and maybe 3/4 as promising as PIBBSS, both of which I've been a part of, which have similar mission statements, and which have been funded at higher levels - and both frequently claim that they want to see cousin orgs founded!

I would donate substantially to this if I had piles of tech or crypto money, but sadly I do not. I hope that other people who do have piles of tech or crypto money will hear me and donate in my place. If were a grantmaker I would almost certainly be directing grant funds to this endeavor.

I don't have the energy right now to write a high quality comment, but since I care about this I figure it's better to write something rather than nothing:

I think this project sounds like a good idea. Most (all?) AI Safety training programs these days don't even seem to touch on what I'd consider the actual core problems of alignment. I think there can be good reasons for many people and programs to mostly focus on other things at the moment, but it really seems almost catastrophically underemphasised at this stage. I don't know every training program of course, but talking to e.g. MATS graduates these days I often get the sense they haven't even really heard the basic case for why alignment might be hard. Looking at various AI Safety course curricular I likewise see an almost complete lack of material engaging with what I'd consider the core problems of alignment. If this continues, eventually I'm not sure this field will even really remember what it was supposed to be about, never mind try to work on it.

I know Mateusz a little. From our limited interactions, I got the impression that he probably knows at least a decent amount about the sort of old-school alignment thinking that I wish the alignment field today was a lot more familiar with. Tsvi's endorsement also means quite a bit to me here. I think he could make a good technical lead for this project. I don't think I know Sofie or Attila. I do know Plex, but haven't really worked with him professionally. Other people say he's good at what he does though. My guess is he'd be a good fit for this role.

I endorce this project.

Context: I have some expereince in AI Safety organising, running AI Safety Camp and some other events. Becasue of my expereince, plex (who is one of the organsiers for this event) reached out to me for feedback on their plans. We had a one hour call. I came away from that call being entusiasticly in favour of this event happening. I also know both plex and Mateusz, and trust them to do a good job.

Looks exciting. My personal view is that there's a lot of progress waiting to be made on theoretical/agent foundations research. The quality of the program will of course depend a lot on the quality of fellows; I'm curious if there are many people already on your radar, or if you think you have good leads there.

A few other thoughts:

- I think trying to persuade people that the alignment problem is hard is often counterproductive. The mindset of "I need to try to solve a extremely difficult problem" is not very conducive to thinking up promising research directions. More than anything else, I'd like people to come out of this with a sense of why the alignment problem isinteresting. Happy to talk more about this in a call.

- Some of the selection criteria seem a bit counterproductive. a) "Decent team players, non-disruptive to the group cohesion" seems like a bad way to select for amazing scientists, and might rule out some of the most interesting candidates. And b) "would care about saving the world and all their friends if they thought human extinction was likely" seems likely to select mainly for EA-type motivations which IMO also make people worse at open-ended theoretical research. Meanwhile c) "highly technically skilled" is reasonable, but I care much more about clarity of thinking than literal technical skills.

If the organizers have good reasons to expect high-quality candidates I expect I'd pitch in 5-10k.

I am enthusiastic about this, and interested in being involved. I know Plex and Mateusz, and have some trust in their taste. I expect the program will focus on the most important issues (ie the most severe and neglected AI risks). I've been to a similar (much shorter) event at the venue, and found it to be a good location, with good countryside walks very conducive to thinking and conversation.

Sounds like a good shot at creating new people working on alignment.

My concern is whether the structure of the program and how it's executed and adjusted along the way would be good. Much of it hinges on the people you can secure as mentors

Also, the quality of participants is extremely important of course, but I'm optimistic about it, although not if you plan the program for March.

Kudos for posting the utility curve and encouraging others to do so, however I'm confused how 1 fellow buys you 70-44=26% utility, and then extra 9 fellows buys you another 100-70=30%.

I will join the chorus of people who are like "oh man nobody is actually trying to, uh, solve the problem of pointing/constraining/steering/limiting vast amounts of optimization power, but instead we are spending resources on many things that are related" (monitoring the situation! finding ways to steer non-vast amounts of optimization power! measuring non-vast amounts of optimization power! building infrastructure for slightly less non-vast amounts of optimization power!). "It surely would be great if more people tried to figure out how to figure out strategies that even in concept work for vast overwhelming optimizers. Or build things that are not that but still useful enough as an alternative." This seems like the kind of event that would cause more thinking on alignment foundations to happen (not necessarilyagent foundations, mind you, we might want to leave that line of retreat open).

I found this project so enticing that I considered foregoing tax benefits and donating a prospective 10% of my income to it (which would be fairly little, but what can you do).

I've interacted with Mateusz for about ten hours in total, and with plex for about 1½ hours, and found them to be good thinkers, plausibly better than me, maybe a bit more obscurantist. I feel like I can have a higher confidence in Mateusz than plex given the amount of public to-me-legible-output, but I may be misremembering as I'm shooting this comment from my hip.

I mean finally if it's another $200k to METR or to this I definitely choose this.

I see that the biggest cost would be the venue + food. In case you don't raise the full 179k, I wonder - would it be possible to organize a MVP / quick test of the idea in a cheaper venue? The cheapest would be EA hotel / Pause House, but idk how good it would be for your purposes.

Another idea would be to look for a sponsor who would host you in a big venue for free / for cheap. Maybe someone EA-related or maybe just someone with a history of hosting events for NGOs. E.g. I know a lovely private-owned venue near Cracow, where I went to a Jacek Kaczmarski festival twice (organized by the NGO "Kaczmarski Underground"). It was lovely in the summer, and you can swim & kayak there. https://www.google.com/maps/place/Gospodarstwo+Rybackie+Brze%C5%BAnica+Marcin+Orlanka/@49.9698821,19.6442588,662m/

Just a few quick-and-dirty ideas from me, not well thought out. Good luck with the project! :)

Having read most of the proposal it seems like you don't have a model about how to make people good at alignment research.

You write things like "deep technical engagement" . I expect you mean studying existing literature.

I expect this won't work. Or at least not work reliably. Part of what makes the alignment problem had is that we don't yet have a good model of the problem. To make progress on alignment you need to have the ability to notice your confusions, to think through your confusion, and to not give up until you have achieved some level of clarity.

You need to be able to handle situations where there is no obvious next step. You need to know how to pick up the problem and look at it from different angles until after some significant effort of analysis you actually manage to make progress, instead of dropping the ball early.

When I tried to teach people how to do alignment research The main problem I run into is that I don't manage to get them to seriously try. That is, to seriously try to solve the actually hard problems.

Either they get distracted by some cool tractible but ultimately inconsequential problem, or they try to run off reading the sequences, all of what Vanessa work, study math, etc.

Of course I'm not saying that reading the sequences, learning math, or reading other peoples work is inteinically bad. It's bad here because it's used as an escape mechanism. It's easier to study linear algebra than to try to make progress on alignment. Learning linear algebra might be hard but at least the path is clear.

There are probably many more important skills I didn't list, that are necessary to become an effective alignment researcher.

The problem: Your proposal doesn't even try to point at this set of skills at all. I expect you're not going to even try to teach this skill-set, because you can't, because you don't have a model of what they even are or how to teach them.

Now all that said, all else equal, I expect this project is good to do. It is just that I would be much more excited about the project if you could lay out clearly a model of what kind of mental procedures are required to make process on alignment, and how to entrain these procedures effectively.

This is a hard problem and I wish more people were to seriously think about it. Especially the people who run events like this, meaning any events with the goal of making people capable AI alignment researchers.

The only person I know who seriously thought about this and then tried to implement his model, on a semi-large scale, is John Wentworth. I think he got a lot right in his MATS program stream, but it also feels like he only tried to teach a fraction of the skills necessary.

Update: Thanks to a private funder, we are now covered for the seminar. We are still fundraising for the subsequent 11 months of the fellowship.

Moreover, applications are now open!

If interested, apply here!

@Mateusz-Baginski , as a somewhat newcomer to Manifund, but very interested in your seminar and fellowship vision for purposes of deepening in-person community building within the AI safety realm, may I ask a question: how do you see the seminar or the fellowship impacting future AI policy or governance outcomes or specific actions/changes? Thank you, and congratulations on the initial seminar funds being funded! ~ Erik

As someone who has worked with multiple organizations in the AI Safety and AI alignment ecosystems, I think the current development of the field seems to indicate that there should perhaps be a clear separation between the Empirical AI Safety track which is some of the most important and valuable work like BlueDot Impact or MATS and a different pipeline for more exploratory (yet valuable in the long term) that defies the systematic incentives and the status quo such as PIBBSS, ILLIAD, and this fellowship.

In other words, the empirical track is essential. But its very success under current incentives creates a risk: we may optimize so hard for what can be measured now that we starve the exploratory work that will only prove its value later. We need both. This project strikes me as exactly the kind of alternative pipeline that embodies the latter: its design prioritizes deep understanding, community, and researcher well-being. These choices directly counteract the pressures that push some current work toward short-term, measurable results, sometimes at the expense of the deeper thinking the foundational problem ultimately requires.

I personally feel like the mismatched in ecosystems did not allow some participants like me to thrive and perhaps find that if I were to be in this ecosystems instead my productivity and research output wouldn't have been negatively impact by the measurement pressures. I encourage people with more resources to fund this work.