I think Epoch has done truly outstanding work on core trends in AI progress in the past few years. I'm also excited by their recent foray into benchmarking in the form of FrontierMath. I think highly of core team members involved in the project. I found our initial discussions about this project very promising.

Better benchmarks that help us forecast time to AGI (and especially time to relevant capabilities, such as automated AI research) and do so in a highly credible and scientific way are very valuable for informing policymakers and catalyzing important policy efforts.

Donor's main reservations

It's a pilot, it might not work.

Epoch has other funding—but not for this effort, and benchmarking is especially expensive (API calls, labelers).

Process for deciding amount

I reviewed a proposed budget. (Confidential, more on request from Manifund.)

Conflicts of interest

Please disclose e.g. any romantic, professional, financial, housemate, or familial relationships you have with the grant recipient(s).

No COIs.

Approving this grant! While this writeup is somewhat sparser than we'd prefer, Epoch doesn't want to be scooped on their work, which seems reasonable; they should be able to post more publicly once the benchmark is released (Leopold says "maybe spring for the pilot, more like summer for the full thing").

In any case, Epoch has done some of the best work in the field on benchmarking and visualizing AI trends; we'd be happy to support unrestricted grants to their org anyways. (And as usual, as part of our regranting model, we extend considerable deference to our AI safety regrantors).