FindTheFlaws

FindTheFlaws is a scalable oversight benchmark developed by Modulo Research and collaborators. The project assembles five challenging datasets drawn from existing benchmarks in medicine, mathematics, science, coding and the Lojban language, and augments them with expert-written solutions and annotations indicating exactly where reasoning goes wrong. Using these datasets, the team evaluates frontier language models on tasks such as grading solutions and identifying the first erroneous step in flawed reasoning, showing substantial variation across models and domains. This resource is intended to support experiments in debate, critique and prover–verifier protocols by providing realistic long-form reasoning traces with ground-truth error labels.

Endorsements support Modulo Research.

No endorsements yet.

Discussion

Start Date

-

End Date

-

Expected Duration

-

Funding Raised to Date

-