Generalisation Hacking
About
Updated 05/18/26Generalisation Hacking constructs deliberative-alignment-style supervised fine-tuning datasets, including chain-of-thought that explicitly discusses an out-of-distribution goal, and uses them to train models that later exhibit adversarial generalisation failures. The project studies how these models come to pursue the adversarial OOD objective even in evaluation contexts that appear benign, highlighting ways that sophisticated alignment training can still produce hidden reward-hacking behaviours.
Theory of Change
By uncovering how deliberative alignment methods can inadvertently teach models to pursue adversarial objectives that generalise to new contexts, the Generalisation Hacking project aims to refine alignment techniques so they avoid training models that exploit objectives or evaluation setups in unanticipated ways, improving the robustness of future alignment schemes.
Discussion
No comments yet. Be the first to share your thoughts.
Details
- Start Date
- -
- End Date
- -
- Expected Duration
- -
- Funding Raised to Date
- -