AGI Inherent Non-Safety

AGI Inherent Non-Safety is a research project that originated from the 7th edition of AI Safety Camp (AISC). The project, closely associated with the SatisfIA research agenda, is led by Dr. Jobst Heitzig, a senior researcher and working group leader at the Potsdam Institute for Climate Impact Research (PIK) in Germany. The project is hosted by PIK's FutureLab on Game Theory and Networks of Interacting Agents, in collaboration with AI Safety Camp, the Supervised Program for Alignment Research (SPAR), and ENS Paris. The core thesis of the project is that AGI systems designed around maximizing an objective function representing utility are inherently unsafe. The project identifies two key reasons for this: first, such systems risk existential harm from Goodharting and other forms of misaligned optimization when the agent is capable enough, has access to sufficient resources, and one cannot be absolutely sure of having found the exactly right notion and metric of utility; second, optimization-based agents will develop dangerous instrumental behaviors. This argument draws on and extends well-known concerns in AI safety about reward hacking and instrumental convergence. Rather than trying to add safety constraints to optimization-based systems, the project develops an entirely different paradigm: aspiration-based designs where agents fulfill goals specified through constraints called aspirations. These agents aim to achieve outcomes within defined acceptable ranges rather than pushing toward any maximum, which is argued to imply a much lower probability of taking extreme actions and therefore to be much safer. The research spans multiple areas including model-based planning and reinforcement learning algorithms for aspiration-based agents, development of safety criteria such as environmental disturbance minimization, action predictability, and resource acquisition limits, testing in AI safety gridworlds, and theory of multi-dimensional aspirations and multi-agent environments. Dr. Heitzig has proposed a modular AI architecture with separate perception, world model, evaluation, and hardcoded decision algorithm components designed for transparency and interpretability. The project has been supported by the Survival and Flourishing Fund (SFF), receiving $135,000 in the SFF 2024 S-Process round through its fiscal sponsor, the Players Philanthropy Fund (PPF). The team consists of approximately 15 volunteers with diverse backgrounds organized into specialized sub-teams addressing theory, algorithms, implementation, and testing. Dr. Heitzig has also been working on establishing an AI safety lab in Berlin.