A nonprofit research institute applying category theory, topos theory, and type theory to develop mathematical foundations and open-source tools for collective sense-making, collaborative modeling, and shaping technology for public benefit.

A nonprofit research organization that works to reduce societal-scale risks from artificial intelligence through safety research, field-building, and advocacy.

An African-led research program dedicated to building talent, generating impactful research, and shaping policy to advance AI safety, based in Nairobi, Kenya.

A university research lab at the University of Louisville directed by Dr. Roman Yampolskiy, one of the founders of the field of AI safety, conducting research on the theoretical limits of AI controllability, AI containment, and cybersecurity.

PauseAI is a global grassroots movement advocating for an immediate pause on the development of frontier AI systems until their safety can be demonstrated and they can be kept under democratic control.

MATS (ML Alignment & Theory Scholars) is the largest AI safety research fellowship and talent pipeline, running intensive 12-week research programs that pair fellows with leading AI alignment mentors in Berkeley and London.

A nonprofit that uses legal advocacy, including amicus briefs, impact litigation, and policy engagement, to mitigate catastrophic risks from advanced AI systems and biotechnology.

CLAIR is building the field of Law and AI Safety, producing and promoting legal scholarship on reducing catastrophic and existential risks from advanced artificial intelligence.

A nonprofit that runs fellowships and educational programs to develop expert, mission-aligned talent for AI safety research and governance.

A project that tracks and evaluates frontier AI companies on their safety practices through a weighted scorecard, focusing on actions labs should take to avert extreme risks from advanced AI.

IASEAI is an independent nonprofit that works to ensure AI systems operate safely and ethically by shaping policy, promoting research, and building a global community around AI safety.

A Swiss non-profit think tank that develops evidence-based policy proposals on AI safety, biosecurity, and emerging technologies, bridging science, politics, and civil society for Switzerland and beyond.

A non-profit AI alignment research organization focused on agent foundations, pursuing formal goal alignment approaches that would scale to superintelligence.

A nonprofit organization based in the US and Europe that works to align AI through better governance, developing and advocating for AI governance mechanisms ranging from laws and regulations to voluntary frameworks.

A philanthropic platform and 501(c)(3) nonprofit that facilitates regranting, impact certificates, and crowdfunding for charitable projects, with a primary focus on AI safety and effective altruism cause areas.

An Israeli academic research and advocacy nonprofit focused on reducing catastrophic and existential risks through AI safety research, biosecurity policy, and standards development.

Atlas Computing is a 501(c)(3) nonprofit that maps neglected AI safety risks, sources expert founders, and prototypes solutions to scale human control over advanced AI capabilities.

Carnegie Mellon University is a leading private research university in Pittsburgh, Pennsylvania, widely regarded as one of the world's top institutions for AI and computer science research. It hosts multiple AI safety and governance programs spanning technical research, policy, and applied AI security.

A French nonprofit that develops AI risk management frameworks, independently rates AI companies' safety practices, and contributes to international AI governance standards.

Youth-led AI policy nonprofit that advances AI safety, governance, and accountability through nonpartisan legislative advocacy and public education, headquartered in Washington, DC.

Nonprofit investigating cyber offensive AI capabilities and the controllability of frontier AI models to help humanity avoid permanent disempowerment by strategic AI agents.

The legal entity behind the Centre for Long-Term Resilience (CLTR), a UK-based independent think tank working to transform global resilience to extreme risks, particularly in AI safety and biosecurity.

AE Studio is a bootstrapped technology studio and AI alignment research organization that funds neglected safety research from its software consulting profits. Their work spans brain-computer interfaces, self-other overlap fine-tuning to reduce LLM deception, and consciousness research.

Contramont Research is a nonprofit AI safety lab that studies where safety and security evaluation methods break down, using cryptographic model organisms to expose fundamental limitations of existing techniques.

Johns Hopkins University hosts AI safety-relevant research led by Prof. Anqi (Angie) Liu, whose group focuses on machine learning for trustworthy AI, including distributionally robust learning and uncertainty quantification under distribution shift.

BIML is an independent nonprofit research institute focused on machine learning security, specifically the work of building security into ML systems at the design level.

A nonprofit that commissions and funds open, expert evaluation and quantitative rating of economics and social science research relevant to global priorities, without the constraints of traditional academic journals.

CARMA is a research and policy think tank working to lower the risks to humanity and the biosphere from transformative AI through integrated risk management, policy research, and technical safety work.

Lightspeed Grants is a fast-turnaround grantmaking program run by Lightcone Infrastructure that in 2023 provided rapid funding for projects aimed at reducing existential risk and improving humanity's long-term future, but has since been inactive and is not currently accepting applications.

A pooled multi-donor charitable fund that rapidly deploys grants to reduce catastrophic risks from advanced AI, covering technical alignment, governance, and evaluations.

Bounded Regret is the personal research blog of Jacob Steinhardt, Associate Professor at UC Berkeley, covering AI safety, machine learning, forecasting, and philosophy.

A global participatory futures research think tank that produces the annual State of the Future report and tracks 15 Global Challenges facing humanity, with growing focus on AGI governance and existential risk.

Team Shard is a small alignment research collective led by Alex Turner (TurnTrout) that studies how reinforcement learning induces values in trained agents, with the goal of learning to reliably instill human-compatible values in AI systems.

Israel's oldest and largest research university, founded in 1912, with particular strength in computer science, engineering, and AI research. It ranks first in Europe and 21st globally for AI research output.

Collider is a coworking and community space in New York City for AI safety and other high-impact professionals to work, collaborate, and convene.

The Power Law is a Substack newsletter by Peter Wildeford (also known as Peter Hurford) covering AI forecasting, AI policy, national security, and emerging technology.

Stanford HAI is an interdisciplinary university institute advancing AI research, education, and policy with a focus on AI that benefits humanity and augments human capabilities. It is best known for publishing the annual AI Index Report.

SPAR is a part-time, remote research fellowship that pairs aspiring AI safety and policy researchers with experienced mentors for 3-month research projects. It is one of the largest AI safety research fellowships by participant count.

Lucid Computing builds hardware-rooted AI verification infrastructure that cryptographically proves where AI chips are located and what they are processing, enabling enforceable compute governance and regulatory compliance.

MIT is a private research university in Cambridge, Massachusetts, widely recognized as a global leader in science, engineering, and technology research, including AI safety and alignment.

A nonprofit that archives humanity's ideas, ideologies, and world-views through structured debate mapping, with a focus on AI safety, alignment, and democratic governance of AI.

AFFINE (Agent Foundations FIeld NEtwork) runs intensive superintelligence alignment seminars and fellowships to upskill promising newcomers in agent foundations and AI alignment research.

Canada's national AI safety institute, established by the federal government in November 2024 to advance the science of AI safety and ensure governments can understand and act on risks from advanced AI systems.

A leading Canadian research university founded in 1957, home to AI safety-relevant research programs including technical AI safety grants from Coefficient Giving and CIFAR's Canadian AI Safety Institute program.

ControlAI is a nonprofit advocacy organization working to keep humanity in control of advanced AI by pushing governments to prohibit the development of artificial superintelligence.

AI Safety Hungary is a Budapest-based nonprofit that runs educational programs and career support to help Hungarian students and professionals enter the AI safety field.

The ARIA Lab (Aligned, Robust, and Interactive Autonomy Lab) at the University of Utah, led by Professor Daniel S. Brown, conducts research on human-AI alignment, reward learning, and AI safety. The lab develops algorithms and theory to enable AI systems to safely learn from and interact with humans.

Personal blog of Victoria Krakovna, Senior Research Scientist at Google DeepMind and co-founder of the Future of Life Institute, covering AI alignment research and related topics.

Stop AI is a grassroots activist organization that uses non-violent civil disobedience and public advocacy to demand a permanent, enforceable global ban on the further development of frontier AI technology.