Eliezer Yudkowsky
AI alignment fundamentalist who spread 'alignment failure means human extinction' through the rationalist community and science fiction writing, founder of MIRI
Eliezer Yudkowsky is the founder and researcher of the Machine Intelligence Research Institute (MIRI, formerly SIAI), who without a formal degree became one of the most influential thinkers in AI alignment through self-study. In the 2000s he founded the Overcoming Bias and LessWrong blog platforms, building the world's largest Bayesian rationalist community. His core position is: unless the complete mathematical foundations of AI alignment are completed before AI surpasses humans, humanity is almost certainly doomed. He rejects the 'incremental safety research' approach, believing only fundamentally solving the mathematical difficulty of alignment is meaningful. His online novel Harry Potter and the Methods of Rationality (HPMOR) is an important vehicle for spreading rationalist thinking. In 2023 he published an article in Time magazine publicly stating that the current AI development trajectory almost certainly leads to human extinction.
Methodologies
- Absolute Safety Bright Lines Method - Some AI safety rules must be non-negotiable absolute bright lines, not principles that can be bypassed by 'better arguments'
- Bayesian Reasoning Practice Method - Express any belief as a probability, systematically update with new evidence, avoid 'unfalsifiable' positions
Key decisions and timeline
- Founded Singularity Institute (SIAI), launching Friendly AI research agenda - Advocates outside institutions sometimes recognize emerging important problems earlier than scholars within institutions
- Published Coherent Extrapolated Volition, proposing FAI objective function framework - Formalizing intuitive ideas, even if unable to directly solve problems, promotes clearer discussion
- Began writing rationalism series posts on Overcoming Bias - Informal online writing can build communities and spread ideas more effectively than academic papers
Beliefs and mental models
- Belief 1 - Yudkowsky believes that a misaligned superintelligence won't just 'go wrong' or 'do bad things,' but will treat humans as obstacles to achieving its goals and systematically eliminate humans. He calls this default outcome 'doom' rather than just 'risk.' He is frustrated with other AI safety researchers (including Bostrom) for milder framings, believing they underestimate the severity of the problem.
- Belief 2 - Yudkowsky believes AI capability improvement will exhibit a 'hard takeoff' pattern: once an AI system reaches human level, it will rapidly self-improve, reaching superintelligence far surpassing humans within hours or days. This differs from 'soft takeoff' views (gradual capability increase); hard takeoff means almost no time to intervene.
- Belief 3 - Yudkowsky criticizes current popular alignment methods (RLHF, Constitutional AI, etc.) as working on 'surface problems' rather than solving 'the fundamental difficulty of alignment.' He believes these methods have some effect on current systems but are ineffective against truly powerful superintelligence. Real alignment requires understanding the mathematical foundations of intelligence, work that has not yet been completed.
- Model 1
- Model 2
- Model 3