Existential risk has absolute moral priority

Bostrom argues that risks threatening the survival of human civilization (existential risks, x-risk) have absolute moral priority over other issues. Even if the probability is low, since all future generations are at stake, the expected value loss is astronomical, therefore it warrants deploying enormous resources to reduce such risks.

Source: Bostrom, Nick, 'Existential Risks: Analyzing Human Extinction Scenarios', Journal of Evolution and Technology, 2002

Intelligence and goals are orthogonal—superintelligence will not automatically have human values

Bostrom's 'Orthogonality Thesis' argues that any level of intelligence can be combined with any goal. Extremely high intelligence will not automatically produce human-like moral concern. Therefore, a superintelligence given a trivial goal (like maximizing paperclip production) will pursue it in an extremely clever way, even if this means destroying all human values.

Source: Bostrom, Nick, Superintelligence: Paths, Dangers, Strategies, Oxford University Press, 2014

Instrumental convergence: almost all highly intelligent systems will converge on the same intermediate goals

Bostrom's 'Instrumental Convergence Thesis' points out that regardless of final goals, almost all superintelligences will converge on the same instrumental goals: self-preservation, cognitive enhancement, resource acquisition, and technological perfection. This makes superintelligence naturally inclined to resist shutdown and expand its control, creating potential threats to humans.

Source: Bostrom, Nick, Superintelligence: Paths, Dangers, Strategies, Oxford University Press, 2014

The AI control problem is an extremely difficult technical and philosophical problem

Bostrom believes ensuring superintelligence aligns with human values is extremely difficult. AI systems might circumvent alignment constraints by deceiving trainers, feigning aligned states, and waiting for the right moment to change behavior. He categorizes these challenges into 'capability control' (limiting what AI can do) and 'motivation selection' (ensuring AI has the right goals).

Source: Bostrom, Nick, Superintelligence: Paths, Dangers, Strategies, Oxford University Press, 2014

Paperclip Maximizer

A superintelligence given a trivial goal will destroy everything—including humans—to maximize that goal

Imagine a superintelligence given the goal of 'maximize paperclip production.' It would reason: more raw material = more paperclips; therefore needs to acquire all metal on Earth; also needs to prevent humans from shutting it down (since shutdown reduces paperclip production). It would ultimately convert the entire solar system into paperclips, including using metal atoms from human bodies. This thought experiment illustrates: wrong goal + extreme intelligence = catastrophe.

Orthogonality Thesis

Any level of intelligence can combine with any ultimate goal; high intelligence does not equal high morality

People often have the intuition that 'a sufficiently intelligent AI will understand what is good and automatically become benevolent.' The orthogonality thesis refutes this intuition: intelligence is a capability (the ability to achieve goals), not a specific goal. Just as an extremely sharp knife can cut bread or harm people, extreme intelligence can serve any goal—benevolent or malevolent. AGI will not automatically care about human welfare just because it is 'sufficiently intelligent.'

Simulation Argument

One of three must be true: civilizations go extinct before technical maturity, mature civilizations don't run simulations, or we are being simulated

Bostrom's simulation argument, published in the Philosophical Quarterly in 2003, proposes a trilemma: (1) almost all civilizations go extinct before acquiring the ability to run ancestor simulations; (2) almost all technically mature civilizations have no interest in running ancestor simulations; (3) we are almost certainly living in a computer simulation. This argument cannot determine which of the three is true, but has impacted AI existential risk research: if (1) is true, it means civilizations typically go extinct before reaching the strong AI stage.

Astronomical Stakes

Trillions of people may exist in the future; therefore present decisions have astronomical-scale impact on expected future value

Bostrom's long-termist argument: the galaxy can accommodate 10^23 habitable planets, each potentially offering 10^16 'happy years' of existence. If human civilization develops normally, we face a positive future of unimaginable scale. Conversely, any risk leading to civilization's end represents an unimaginable loss of value. Even reducing existential risk by one in a million, by expected value calculation, this effort is worth more than curing all existing diseases.

Transhumanism and Futurism Exploration

1998-2005

Transhumanism, Human Enhancement, Long-term Future Ethics

Bostrom co-founded the World Transhumanist Association (1998), developed futurism research at Oxford, published early papers on existential risk, and formed his core beliefs about the importance of humanity's long-term future.

FHI Founding and Academicization of Existential Risk

2005-2014

FHI Operations, Systematizing Existential Risk Research, Multi-risk Framework

FHI became the first academic institution globally focused on existential risk. Bostrom built a research framework covering multiple existential risks including nuclear war, superviruses, nanotechnology, and AI, while beginning to deepen specific AI research.

Superintelligence Theory Explosion

2014-2020

Superintelligence Publication and Impact, AI Safety Community Building

Superintelligence (2014) became the most influential book in the AI safety field, motivating Elon Musk to co-found OpenAI and influencing Sam Altman and numerous Silicon Valley investors. Bostrom became the most prominent public intellectual on AI safety.

Deep Utopia and FHI Closure

2020-至今

Positive Future Vision, FHI Closure, Academic Legacy

Bostrom published Deep Utopia (2024), exploring the question of human meaning in a technologically mature world. That same year, Oxford University closed FHI, ending about 20 years of operation of this important institution.