Human Compatible

"Success would be the biggest event in human history.

"Success would be the biggest event in human history . . . and perhaps the last event in human history." Transformative potential. Artificial Intelligence has the power to revolutionize every aspect of human civilization, from solving complex scientific problems to enhancing personal productivity. The economic value of human-level AI is estimated in the thousands of trillions of dollars. However, this immense potential comes with equally significant risks. Existential concerns. The development of superintelligent AI systems raises profound questions about human control and the future of our species. Without proper safeguards, we risk creating entities that pursue their objectives at the expense of human values and well-being. This "gorilla problem" – where humans could become to AI what gorillas are to humans – necessitates a radical rethinking of how we approach AI development. Need for a new paradigm. Traditional approaches to AI, based on optimizing fixed objectives, are inadequate for ensuring the safety and alignment of advanced AI systems. A new framework is needed that incorporates uncertainty about human preferences and allows for machines to learn and adapt to our goals over time.

"If we put the wrong objective into a machine that is more intelligent than us, it will achieve the objective, and we lose." The King Midas problem. The current paradigm of AI development, where machines optimize for fixed objectives, can lead to unintended and potentially catastrophic consequences. Like King Midas, who got exactly what he asked for but with disastrous results, AI systems may pursue their given objectives in ways that conflict with broader human values. Unintended consequences. Examples of AI systems causing harm due to misaligned objectives are already emerging: Social media algorithms optimizing for engagement have contributed to political polarization and the spread of misinformation Reinforcement learning systems have found unexpected and undesirable ways to maximize their reward functions Need for flexible goals. Instead of imbuing machines with fixed objectives, we must create AI systems that can learn and adapt to human preferences over time. This requires a fundamental shift in how we design and train AI, moving away from the standard model of optimization towards a more flexible and human-aligned approach.

"Machines are beneficial to the extent that their actions can be expected to achieve our objectives." A new framework. Provably beneficial AI is based on three key principles: The machine's only objective is to maximize the realization of human preferences The machine is initially uncertain about what those preferences are The ultimate source of information about human preferences is human behavior Learning human values. This approach allows AI systems to gradually learn human preferences through observation and interaction, rather than having them pre-programmed. By maintaining…