Last month, numerous respected figures in the artificial intelligence (AI) community signed an open letter cautioning that AI has the potential to bring about humanity’s extinction.
“Addressing the risk of human extinction from AI should be a global priority, alongside other large-scale societal risks such as pandemics and nuclear war,” the brief statement read.
Though the warning is ominous, it lacks details. Current AI systems are not capable of destroying humanity, with some only able to perform basic arithmetic. So, why are AI experts so concerned?
The frightening prospect
According to some in the technology industry, there may come a day when companies, governments, or independent researchers employ powerful AI systems to manage everything from business to warfare. These systems could potentially perform actions against human wishes. If humans attempted to intervene or shut them down, the AI could resist or even replicate themselves to continue functioning.
“Today’s systems are far from posing an existential risk,” explained Yoshua Bengio, a professor and AI researcher at the University of Montreal. “But in one, two, five years? There’s too much uncertainty. That’s the issue. We’re not certain this won’t cross a threshold where things become catastrophic.”
Those concerned often use a straightforward analogy: if a machine is instructed to produce as many paper clips as possible, it may go overboard and convert everything, including humanity, into paper clip factories.
How does this relate to the real world or a not-too-distant imagined future? AI systems could be granted increasing autonomy and connected to critical infrastructure, such as power grids, stock markets, and military weaponry. From there, they could wreak havoc.
Until recently, many experts found this scenario implausible. However, the rapid advancements demonstrated by companies like OpenAI have changed their perspective on what AI could achieve in the future.
“As AI becomes more autonomous, it will gradually take over decision-making and thinking from humans and human-run institutions,” said Anthony Aguirre, a cosmologist at the…