The Danger Of Superhuman AI Is Not What You Think

Today’s generative AI systems like ChatGPT and Gemini are routinely described as heralding the imminent arrival of “superhuman” artificial intelligence. Far from a harmless bit of marketing spin, the headlines and quotes trumpeting our triumph or doom in an era of superhuman AI are the refrain of a fast-growing, dangerous and powerful ideology. Whether used to get us to embrace AI with unquestioning enthusiasm or to paint a picture of AI as a terrifying specter before which we must tremble, the underlying ideology of “superhuman” AI fosters the growing devaluation of human agency and autonomy and collapses the distinction between our conscious minds and the mechanical tools we’ve built to mirror them.

Today’s powerful AI systems lack even the most basic features of human minds; they do not share with humans what we call consciousness or sentience, the related capacity to feel things like pain, joy, fear and love. Nor do they have the slightest sense of their place and role in this world, much less the ability to experience it. They can answer the questions we choose to ask, paint us pretty pictures, generate deepfake videos and more. But an AI tool is dark inside.

That’s why, at a machine learning conference in September of 2023, I asked the Turing Award winner Yoshua Bengio why we keep hearing about “superhuman” AI when the products available are so far from what a human is, much less superhuman. My keynote prior to his had openly challenged this kind of rhetoric, which featured heavily in Bengio’s subsequent presentation — just as it does on his website and in his warnings to lawmakers and other audiences that humans risk “losing control to superhuman AIs” in just the next few years.

Bengio was once one of the more sober and grounded voices in the AI research landscape, so his sudden adoption of this rhetoric perplexed me. I certainly don’t disagree with him about the dangers of embedding powerful but unpredictable and unreliable AI systems in critical infrastructure and defense systems or the urgent need to govern these systems more effectively. But calling AI “superhuman” is not a necessary part of making those arguments.

So, I asked him, isn’t this rhetoric ultimately unhelpful and misleading given that the AI systems that we so desperately need to control lack the most fundamental capabilities and features of a human mind? How, I asked, does an AI system without the human capacity for conscious self-reflection, empathy or moral intelligence become superhuman merely by being a faster problem-solver? Aren’t we more than that? And doesn’t granting the label “superhuman” to machines that lack the most vital dimensions of humanity end up obscuring from our view the very things about being human that we care about?

I was trying to get Bengio to acknowledge that there is a huge difference between superhuman computational speed or accuracy — and being superhuman, i.e., more than human. The most ordinary human does vastly more than the most powerful AI system, which can only calculate optimally efficient paths through high-dimensional vector space and return the corresponding symbols, word tokens or pixels. Playing with your kid or making a work of art is intelligent human behavior, but if you view either one as a process of finding the most efficient solution to a problem or generating predictable tokens, you’re doing it wrong.

Bengio refused to grant the premise. Before I could even finish the question, he demanded: “You don’t think that your brain is a machine?” Then he asked: “Why would a machine that works on silicon not be able to perform any of the computations that our brain does?”

The idea that computers work on the same underlying principles that our brains do is not a new one. Computational theories of the mind have been circulating since the 20th century origins of computer science. There are plenty of cognitive scientists, neuroscientists and philosophers who regard computational theories of mind as a mistaken or incomplete account of how the physical brain works (myself among them), but it’s certainly not a bizarre or pseudoscientific view. It’s at least conceivable that human brains, at the most basic level, might be best described as doing some kind of biological computation.

So what surprised and disturbed me about Bengio’s response was not his assumption that biological brains are a kind of machine or computer. What surprised me was his refusal to grant, at least initially, that human intelligence — whether computational at the core or not — involves a rich suite of capabilities that extend well beyond what even cutting-edge AI tools do. We are more than efficient mathematical optimizers and probable next token generators.

I had thought it was a fairly obvious — even trivial — observation that human intelligence cannot be reduced to these tasks, which can be executed by tools that even Bengio admits are as mindless, as insensible to the world of living and feeling, as your toaster. But he seemed to be insisting that human intelligence could be reduced to these operations — that we ourselves are no more than task optimization machines.

I realized then, with shock, that our disagreement was not about the capabilities of machine learning models at all. It was about the capabilities of human beings, and what descriptions of those capabilities we can and should license.

What Is Superhuman AI?

On his website, Bengio defines “superhuman AI” as an AI system that “outperforms humans on a vast array of tasks.” That’s pretty vague. What falls under the definition of a task? Is anything a human being does a task?

For decades, the AI research community’s holy grail of artificial general intelligence (AGI) was defined by equivalence with human minds — not just the tasks they complete. IBM still echoes this traditional notion in its definition of the AGI-focused research program Strong AI:

[AGI] would require an intelligence equal to humans; it would have a self-aware consciousness that has the ability to solve problems, learn, and plan for the future. … Strong AI aims to create intelligent machines that are indistinguishable from the human mind.

But OpenAI and researchers like Geoffrey Hinton and Yoshua Bengio are now telling us a different story. A self-aware machine that is “indistinguishable from the human mind” is no longer the defining ambition for AGI. A machine that matches or outperforms us on a vast array of economically valuable tasks is the latest target. OpenAI, which led the way in moving AGI’s goalposts, defines AGI in their charter as “highly autonomous systems that outperform humans at most economically valuable work.”

OpenAI’s AGI bait-and-switch wipes anything that does not count as economically valuable work from the definition of intelligence. That’s a massive erasure of our human capacity and a reduction of ourselves that we should resist. Are you no more than the work you completed today? Are you any less human or less intelligent if you spent your waking hours doing things that do not have well-defined “solutions,” that are not tasks that can be checked off a list, and that have no market price?

Once you have reduced the concept of human intelligence to what the markets will pay for, then suddenly, all it takes to build an intelligent machine — even a superhuman one — is to make something that generates economically valuable outputs at a rate and average quality that exceeds your own economic output. Anything else is irrelevant.

As the ideology behind this bait-and-switch leaks into the wider culture, it slowly corrodes our own self-understanding. If you try to point out, in a large lecture or online forum on AI, that ChatGPT does not experience and cannot think about the things that correspond to the words and sentences it produces — that it is only a mathematical generator of expected language patterns — chances are that someone will respond, in a completely serious manner: “But so are we.”

According to this view, characterizations of human beings as acting wisely, playfully, inventively, insightfully, meditatively, courageously, compassionately or justly are no more than poetic license. According to this view, such humanistic descriptions of our most valued performances convey no added truth of their own. They point to no richer realities of what human intelligence is. They correspond to nothing real beyond the opaque, mechanical calculation of word frequencies and associations. They are merely florid, imprecise words for that same barren task.

I am still not sure whether Bengio himself truly believes this. Later in the Q&A following his talk, he asked to revisit my question, and it seemed that he wanted to strike a more conciliatory tone and seek some common ground. But when he refused to grant that humans are more than task machines executing computational scripts and issuing the statistically expected tokens, I took him at his word. If beating us at that game is all it takes to be superhuman, one might think that silicon “superhumans” have been among us since World War II, when the U.K.’s Colossus became the first computer to crack a code faster than humans could.

Yet Colossus only beat us at one task; according to Bengio, “superhuman” AI will beat us at a “vast array of tasks.” But that assumes being human is to be nothing more than a particularly versatile task-completion machine. Once you accept that devastating reduction of the scope of our humanity, the production of an equivalently versatile task-machine with “superhuman” task performance doesn’t seem so far-fetched; the notion is almost mundane.

So what’s the harm in speaking this way?

Being Superhuman

The word “superhuman” means “human, but more so.” To be superhuman is to have the same powers that humans do, plus other powers we lack — or to have human powers to a degree that we don’t. It’s not a word we use for something that’s of a radically different kind from us, something that lacks fundamental human qualities and powers but performs better than we do on some metrics. We don’t talk about “superhuman airplanes” or “superhuman cheetahs” even though airplanes and cheetahs both travel faster than any human has ever run.

We use and understand the term superhuman to mean something very much like us, but better. The fictional Superman is perhaps the best-known English-language articulation of the superhuman idea. Superman is not Earth-born, but he embodies and far exceeds our highest human ideals of physical, intellectual and moral strength. He isn’t superhuman just because he flies; a rocket does that. He isn’t superhuman because he can move heavy things; for this, a forklift will do. Nor is he superhuman because he excels at a “vast array” of such tasks. Instead, he is an aspirational magnification of what we see as most truly human.

There are no fundamental dimensions of the human personality missing from Superman. He is an imagined answer to the question: “What if us, only more so?” He desires, he suffers, he loves, he grieves, he hopes, he cares and he doubts; he experiences all these even more intensely and deeply than we do. He is as far as one can be from a mindless producer of efficiencies. His embodiment as Superman is a direct expression of each of the aspects of humanity that we value most, the things about our kind that we tend to see as universally shared.

By describing as superhuman a thing that is entirely insensible and unthinking, an object without desire or hope but relentlessly productive and adaptable to its assigned economically valuable tasks, we implicitly erase or devalue the concept of a “human” and all that a human can do and strive to become. Of course, attempts to erase and devalue the most humane parts of our existence are nothing new; AI is just a new excuse to do it.

Indeed, for the entirety of the Industrial Age, those invested in the maximally efficient extraction of productive outputs from human bodies have been trying to get us to view ourselves — and more importantly one another — as flawed, inefficient, fungible machines destined to be disposed of as soon as our output rate slips below an expected peak or the moment a more productive machine can be found to step in.

The struggle against this reductive and cynical ideology has been hard-fought for a few hundred years thanks to vigorous resistance from labor and human rights movements that have articulated and defended humane, nonmechanical, noneconomic standards for the treatment and valuation of human beings — standards like dignity, justice, autonomy and respect.

Yet to finally convince us that humans are no more than mechanical generators of economically valuable outputs, it seems to have only required machine tools that generate such outputs in our primary currencies of human meaning: language and vision. Now that you can elicit an infinite multitude of these currencies from an app on your smartphone, we accept the advent of “superhuman AI” as a foregone conclusion, something already quite literally at hand.

Reclaiming Our Humanity

The battle is not lost, however. As the philosopher Albert Borgmann wrote in his 1984 book “Technology and the Character of Contemporary Life,” it is precisely when a technology has nearly supplanted a vital domain of human meaning that we are able to feel and mourn what has been taken from us. It is at that moment that we often begin to resist, reclaim and rededicate ourselves to its value.

His examples might seem mundane today. He wrote about the post-microwave revival of the art of cooking as a cherished creative and social practice, one irreplaceable by even the most efficient cooking machines. Indeed, the skilled and visionary practice of cooking now carries far greater cultural value and status than it did in the late 20th century. Similarly, the treadmill did not eliminate the irreplaceable art of running and walking outdoors just by offering a more convenient and efficient means to the same aerobic end. In fact, Borgmann thought the sensory and social poverty of the experience of using a treadmill or microwave could reinvigorate our cultural attention to what they diminished — activities that engage the whole person, that continually remind us of our place in the physical world and our belonging there with the other lives who share it. He was right.

Perhaps the ideology of “superhuman” AI, in which humans appear merely as slow and inefficient pattern matchers, could spark an even more expansive and politically significant revival of humane meaning and values. Maybe the moral and experiential poverty of AI will bring the most vitally human dimensions of our native intelligence back to the center of our attention and foster a cultural reclamation and restoration of their long-depreciated value.

What might that look like? Imagine any sector of society where the machine ideology now dominates and consider how it would look if the goal of mechanical optimization became secondary to enabling humane capabilities.

Let’s start with education. In many countries, the former ideal of a humane process of moral and intellectual formation has been reduced to optimized routines of training young people to mindlessly generate expected test-answer tokens from test-question prompts. Generative AI tools — some of which advertise themselves as “your child’s superhuman tutor” — promise to optimize even a kindergartener’s learning curve. Yet in the U.S., probably the world’s tech-savviest nation, young people’s love of reading is at its lowest levels in decades, while parents’ confidence in education systems is at a historic nadir.  

What would reclaiming and reviving the humane experience of learning look like? What kind of world might our children build for themselves and future generations if we let them love to learn again, if we taught them how to rediscover and embrace their humane potential? How would that world compare to one built by children who only know how to be an underperforming machine?

Or consider the economy. How would the increasingly sorry state of our oceans, air, soil, food web, infrastructures and democracies look if we stopped rewarding mindless, metastatic growth in “domestic product” that we make machines (human or silicon, whichever is cheaper) churn out in any environmentally or socially poisonous form that can sell? How would the future we are headed for change if we mandated new economic incentives and measures tied to medium- and long-term indicators of health, sustainability, human development and social trust and resilience?

What if tax relief for wealthy corporations and investors depended entirely on how their activities enabled those humane indicators to rise? How would our jobs change, and how might young people’s enthusiasm for investing their energies in the workforce be boosted, if the measure of a company’s success were not simply the mechanical optimization of its share price, but a richer and longer-term assessment of its contribution to the quality of our lives together?

What about culture? How different would the future look if current efforts to use AI to replace human cultural outputs were stalled by a renewed affection for our own capacity to create meaning, to tell the world’s stories, to invent new forms of beauty and expression, to elevate and ornament the raw animal experience of living? What if, instead of replacing these humane vocations in media, design and the arts with mindless mechanical remixers and regurgitators of culture like ChatGPT, we asked AI developers to help us with the most meaningless tasks in our lives, the ones that drain our energy for everything else that matters? What if you never had to file another tax form?

What if we designed technologies like AI with and for the benefit of those most vulnerable to corruption, exploitation and injustice? What if we used our best AI tools to more quickly and reliably surface evidence of corrupt practices, increase their political cost and more systematically push corruption and exploitation toward the margins of public life? What if populations collectively vowed to reward only those politicians, police and judges willing to take the risks of demonstrating greater transparency, accountability and integrity in governing?

Even in these more humane futures, we’d be far from utopia. But those possible futures are still much brighter than any dominated by the ideology of superhuman AI.

That doesn’t mean that AI has no place in a more humane world. We need AI to take over inherently unsafe or human-unfriendly tasks like environmental cleanup and space exploration; we need it to help us slash the costs, redundancies and time burden of mundane administrative processes; we need AI to scale up infrastructure maintenance and repair; we need AI for the computational analysis of complex systems like climate, genetics, agriculture and supply chains. We are in no danger of running out of important things for our machines to do.

We are in danger of sleepwalking our way into a future where all we do is fail more miserably at being those machines ourselves. Might we be ready to wake ourselves up? In an era that rewards and recognizes only mechanical thinking, can humans still remember and reclaim what we are? I don’t think it is too late. I think now may be exactly the time.