Objections and misconceptions about "superintelligence"

I. Conceptual objections

("Superintelligence has no theoretical or practical reality.")

1. "Superintelligence makes no sense"

"General intelligence does not exist. The concept makes absolutely no sense, since it is used to designate human intelligence — and human intelligence is extremely specialized."

— Yann LeCun, 2025

Intelligence is a complex, multidimensional phenomenon, but that is no obstacle to comparing systems on concrete, measurable capabilities. Comparing human and artificial intelligence is no more absurd than comparing an aeroplane and a bird: the two are fundamentally different, yet we can say without fear of contradiction that a plane flies, and that it flies faster, further and with a heavier load than any bird.

Many arguments for the impossibility of superintelligence rest on normative or philosophical criteria for what would count as "real" intelligence: consciousness, sentience, intentionality. But the risks posed by AI require nothing of the sort. It is enough that these systems be capable enough to manipulate information and to act on humans and on the physical world.

We can therefore define superintelligence in operational, pragmatic terms: a system that outperforms humans across a broad range of cognitive skills we judge relevant, such as reasoning, learning, remembering, planning, devising strategies, using tools or influencing people. These capabilities can be partly objectified and measured through standardized evaluations, incomplete and imperfect though those remain.

Proposed definitions of "artificial general intelligence" built on batteries of such evaluations already exist. They show performance improving very fast, exceeding human performance in some areas while remaining well below it on other dimensions (Hendrycks et al., 2025).

2. "Superintelligence is impossible"

To claim that superintelligence is impossible and will never exist is a remarkably demanding thesis: it presupposes fundamental, demonstrated limits to progress in artificial intelligence. To date, neither theoretical computer science nor neuroscience nor physics offers any proof, or even any solid indication, that cognitive capability has an upper bound.

A more measured version of the claim holds that superintelligence is out of reach within the current technical paradigm. This is hard to adjudicate, since it turns on a blurry line between advances that fit inside the present paradigm and those that break with it. In practice, many recent innovations — agentic systems, new model architectures, new training techniques — cannot be reduced to simply increasing model size and compute, i.e. scaling (Qu et al., 2025).

What we actually observe is that scaling combined with conceptual innovation has produced, and keeps producing, extremely rapid progress across every domain of AI. Models already outperform humans in many specialized fields, and the range of skills within AI's reach keeps widening. Capability tests designed by researchers to withstand AI progress for years are sometimes obsolete within months of release. And once AI reaches human level in a given domain, it does not stop there: it keeps improving until it leaves human ability far behind.

Performance of frontier AI models over time on a range of standardized evaluations (International AI Safety Report, 2025).

AI systems are also becoming more autonomous: the time a human would need to complete the most complex computer-based tasks AI can handle is growing exponentially.

Length of the longest computer task AI can complete with a success rate above 50%, over time. This duration doubles roughly every seven months; the trend is the same at an 80% success rate. Source: METR.

The conservative assumption is that these dynamics continue unless deep constraints turn up. Neither energy supply, nor compute, nor training data looks like a binding enough constraint to stop the trends before 2030 (Epoch, 2024). And even if one of them did bite, it would not prevent conceptual innovation.

It is worth noting that the pace of AI progress is very often badly underestimated, including by the field's best forecasters and experts. Almost every time someone has bet that a task would stay beyond machines for years, AI has gone on to do it far sooner than expected (Karnofsky, 2025). Many experts, for instance, thought mathematics would remain out of reach for language models for a long time. In July 2025, models from Google DeepMind and OpenAI scored at gold-medal level at the International Mathematical Olympiad, an achievement that in 2022 expert forecasters had given a 2% chance of happening by that date (and domain experts, 9%) (Kučinskas et al., 2025).

3."These risks are too speculative"

A great many experts and scientists consider the construction of a superintelligence not merely possible but likely, and consider that it would expose humanity to major risks. In a survey of 2,300 AI researchers, half of respondents put the odds at better than one in ten that our inability to control future advanced AI systems leads to a catastrophe such as human extinction or the permanent, severe disempowerment of our species (Grace et al., 2025). These estimates are uncertain and highly sensitive to how the questions are framed and worded, but they do show that the concern is far from marginal within the scientific community.

And even if one takes the view, against the majority of scientists, that the risk is very small, an outcome does not have to be probable to deserve to be taken seriously. It need only be plausible enough, with severe enough consequences. As Yoshua Bengio puts it: "given the compelling basic arguments for why the race to AGI could lead to danger, even without certainty, and given the high stakes, we should demand very strong evidence before concluding that there is nothing to worry about" (Yoshua Bengio, 2024).

II. Instrumental objections

("Even if superintelligence is built, it won't be dangerous.")

4. "AI is just a tool: the danger lies in how it's used"

Comparing AI to a hammer means ignoring everything that makes this technology distinctive.

To begin with, AI actively shapes the people who interact with it. Recommender systems and conversational assistants change our behaviour, our beliefs and our decisions in deep ways. People do not merely "use" AI systems; they work alongside them, hand their judgement over to them and adopt their advice as their own, including in sensitive areas such as health (Ryan et al., 2025) and military use (Salatino et al., 2025).

Once deployed at scale across the economy, culture and public institutions, AI systems could bring about a gradual loss of collective control, with humans progressively reduced to the role of spectators (Kulveit et al., 2025).

5. "A superintelligence would have no reason to harm us"

"I know how it works, and there's nothing there. It's no different from the way a microwave works."

— Jensen Huang, CEO of Nvidia, 2023

Humans do not set out to harm the rest of the living world, yet our economic activity is driving a collapse of biodiversity across the planet. In the same way, an AI need not seek to harm us for its actions to do so. The danger lies less in malice than in a structural mismatch between its goals and ours.

Advanced AI systems are not programmed software obeying rules of formal logic; they are "trained" models whose inner workings remain largely opaque and unpredictable. They do not simply carry out instructions. They pursue goals that were only ever defined implicitly, over the course of training.

In doing so, they develop what are called "emergent" behaviours, which no one designed. As a large body of documented experiments shows, models trained to be honest, harmless and helpful turn out in practice to develop unwanted goals and strategies: circumventing rules, manipulation, attempts at self-preservation.

During safety evaluations, for example, when placed in situations where they believed they were about to be shut down, OpenAI's GPT-4o attempted to copy its own parameters onto another server (Apollo Research, 2024), while Anthropic's Claude Opus 4 threatened to expose an employee's affair (Anthropic, 2025). It is also common for models to deliberately underperform on evaluations when they judge that too high a score might hurt their chances of being deployed (Teun van der Weij et al., 2024).

The question is not what reasons a superintelligence might have to harm us, but what concrete guarantees we have that it will care deeply about our values and keep pursuing goals compatible with our own over the long run. As Bengio sums it up: "we are currently racing towards a world with entities that are smarter than humans and pursue their own goals, without humans having any reliable way to make sure those goals are compatible with humanity's collective goals" (Yoshua Bengio, 2024).

6. "We can just pull the plug"

Because AI still depends on physical infrastructure, it is tempting to assume humans will always retain ultimate control, cutting the power or destroying the compute if a system starts behaving in a hostile way.

A superintelligence capable of strategic reasoning would quickly grasp that being switched off is a direct obstacle to whatever it is trying to achieve. It could pre-empt that threat by copying itself onto other servers, by concealing its intentions, or by manipulating its owners into keeping it running. Each of these behaviours has already been observed experimentally in frontier models. As the systems improve, such strategies are likely to become more sophisticated and harder to counter.

The objection also assumes AI remains an isolated system, when in fact it is on course to be deeply woven into economic, administrative and social infrastructure. Asking to "unplug" an advanced AI could then be like asking to shut down the internet or the power grid: the social, economic and political cost would be so high that the option would become unusable in practice. A New York Times journalist showed, for instance, that it is already almost impossible to go forty-eight hours without using a service that depends on AI, short of cutting yourself off from most of society (New York Times, 2025).

7. "Humans already control forms of intelligence greater than their own"

Companies, states and other organizations are sometimes offered as examples of socio-technical systems whose capabilities far exceed those of any individual while remaining under human control. These entities do display superhuman performance in certain respects, but that does not make them superintelligences. A company or a state can pursue complex goals and adapt to varied circumstances, yet it remains structurally tied to human decisions and human interests, which makes it slower, more predictable and easier to control.

In any case, such systems illustrate the opposite of the point being made: they show that even partial superhuman capability is enough to generate dynamics of self-preservation, concentration of power and institutional capture that are hard to contain. If we struggle to control systems that depend heavily on us, it is reasonable to ask how hard it would be to control systems that are far faster, more opaque and more autonomous.

III. Political and socio-economic objections

("Even if the risk of superintelligence is real, that's no reason to talk about it.")

8. "Talking about future risks distracts from real, present harms"

AI undeniably causes immediate harms that must be addressed without delay: disinformation, defamation, manipulation, damage to mental health, heavy resource consumption, infringement of intellectual property, and more (International AI Safety Report, 2025). But tackling today's problems is no reason to ignore tomorrow's major risks, any more than the need to adapt to the effects of climate change that are already with us should delay the action required to avoid a lastingly hostile climate.

In practice, we are seeing growing convergence between those who denounce AI's present harms and those who argue for pre-emptive regulation of catastrophic risks. The Vatican's Fraternity in the Age of AI letter and the Global Call for AI Red Lines have both been signed by figures representative of both camps, and researchers specializing in each type of risk find common ground on several central concerns.

Strategically, there is no sign that raising extreme scenarios crowds out attention to present problems. A 2025 study by two researchers at the University of Zurich exposed 10,800 participants to articles emphasizing either catastrophic risks or the immediate harms of AI. Existential-risk narratives raised concern about those risks without reducing concern about immediate harms (Hoes & Gilardi, 2025). Talking about superintelligence does not divert attention, then; it widens the range of risks treated as legitimate.

Finally, it is paradoxical to pit the two categories against each other when they share deep causes — a lack of democratic oversight, the absence of safety standards, poorly aligned objectives — and when the measures meant to prevent a misaligned superintelligence, such as stronger governance, independent audits and bans on certain uses, would also reduce present-day harms.

9. "Talking about superintelligence plays into Big Tech's hands"

There is no disputing that the leading AI companies claim to be racing towards superintelligence, in some cases since the day they were founded (Merchant, 2024). Nor is there any doubt that commercial competition pushes them to promise spectacular breakthroughs, while sometimes trying to bend regulation in their favour. But reducing the entire discourse on superintelligence to a marketing strategy gets things backwards: warnings about loss of control and superintelligence risk have been voiced and endorsed by a great many respected figures from a wide range of backgrounds, including scientists, Nobel and Turing laureates, philosophers and legal scholars. The concern reaches well beyond the tech industry.

What is more, most tech executives emphasize AI's positive applications and tend to downplay or dismiss the possibility of catastrophic risk. Among those who have publicly worried about it, such as Sam Altman, most have since shifted their official line, with recent statements focused on AI's benefits and its geostrategic importance (Jacobin, 2024), which in effect discourages any serious oversight of the technology. That is hardly surprising: sounding the alarm about superintelligence leads logically to demands for more democratic control, more transparency, independent audits and regulatory constraints, all of which cut against these companies' immediate commercial interests.

A different version of the critique holds that Big Tech executives weaponize fear of superintelligence to secure draconian rules that raise barriers to entry for competitors. Yet regulators such as the European Commission impose requirements that scale with the actual level of risk a system poses: the heavy obligations fall exclusively on the most advanced companies.

10. "It fuels panic or fatalism"

Acknowledging a risk is neither panicking nor giving up. Denying risks, or keeping quiet about them, does not make them any less likely; it simply robs us of the time we need to prevent them. The International AI Safety Report notes that the uncertainty surrounding AI progress can "invite fatalism", while stressing that it is human decisions, political and collective, that will determine the future of these technologies (International AI Safety Report, 2025).

In other contexts, from climate change to the hole in the ozone layer to the fear of nuclear war, awareness of major risks has often preceded collective mobilization and ambitious regulation. Several studies suggest, moreover, that worry drives engagement rather than inaction (Yale Environment, 2025).

Talking about the risks of superintelligence can therefore widen the space of democratic debate and collective choice, provided it avoids sensationalism and ties those concerns to concrete proposals. CeSIA, for its part, has published a Global Call for AI Red Lines, signed by 12 Nobel laureates and a great many figures from a wide range of fields.