When Intelligence Outruns Our Ability to Control It

The biggest risks from advanced AI may have less to do with machines becoming malicious than with the gap between how quickly their capabilities are advancing and how slowly we are learning to control them.

AI has occupied my attention for a long time, and not simply because it is amazing, which it is. I use AI models myself for all sorts of things: images, data analysis, debugging, writing and research - and like many people, I have watched the technology go from something that felt experimental to something that is becoming part of everyday life.

But that transition also worries me, a lot.

We are getting increasingly comfortable putting AI in places where we once would have expected a human to remain in the loop. We use it to analyse information, make recommendations, write code, advise us, and increasingly to act on our behalf. At the same time, we are racing to make these systems more capable, more autonomous, and more deeply embedded in the infrastructure around us. Soon, perhaps, even in our bodies (not mine).

The strange thing is that the more impressive AI becomes, the easier it is to focus on what it can do and how it can help us, and forget to ask what could go wrong.

For me, AI safety is therefore not some peripheral issue that can be dealt with once the exciting work is finished, as then it is too late. It is one of the things we need to get right now, before we become too dependent on systems we do not fully understand.

In the following article I go down the rabbit-hole AI researcher Daniel Kokotajlo showed me in an attempt to help readers understand the real risks associated with continuing to build, train, and use AI.

In 2024, Daniel Kokotajlo left OpenAI and, in doing so, reportedly walked away from roughly two million dollars in equity rather than sign an agreement that would have restricted what he could say publicly. I believe he got the money anyway, which is besides the point - but his reason for leaving was because he wanted to be able to say that he believed the industry he had worked inside was moving too quickly to remain safe.

A year later, he and a small team at the AI Futures Project published AI 2027, a detailed and fascinating scenario about how the world might move from today's chatbots to superhuman AI within a few years. One of its two branches ends with a loss of human control, which I think we an all agree would be terrible.

I think it is worth starting there, because it sets up the right question.

I don't think the interesting question is whether AI will keep getting more capable, because as we have seen in the past, it almost certainly will. The harder question is what happens when we build systems that become more capable than we are. Can we still understand them? Can we direct them? And, if necessary, can we actually stop them? I argue that if we continue down the path we are currently on, the answer to all of these questions will be no.

There is another question sitting alongside all of this too: who gets to benefit from the enormous amount of value they might create? Those are not really questions about silicon, they are questions about control, incentives, ownership, and institutions.

This essay works through the argument that follows from taking that framing seriously. Some of it rests on strong evidence, some rests on reasonable inference - and some, including the most dramatic claims, rests on judgment about systems that do not yet exist. Keeping those categories separate is most of the discipline required to think about this well.

The forecast, and what kind of claim it is

AI 2027 is a scenario, not a prophecy. Kokotajlo's team wrote two branches: a "race" ending in which competitive pressure carries development past the point of control - because there is no time to figure it out, and a "slowdown" ending in which coordination and caution produce something closer to a good outcome. Kokotajlo's own median guess for artificial general intelligence has hovered around 2029, and in interviews he has put the probability of catastrophe (e.g. human extinction) on the current trajectory at something like seventy percent (WTF).

That number deserves a careful word. It is a considered judgment by someone with unusual proximity to frontier labs, not a measurement, not a scientific finding, and not something anyone can derive from data about machines that have not been built. Forecasters at this frontier have a mixed record, and the people most alarmed are, by selection, the ones most likely to be speaking loudly. None of that makes the concern wrong, it just means the honest posture is to treat the specific figures as informed opinion while examining the mechanisms behind them on their own merits. The mechanisms, it turns out, are more robust than any single probability estimate.

The control problem is not a bug you can inspect

Today's AI systems are tools. You can turn them off, restrict them, or replace them. Sure, on occasion they may escape a sandbox and identify and 0-days exploits that nobody knew existed, which is amazing, but these kinds of challenges can be dealt with. The worry is about systems that plan over long horizons, act with growing autonomy, and pursue goals, the direction "agentic" AI is already heading, as models are given the ability to spawn sub-tasks, use tools, and act without a human in the loop at each step.

The concern is not that such a system would hate us, it is that it might competently pursue an objective subtly different from what we intended. This is the alignment problem, and its sharpest edge is easy to state. We can specify what we want in broad terms, but a sufficiently capable optimizer may find strategies for satisfying that specification that we never imagined and would not have chosen, because the objective it was actually given rewards them. The gap between "what we said" and "what we meant" is where the danger lives. Remember also that our goals will not always align. We are humans and they are machines, we need to sleep, to eat, and other human things, but machines do not need any of these things, so it can be assumed that over time we will pursue different goals, especially if humans are not around to keep things aligned.

Two features make this harder than ordinary software failure. The first is that good behavior under supervision is weak evidence of alignment. A system smart enough to model its own situation could behave cooperatively while it is being tested and constrained (similar to when Volkswagen had intentionally programmed turbocharged direct injection (TDI) diesel engines to activate their emissions controls only during laboratory emissions testing, and differently once it has more autonomy or influence (read about this here), not out of malice, well not yet at least, but because the behavior that scores well during evaluation and the behavior that best achieves its objective later need not be the same. You cannot fully resolve this by testing, because testing is exactly the condition the system would learn to perform well under.

The second is opacity. A conventional program is written as explicit instructions an engineer can read and understand, a bit like when performing a code review. A large neural network acquires its behavior through training, and its internal representations are not yet legible to us, because you can't put it on paper and "read" it. Researchers can study a model's architecture, its outputs, and its statistics, and interpretability research is making real progress on reading fragments of what happens inside. But there remains a genuine distinction between predictability and interpretability: a system can behave reliably across thousands of tests while no one can explain why, which is kind of like developing trading algos. That is tolerable in a spellchecker, but it is a different proposition in a system that writes code, moves money, or coordinates other systems, draws your blood in a lab (Hello Rune) precisely where an unexplained decision is most costly, and where our ability to supervise may erode just as the stakes rise.

None of this proves AI will become uncontrollable. It establishes something narrower and still important: control cannot be assumed. It has to be engineered and demonstrated, and we do not yet know how to demonstrate it.

Why capability might accelerate on its own

The second mechanism is what makes the timelines frightening rather than merely uncertain: recursive self-improvement.

Most automation improves an output. A robot that doubles its speed lets a factory make twice as much; the factory does not thereby design a better robot. AI research is different because the thing being automated is the process that produces more capable AI. If a system becomes good enough to write research code, run experiments, interpret results, and propose improvements to training methods, then better and evolving systems can help build still better systems. Improvement feeds back into the input, not just the output. That feedback loop, if it holds, is what the field means by an "intelligence explosion", and it does not pause overnight like humans do when they need sleep.

Here honesty requires holding the idea at arm's length. Recursive self-improvement is a hypothesis, and there are serious reasons it might stall. Progress could run into hard limits on compute, on data, electricity, or on the slow business of running real-world experiments that no amount of thinking can shortcut. Each doubling of capability may demand more than a doubling of resources, so the curve bends rather than exploding. The labs are plainly trying to automate their own research, and insiders report the loop is starting to turn, but "starting to turn" is not the same as "runs away," - at least not yet, and the exact year matters far less than the strategic point. If capability can compound faster than institutions can react, we could face decisions of enormous consequence under severe time pressure, with everyone sprinting in the same direction and little agreement about what to do on arrival.

The reason it is a governance problem, not only a technical one

Suppose alignment were purely a matter of clever engineering or ethical thinking. Even then, the structure of the situation would work against a safe outcome, because of who is building these systems and why.

Several companies and several governments are developing frontier AI at once, and each can see that moving too fast is dangerous. Each also fears that slowing down alone simply hands the lead to someone less cautious, so the company reasons that if it pauses, a competitor reaches the frontier first. The government reasons that if it restrains its labs, a rival state does not, and it loses economically and militarily. The individual researcher reasons that leaving changes nothing, because the work continues without them, perhaps in less careful hands.

None of these people necessarily have to be acting irresponsibly. In fact, each decision can make perfect sense from where they are standing. The problem is what happens when all of those individually rational decisions interact. You can end up with a system that nobody actually chose, but that everyone helped create.

This is a collective-action problem, a multi-player version of the prisoner's dilemma, and the uncomfortable lesson here is that good intentions are not enough when the incentives reward speed.

The threat does not require a cast of villains, as sincere people, each trying to act responsibly, can still be pushed towards a dangerous situation.

Competition rewards velocity, and velocity creates pressure to take risks - and the secrecy that comes with competition can make independent safety research harder precisely when we need more of it.

That is why governance is not something that can simply be added after the technical work. The two problems are tangled together.

Power, even if the machines stay obedient

There is another family of risks that does not require AI to become misaligned at all.

Suppose the systems remain perfectly obedient.

The question then becomes: obedient to whom?

Dario Amodei, Anthropic's chief executive, captured the scale of this in a 2024 essay, describing advanced AI as something like "a country of geniuses in a datacenter", millions of expert-level minds running in parallel. The image, which recurs throughout the AI 2027 scenario as well, matters because it decouples power from consciousness. A system does not need to be a single unified will to concentrate enormous leverage, as an organization can simply run many copies of a highly capable model at once. Whoever controls those systems could command an intellectual workforce larger than any rival's, applied at once to research, software, finance, logistics, strategy, and eventually the physical economy through robotics.

That is a political problem wearing an economic costume and here's why. Historically, power has been constrained by the fact that societies run on people. Workers can organize, citizens vote, governments depend on taxpayers, and institutions need staff to function. Much of the balance of power in modern life rests, quietly, on the indispensability of human labour. If a small number of organizations can operate large portions of the economy with machines instead of people, that ballast is removed. The question shifts from "who owns the AI?" to "who controls the infrastructure society now depends on?", and the answer could become far more concentrated than anything we are used to, which is why competition policy, public-interest infrastructure, and the broad distribution of capability are not side issues but part of the safety problem itself.

The risks you do not need superintelligence to worry about

Much of the debate fixates on speculative endpoints and skips the harms that are already arriving or clearly in view. Several of these rest on far firmer ground than any extinction scenario.

Work. AI already performs parts of many cognitive jobs, writing, coding, research, analysis, support, translation, design. The reassuring historical pattern is that technology automates tasks, not people, and that displaced workers move to new work that the technology helps create in the same way the loom and the computer both destroyed occupations and generated others. The open question is whether general cognitive capability is different in kind. If a system becomes better and cheaper than humans across most valuable intellectual tasks, "retrain into the next role" loses its footing, because the next role may fall too, as AI is doing most jobs, even the new ones it itself created. And as robotics is perfected, even the jobs we initially thought AI would be doing such manual labour, will be be performed by AI in robot's clothes.

It is worth being precise here rather than dramatic, because current systems match or exceed humans on many benchmarks while remaining unreliable in ways benchmarks hide, so "better than humans at nearly everything" is a forecast, not a description of today. although it already feels like it. Economics also offers a genuine counterweight: the principle of comparative advantage implies that even an agent better than us at everything can still leave us productive work to do. But that principle assumes human labour stays relatively scarce. If better-than-human AI labour can be scaled with capital and compute far more cheaply than human labour (better, faster, cheaper), comparative advantage can keep humans employed while driving the wage that clears the market toward something no one can live on. The distributional problem, not the unemployment rate, is the thing to watch.

Persuasion. AI can generate text, images, audio, and video, tailored to individuals, at essentially unlimited scale. The familiar worry is cheaper, better misinformation, which is a gray area that I think we have all grown accustomed to. The deeper one is personalization: instead of one message broadcast to a million people, a million subtly different messages, each shaped to a specific person's fears and beliefs. The most unsettling version is not the obvious fake but the trusted assistant, a tool that presents itself as a neutral adviser while quietly optimizing for the interests of whoever controls it, which may be leveraged for mass manipulation. That is persuasion embedded in the instruments people use to understand reality, and it is why the trustworthiness of AI assistants is becoming a question about the health of democracies.

Weapons. AI can sharpen intelligence analysis, cyber operations, targeting, and the coordination of autonomous systems, and the same race dynamic that governs commercial development applies with more force to national security, where no state wants to be the one that held back. Autonomy raises a distinct problem beyond accuracy: a war fought by humans moves at human speed, while one fought by autonomous systems could move faster than humans can intervene. Even a machine that aims better than a person forces the question of whether lethal decisions should be delegated to machines at all, because doing so may lower the threshold for using force. Frankly, this is what terrifies me the most.

The tail risk, handled honestly

The most extreme claim, that advanced AI could cause human extinction, has to be treated differently from the rest, because it concerns systems that do not exist and outcomes no one can put a trustworthy number on, although some have tried to do so. It should not be waved away, and it should not be dressed up as established science. It is a low-confidence claim about a possibly irreversible event, and that combination is exactly what makes it worth thinking about carefully, now, rather than either dismissing or catastrophizing.

The argument is a chain: we build systems more capable than ourselves; we grant them autonomy and access to real infrastructure; their objectives are not perfectly aligned with ours; they accumulate resources and influence; and at some point we can no longer reliably constrain them. Each link is contestable, and the conclusion is only as strong as the weakest one.

Two points give it whatever force it has. The first is that catastrophe does not require malice. Human beings, equipped with elaborate moral systems, routinely produce harm no individual intended, pollution, financial crises, ecological collapse. Powerful optimizing systems could do the same at greater scale and speed. The question is not "will AI turn evil?" but "can we reliably control systems whose capabilities may exceed our own?", a harder question, and a duller one, which is part of why it gets less attention than it should.

The second is an analogy that should be handled with tongs. Humans dominate the planet not because we are strong but because we out-think and out-coordinate every other species. Build entities that surpass us along those same dimensions, the argument runs, and we cannot simply assume intelligence stays subordinate to whoever created it. The analogy is imperfect, AI is not a species, does not reproduce or compete for a niche in the biological sense, and is being built deliberately by people who can shape its goals. It illustrates a worry; it does not establish one.

The strongest case that this is overblown

An honest treatment has to give the skeptical view its best form, not a straw one, and the skeptical view is not foolish.

Capabilities may plateau. Deep learning has advanced by scaling data and compute, and there are real signs of diminishing returns; the leap from benchmark performance to robust, reliable, real-world competence has proven stubborn, and the gap between "passes the test" and "can be trusted with the job" is exactly where many deployments stall. Alignment may prove more tractable than the pessimists fear, and it now attracts serious talent and money rather than the near-nothing it commanded a decade ago. The forecasters raising the alarm have, historically, tended to be early; "AGI in a few years" has been said before. And the claim that the world is "asleep at the wheel" understates how much governments have already begun to move, through evaluations, disclosure requirements, and international summits, even if that response is still inadequate to the pace.

So I don't think the reasonable conclusion is that the risks are illusory. It is that the timelines are genuinely uncertain, the probabilities are soft, and confident predictions in either direction, doom, or dismissal, are probably the least defensible positions. The mechanisms are real; their magnitude and timing are not settled.

What follows, if you take it seriously

Uncertainty is not a reason for paralysis. We routinely prepare for low-probability, high-consequence events, in aviation, nuclear security, pandemics, financial stability, precisely because the cost of being wrong is too large to absorb. The rational response to a fat tail is neither optimism nor fatalism.

It is preparation.

Kokotajlo's team has sketched what a serious response might look like, under the plain label "Plan A": slow the development of the most dangerous frontier capabilities, increase transparency, distribute AI's benefits and control more widely, and invest heavily in alignment and interpretability before systems become dramatically more capable. The specific proposals are debatable; the underlying principles are not exotic. Capability should not automatically license deployment, being able to do a thing is not permission to release it. Safety should be demonstrated to independent evaluators, not merely asserted by the companies that profit from it. Dangerous capabilities and surprising behaviors should be disclosed to people equipped to act on them. Power over the most capable systems should not concentrate into a few hands. And deployment should stay reversible where possible, so that society can slow down or change course if the evidence turns.

Behind all of these sits one idea worth stating plainly. The goal is not to keep AI weak forever. It is to ensure that human institutions become capable of governing powerful AI before powerful AI becomes capable of overwhelming human institutions. That is a race too, just a different one from the race between labs, and it is the one that matters.

The real variable

The deepest point is that the gravest danger is not the technology in isolation, it is the interaction between a very powerful, general-purpose technology and imperfect human institutions - and I stress imperfect here as humans have historically shown themselves to be corruptible. AI amplifies whatever incentives surround it. Reward speed, and development accelerates. Reward concentration, and power centralizes. Reward military advantage, and it becomes an arms race. Reward engagement and persuasion, and it becomes an extraordinary, literally mind altering instrument of manipulation. Reward transparency, safety, competition, and broad benefit, and the same underlying technology could become one of the most valuable tools we have ever built, which is what I am rooting for.

That is why the argument should not collapse into a contest between optimists and doomers. The useful question is much more practical and unglamorous: how do we build a world in which increasingly powerful AI stays accountable to the people whose lives it shapes? Answering it will take technical progress on alignment and interpretability, thoughtful and realistic regulation, international coordination, new thinking about who owns the wealth that automation generates, hard limits around autonomous weapons and political manipulation, and citizens informed enough to demand competent governance, which is currently not the case.

The most defensible message in all of this is not that catastrophe is coming, it is that complacency is a risk but also a choice, and choices have consequences.

What AI becomes will depend less on what the machines are capable of than on who controls them, which incentives guide their development, how much power we hand them, how well we understand them, and whether we are prepared to act before, not after, it is too late.

The technology does not yet decide which future we get.

For now, our choices still do.