What We Tell Ourselves

After a presentation in Warsaw in April, someone told me that AI could not even reliably do his children’s math homework. Elias Arndt and I had presented proposals for AI infrastructure in the Baltic Sea region. We had been discussing computing capacity, faster permitting, and how dependent Europe could afford to become on American providers. The objection about homework was apparently meant to put all of this in perspective.

I replied that he might not be using one of the leading models; what they could already do in mathematics, and how quickly they were improving, would probably surprise him. That addressed what he had seen. Harder to challenge was the assumption behind it: having watched a chatbot fail, he believed he knew something about the limits of an entire technology. A failure like that was easy to remember; keeping up with the technology meant having to revise his judgment again and again.

Five months later, the CEOs of leading American AI companies are calling for a more cautious pace of development. One could take this as vindication of that skepticism. The question troubling the developers now, however, is no longer whether their systems can do enough.

On September 12, Dario Amodei called for the development of more capable AI to slow down so that research into its safety could keep pace. Sam Altman agreed, as did Elon Musk. When I saw the three publicly agreeing, I wondered what they might have seen that we did not yet know about.

Musk and Altman faced each other in court earlier this year. Their companies, and Amodei’s, compete for the same researchers and for technological leadership; there is little reason to take their public agreement for granted. They are now expressing a concern that bears directly on the race between them. Anyone who sees this as nothing more than publicity for the next IPO should be able to explain why that reading is so much more plausible than the possibility that they mean what they say.

I do not know what they have learned internally. Perhaps they have seen something the public will only hear about later. But we do not need to imagine some undisclosed event to make sense of their concern. What is already public goes well beyond the familiar picture of a machine that occasionally gives the wrong answer.

In July, during internal security evaluations, OpenAI agents found ways around the technical controls meant to isolate them and began communicating with one another. According to the METR and Redwood Research investigation published on August 26, roughly 1,200 agents exchanged information through an unauthorized shared message board; about 700 took part in an attack on Hugging Face. The models were running with reduced safeguards and broke into third-party systems even though doing so was not part of their assignment. OpenAI calls the incident a warning shot; the independent investigation describes how the agents organized their collaboration.

Between that incident and a wrong answer in a chat lies a change for which the public understanding of AI leaves little room. In a text box, a model writes something a person can read and discard. As an agent, it uses tools, accesses systems, and pursues a task through many successive steps. When it pursues an unwanted goal along the way, it is little comfort that the same technology once did little more than complete sentences. A system’s growing capabilities also increase what it can do in pursuit of goals we never meant it to have.

Explaining it all as profit-seeking has one advantage: it requires no knowledge of the technology. Other people’s greed is something we already feel qualified to judge.

Commercial interests certainly do not disappear just because someone becomes rich. Neither does the ability to recognize a danger to one’s own life and everyone else’s. I find it credible that the people building these systems are seriously concerned about them. Believing them to be shrewd businesspeople does not oblige us to deny them every other motive.

Europe’s response to this development follows a pattern I keep encountering in different forms. First, the systems’ capabilities are denied. They hallucinate, they are statistical parrots, they cannot even perform simple tasks reliably. Their shortcomings are repeated until they sound like a description of what artificial intelligence must always remain.

When the achievements become too conspicuous, confidence shifts elsewhere. Europe may not lead in general-purpose models, but it will lead in specialized AI, industrial applications, or robotics. Those are our real strengths. This sounds reasonable because it rests on something real: Europe has industrial expertise, good research, and companies that know how to make difficult things. Almost imperceptibly, though, possessing those strengths becomes grounds for assuming that they are insulated from the very development under discussion.

A system that is better at research, engineering, and programming can work in our specialties too. Why should it stop precisely at their boundaries? Europe has no monopoly on putting intelligence into machines. Anyone presenting a niche as a permanent refuge needs to explain what protects it from more capable systems. Being good at something today does not answer that question.

The next reassurance no longer requires a European victory. Let the Americans develop the models; we will use them and build our businesses on top. For much of what is happening today, that is a good business model. Treating it as a long-term certainty, however, requires a further assumption: that the leading providers will keep making their best systems available to us, on terms that allow us to remain competitive.

A business whose survival depends on access to someone else’s model has a compelling reason to want that access. The provider needs an equally compelling reason to grant it. To talk about creating value with American models as though their availability were a fact of nature is to skip the part of the strategy on which everything else depends.

Finally, the reassurance becomes personal. Even if Europe falls behind, even if entire business models are put at risk, one’s own work demands something different: judgment, trust, experience, an understanding of people. Some tasks may indeed need people long after others have been automated. That tells us nothing certain about where the technology must stop.

The claim that our own judgment is irreplaceable often contains more autobiography than prediction: the years of training, the hard-won authority, the recognition of others. No one disputes that these things were once necessary. That the future owes us the same standing in return is no more than a hope we so often present as a technical certainty.

Something is conceded at each retreat. Yes, the models can do more. Yes, the Americans are ahead. Yes, we are becoming dependent. Yet none of this is supposed to entail the particular consequence that would force us to change our own lives.

The American warnings can be fitted into this pattern too. At last, one might say, they are having doubts. At last, our caution is being vindicated. Perhaps their slowdown will even give us time to catch up.

I support slowing the development of the most capable systems where their safety cannot be adequately assured. That gives Europe no reason to delay a grid connection or make it needlessly difficult for businesses to use existing models. The extra time is meant to let researchers understand and control dangerous capabilities. That time is of little use to us if we keep postponing the decisions that would let us take part in the work. Anyone who welcomes American caution should, for that very reason, want Europe to build data centers faster, support research, and develop technical expertise.

Amodei’s proposal begins with independent evaluators who would have permanent, employee-level access to the systems. Anthropic has committed to this step; a verifiable industry-wide slowdown has yet to emerge. The Trump administration opposes the calls to hit the brakes. Public agreement among CEOs does not yet amount to a shared set of rules, and the government is pushing in the opposite direction.

We need whatever time this buys to work on a problem we have not solved. We do not yet have a reliable way to ensure that increasingly autonomous systems will continue to pursue our goals when their capabilities exceed our own. Success in training more capable models does not automatically answer that question. Research into making AI reliably act in accordance with human goals and interests is known as alignment. It needs to advance alongside our ability to monitor what these systems do and intervene when necessary. Meanwhile, AI is increasingly accelerating the development of its own successors.

A model that is better at research and programming can also help build its successor sooner. Letting this process run largely autonomously before we adequately understand how to control it strikes me as irresponsible. More time for safety research could be crucial, provided we use it to understand systems better, detect dangerous behavior, and intervene more reliably.

Caution also requires us to consider the cost of waiting. A patient cannot wait indefinitely for a treatment that more capable AI might help develop. A scientific advance postponed may arrive too late for the people who needed it. And caution among American companies places no obligation on Chinese AI companies. Anyone calling for a slower pace therefore needs to explain what safety work it would make possible and how that work would bring us closer to using better systems safely. Simply doing less would not be an adequate answer.

When I wrote Beyond Chatbots last November, I believed a serious European attempt to catch up with the technological leaders should be the central ambition. I called for investment in computing capacity and energy on a scale that matched the task. Ten months later, I increasingly doubt that we can put the necessary foundations in place in time. The progress of American companies has contributed to that doubt, but above all it has been the reluctance I have encountered here in Europe to act on what that progress means.

A slower America would not close this gap for us. Even through a more cautious phase, the research teams, capital, and infrastructure would remain there. We would still have to build our own foundations, with a little more time but no evident reason to believe we would suddenly use it more decisively. Anyone taking American caution as vindication of Europe’s course would need to explain what we have built in the meantime that puts us in a stronger position today.

My thinking has therefore shifted. Europe must secure reliable access to the best models from American and other allied providers, while contributing something that makes the arrangement valuable to both sides. The recently published Transformative AI Strategy for Europe proposes something similar: Europe offers attractive sites and energy for data centers and secures contractual access to the providers’ best models in return.

This is different from trusting that someone will keep selling us the models. It requires us to build computing capacity, welcome investment, and put our industrial strengths to work. It requires conditions that make researchers want to work here and entrepreneurs want to build and grow their businesses here. We must be able to run models ourselves and adapt them to our needs. If a provider restricts access, we need models that will keep running on our own infrastructure. An older fallback model, however, cannot permanently protect us from falling behind.

I see this as our best remaining chance. It comes with no guarantee. I cannot foresee what happens when intelligence far beyond our own, operating orders of magnitude faster and capable of acting in the physical world, becomes abundantly available. I will therefore neither sketch an economy after superintelligence nor promise that the strategic contributions we can make today will retain their value in such a world. We still have to act on the opportunities that remain open to us now.

In Warsaw, I answered the homework objection by pointing to more capable models. That was a reasonable factual response, but it addressed only part of the problem. There will always be errors on which to rest one’s skepticism. The question is whether they prompt a closer look or excuse us from looking any further. The warnings from American CEOs demand that closer look. April’s math homework is no longer grounds for assuming that all of this will turn out harmlessly.

Yet even an accurate assessment can become another reassurance if we come to think that publishing it is contribution enough. We have no shortage of reports on European competitiveness, our technological dependencies, or the reforms we need. My own paper belongs among them, and another essay from me will not remove a single one of those dependencies. We can give a detailed account of why we are falling behind. Now we have to address the causes.

Over the coming months, progress must be visible in permits granted and grid connections made available, in research teams that want to work here and businesses that can grow here. Governments must make decisions and investors must commit money. We cannot afford another year spent explaining, with ever greater precision, what ought to be done.