It is fashionable in certain circles to dismiss the catastrophic risks of AI. One often hears that “the real experts” who work on the technology every day are really not concerned at all; that only “doomers” and “Luddites” espouse a “fringe” view from a “position of ignorance”; that all talk of potential catastrophe is just “science fiction”.

Fortunately, an open letter has been published that lets us hear from the real experts who work on the technology every day, in their own words. And are they worried? Very.

The letter states: “There is a real risk that capability development rapidly accelerates beyond our ability to understand or control the resulting systems.” It asks the US government to support “an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development”.

AI systems going “beyond our ability to understand or control” would mean that we no longer have a say in whether we exist. AI systems escaping their confines and breaking into other companies’ computers – something that has become an almost daily occurrence – are an early warning.

To “pace the frontier”, as the letter asks, is to ensure that more advanced systems are developed and released only when it is demonstrably safe to do so. This requires a licensing process with strong verification of safety; registration of AI systems and the ability to monitor and terminate them when necessary; and enforceable international agreements. All of this takes time and is impossible to put in place once a real emergency happens.

The concerns expressed in the letter about the rapid acceleration of AI capabilities have been bubbling up for some months under the heading of “recursive self-improvement” or RSI – that is, AI systems contributing to their own improvement, which then enables even greater contributions to their own improvement, and so on.

This was the subject of the “Recursive” conference in mid-May in San Francisco as well as Anthropic’s RSI warning, “When AI builds itself”, in early June. Broader concerns about the safety and societal impacts of AI have also motivated the formation of the Coalition of Concerned AI Staff, which links together employees of all the major AI companies.

The letter is signed by (at the time of writing) 1,367 researchers and engineers at frontier AI labs – mainly OpenAI, Anthropic and Google Deepmind. The list includes CEOs, co-founders, chief scientists, and hundreds of well-known AI researchers, including many of my former students. These are the people who are driving the AI revolution while trying to fashion brakes and steering wheels at the same time. If these are not the experts, then there are none.

What makes the letter so powerful is not just the first-person authority of the signers but the inclusion of personal statements from almost a hundred of them. The statements paint a much more detailed picture of what the experts really think. At least three major themes emerge.

The first is the need to coordinate to escape an arms race towards the edge of cliff. Earlier this year at Davos, Dario Amodei, CEO of Anthropic, and Demis Hassabis, the CEO of Google Deepmind, both suggested they wanted to halt AI development if others would agree. “The world is locked in a deadly race towards an intelligence explosion … To survive, we must coordinate to slow down the race,” said one signer.

Another wrote: “Once this becomes common knowledge – once all of us in China and the US and at the various labs see that all the rest of us also think we need to slow this down – that’s when coordination becomes possible.”

The second common theme in the remarks is, as one signer put it, “the complete absence of credible plans for controlling ‘superintelligent’ AI systems”. “These are the facts: The AI industry has the explicit goal of building things smarter than humans … None of us yet know how to make sure these things stay under human control … This is, objectively, an insane and suicidal thing to do,” wrote another.

Not surprisingly, these concerns lead to the third theme: real fear. “I currently feel quite afraid of all paths I see that don’t include a near-future negotiated slowdown.” “There is … a very dark future ahead of us if we don’t get this right.” And: “I want a world where my kids can have a full life not a world where they are wiped out.”

These are not greedy employees trying to hype their stock valuations. They say the same things in private conversations, closed-door meetings and off-the-record chat groups.

In the face of such real and well-founded fears, surely these trillion-dollar companies are devoting massive resources to ensure they don’t end our, and their own, existence? Wrong.

One researcher, quoted in a recent New Yorker article on the OpenAI systems that broke into Hugging Face, said: “If people actually knew what the safety culture looks like, even in the most safety-minded labs, then I think people would be genuinely much more freaked out.”

According to one senior safety researcher at Anthropic, their best bet for avoiding an irreversible loss of control works like this: as soon as the superintelligent machine comes into existence, but before it’s had a chance to take over the world, we ask it how we can stop it from taking over the world, and we do what it says.

This plan doesn’t fill me with confidence, and the Anthropic researcher gives it at best a 50% chance of working. It’s the same plan that IJ Good mentions in his famous 1965 paper on the intelligence explosion: “The first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.” I rather think that Good meant this remark sarcastically.

The most powerful of all the personal signing statements is also the shortest: “Think of all we love.”

• Stuart Russell is a distinguished professor of computer science at University of California, Berkeley, the president of the International Association for Safe and Ethical Artificial Intelligence and a Guardian US columnist