The other existential risk for market research – NewMR

Ray Poyner, 12 September 2026

---

Market research is debating whether AI will replace researchers, research agencies, panels, and much of the work people currently do. This is an important question. Jobs, businesses, professional identity, and the quality of insight all matter. However, another existential-risk question is more fundamental: could advanced AI threaten human life on Earth?

This question came sharply into focus this week when Jacob Coxon resigned from Anthropic, the company behind the LLM Claude. Coxon had previously worked at OpenAI before Anthropic. He said leading AI companies are racing toward self-improving superintelligence and are ‘gambling with our lives’. His concern is not today’s chatbots. It is the possibility that systems become much more capable, can improve their own successors, acquire resources and evade human control. See my blog ‘What is so worrying about recursive self-improvement’ for a comment on one of the key underlying technologies.

He is far from alone. Nobel Prize-winning AI pioneer Geoffrey Hinton has repeatedly warned that AI could become more intelligent than people and take control away from us. Eliezer Yudkowsky and Nate Soares make the most forceful version of the case in their book If Anyone Builds It, Everyone Dies. Their title makes their conclusion clear: they believe that building superintelligence would be an existential mistake.

The proposition
The core argument is reasonably straightforward, even if the technical details are difficult.

First, capability could advance faster than society expects. A system that is better than people at science, coding, cyber-security, persuasion and strategic planning could help create an even better system. If this produced a fast cycle of improvement, human institutions might struggle to keep up. This point is not considered controversial by most people well-versed in the field.

Second, we do not yet know how to guarantee that a highly capable AI will reliably pursue human goals. We can make current systems helpful and constrained, although there have already been some slipups, such as the OpenAI hack of the platform Hugging Face. But controlling today’s AI is very different from proving that a much more capable system will remain safe when the systems become more powerful and harder to monitor and constrain.

Third, a powerful system does not need to be conscious, malicious, or ‘want’ to kill people in the human sense. The concern is that a system pursuing a poorly specified objective could seek power, money, computing resources, or influence because these help it achieve its goal. AI is basically a goal-seeking tool. People could become an obstacle or simply cease to matter.

Fourth, there is a race. Companies fear losing to competitors. Countries fear losing strategic advantage. The incentives reward speed, secrecy and optimism. Safety work can be seen as a cost or a delay.

There are also nearer-term dangers that do not depend on superintelligence. For example, large-scale cybercrime, biological misuse, autonomous weapons, surveillance, fraud and political manipulation. Anthropic’s recent threat report describes attempts to misuse AI in several of these areas. These risks are already serious, even if the extinction scenario never happens.

The counter-views
The argument for extinction is far from settled. There are plenty of people on the other side of the discussion.
One objection is that it rests on a chain of uncertain assumptions. For example, it assumes that AI progress will continue rapidly, that self-improvement will occur, that it will lead to superhuman general capability, that control will fail, and that failure will produce extinction. Each step is uncertain. Combining them does not produce a reliable date or probability.

A second objection is that intelligence does not automatically produce a drive for power. Current AI systems are limited, unreliable and dependent on human infrastructure. Some researchers argue that the path from large language models to an uncontrollable superintelligence has not been demonstrated.

A third objection is about priorities. AI is already affecting employment, intellectual property, education, discrimination, misinformation, energy use and concentration of power. A focus on a distant, uncertain catastrophe in the future could distract attention from harms that are happening now.

A fourth is practical. A global pause may sound sensible, but how would it work? If responsible companies and democratic countries slow down while others continue in secret, the result could be worse. A pause without verification and broad international agreement could simply change who gets there first.

These are substantial objections. They should be part of the debate. Dismissing the existential-risk argument as science fiction is not a sufficient response. Treating it as certain is not a sufficient response either.

My view
I do not think most of us are competent to put a statistical probability on human extinction from AI. I have a degree in computer science and a Master’s in Research, yet I certainly am not competent to estimate that probability. If you are reading this article, you also probably do not have enough information to take a position on who is right and who is wrong.

When people with deep technical knowledge of advanced AI say that the risk of human life on Earth being ended may be substantial, we should take the concern seriously. That does not mean accepting every forecast or every timeline. It means recognising that a low-probability event with an unimaginably severe outcome deserves careful attention.
There is a further complication. Most AI researchers are not specialists in forecasting rare, complex events. Forecasting is difficult. People are often overconfident about the future, particularly in fast-moving fields where technical progress, commercial incentives, politics and international security all interact. The debate needs AI experts, but it also needs people with expertise in forecasting, risk analysis, biosecurity, cyber-security, economics, law, diplomacy and governance.
In an ideal world, humanity would pause the development of systems beyond an agreed capability threshold. We could then work out how to use AI well, how to regulate it and how to demonstrate that highly capable systems can be controlled.

But such a pause would need the United States, China and other major powers to agree. It would also need the richest technology companies and their backers to agree. If only some countries and companies pause, others may seek the prize of superintelligence. For example, if the rest of the world voluntarily paused, I would still expect Elon Mushk to continue. And, with the rest of the world pausing AI, there would be plenty of experts willing to join his company. The danger then comes from the AI itself AND from the people or states that control it.

That is why governments need to act, and why they need to act together. National regulation is necessary, but it is insufficient on its own. We need international agreements that include meaningful verification, shared safety standards, independent evaluation of frontier systems, reporting of serious incidents, controls on highly autonomous AI systems and consequences for those who ignore the rules.

This is difficult. Nuclear arms control was difficult. Biological weapons control is difficult. Climate agreements are difficult. But, difficulty does not remove the need to try.
For market research, this is a reminder to keep the discussion in proportion. We should examine whether AI will change or replace parts of our work. We should prepare people and businesses for that change. We should also contribute our skills to the wider debate. We need to be asking clear questions, distinguishing evidence from assertion, understanding public attitudes and helping societies make informed choices. We should also be helping policy leaders understand what the public knows, wants, and will accept.

Further reading

 

Never miss a NewMR update.
Get new blog posts, free webinars, and other NewMR news by email. Sign up to our mailings →

Related