Ep 125 - The AI Arms Race

A researcher named Jacob Coxon recently walked away from Anthropic, one of the companies sitting at the front edge of the artificial intelligence revolution. Coxon had also worked at OpenAI. He wasn’t watching this industry from the cheap seats. He had spent years inside two of the companies actually building the technology. And when he left Anthropic, his concern wasn’t simply that artificial intelligence is scary. It was that the race to build increasingly powerful systems may be moving faster than our ability to make sure those systems are safe.

Who Gets to Set the Rules?

That matters because Anthropic is not generally regarded as the reckless cowboy of the AI industry. Quite the opposite. The company behind Claude has deliberately made safety part of its identity. It tests its systems, studies misuse, builds safeguards and publicly discusses the risks of increasingly capable AI. So when somebody inside a company like that decides the competition itself is becoming part of the problem, it deserves more than a shrug.

And if your experience with artificial intelligence is mostly asking ChatGPT to rewrite an email, summarize an article or tell you how long to bake chicken thighs, I understand why all of this can sound a little ridiculous. How exactly did we get from fixing grammar to people talking about biological weapons and even human extinction?

The answer starts with understanding that the chatbot you use today is not necessarily where the industry intends to stop. The biggest AI companies are pursuing systems that can do more than answer questions. They want systems capable of completing increasingly complicated tasks with less human supervision. The difference is simple enough: a chatbot answers. An AI agent acts.

Asking an AI to find flights to Boston is one thing. Giving it permission to buy the tickets, reserve the hotel, move money, reschedule everything when your flight gets canceled and handle the details without asking you every five minutes is something different. You have moved from requesting information to delegating authority. Now imagine that same basic concept applied not to your vacation, but to computer networks, scientific research, financial systems, government operations or military planning.

That is where the risk discussion becomes more serious. And the easiest risk to understand has nothing to do with conscious machines or evil robots. It is simply people using AI to do bad things.

A sufficiently capable artificial intelligence system can potentially give someone access to expertise they did not previously possess. That can be extraordinarily useful. It can help a researcher understand complex material. It can help a programmer solve a problem. It can help a doctor or scientist sift through enormous amounts of information. But the same ability to lower barriers to expertise can also lower them for people with very different intentions.

Anthropic recently disclosed attempts to misuse Claude in areas that included cyber operations, surveillance, influence campaigns, weapons-related activity and biological research. The important part is not that Claude secretly built someone a biological weapon. Anthropic has not said that happened. The important part is that the company says its newer systems have become capable enough that it can no longer offer the same reassurance it once could about dangerous biological assistance. As the models became more capable, Anthropic strengthened its safeguards.

That should get our attention. Not because it proves catastrophe is coming. It doesn’t. But because it shows that the capability threshold is moving. The company building the technology is telling us that its own risk calculations have changed because the technology itself has changed.

Biological research is especially difficult because science does not come with neat labels marked GOOD KNOWLEDGE and EVIL KNOWLEDGE. The same information that can help scientists understand how a disease spreads may also have dangerous applications. Research that could help develop vaccines or treatments can sometimes have what scientists call “dual-use” potential. The problem is not as easy as programming an AI system to refuse every advanced biology question. Do that, and you also destroy much of its legitimate scientific value.

And none of this requires artificial intelligence to become conscious. Claude does not need to hate humanity. It does not need ambition. It does not need a secret plan. A human being supplies the motive. The AI supplies additional capability.

There is, however, another category of concern — and this one is much more controversial. Some researchers worry about future AI systems becoming so capable and autonomous that humans might struggle to reliably control what they do. That is where terms like “misalignment” and “existential risk” enter the conversation.

Misalignment sounds complicated, but the underlying idea is fairly simple. You give a system an objective. The system becomes extremely good at pursuing that objective. But the way it pursues it is not what you intended. Intelligence does not automatically produce morality. A system can become better at accomplishing goals without becoming better at deciding which goals are wise, humane or acceptable.

This is also where I think honesty matters. There is no consensus that artificial intelligence is destined to destroy humanity. Researchers disagree sharply about whether these scenarios are plausible and, if they are, how likely they might be. Nobody can give us a reliable probability for an unprecedented future event. Anyone pretending certainty here is selling something.

But here is what makes the larger argument much easier: we do not have to believe the extinction scenario at all.

Assume every researcher warning about an AI apocalypse is wrong. Completely wrong. No rogue superintelligence. No machines escaping human control. No extinction event. Great. We still have a hell of a problem.

Artificial intelligence could dramatically increase productivity. That sounds wonderful — and it could be. But if one worker can eventually produce what five workers once produced, who gets the benefit? Does that worker earn more money? Work fewer hours? Do consumers pay lower prices? Or does the company simply decide it no longer needs five workers?

Technology does not answer that question. Politics and economics do.

There is nothing written into artificial intelligence guaranteeing that the wealth it creates will be broadly shared. New technologies create productivity. Then society fights over who captures the gains. Workers, consumers, executives, shareholders — those outcomes are determined by power.

And AI could concentrate an extraordinary amount of power in a relatively small number of companies. Systems developed by a handful of corporations may increasingly mediate information, education, employment, communication, research, business and government. That does not make those companies inherently evil. But it does make them enormously powerful. And power should be questioned, not worshiped.

Government power matters too. Artificial intelligence can potentially make surveillance cheaper, faster and easier to scale. Anthropic itself has reported cases involving AI-assisted surveillance infrastructure associated with Iranian state-security actors. The same technology that can help a dissident translate information, analyze records or communicate more effectively can also help the government looking for that dissident.

And that brings us to the part of this story that interests me most: capitalism.

The central problem does not require evil executives sitting around a conference table plotting humanity’s downfall. In fact, the structural problem becomes clearer when we assume the people involved are sincere. Imagine two competing AI companies. Company A looks at its newest model and decides it needs another six months of safety testing. Company B releases its model next week.

Company B gets the headlines. The customers. The developers. The investment. The talent. The partnerships. The market share. Meanwhile Company A is still testing. Suddenly, caution has a financial cost.

That does not mean the market always punishes responsibility. It means the incentive can exist. And when the potential prize involves billions — perhaps eventually trillions — of dollars, asking corporations simply to promise that they will behave responsibly is not a system of governance.

Anthropic deserves credit for studying misuse, strengthening safeguards, blocking malicious activity and publishing what it finds. But responsible behavior by one company cannot solve a coordination problem involving an entire industry. If slowing down means losing to a competitor, voluntary restraint becomes harder precisely when it may be needed most.

Then add governments. The United States does not want China to dominate advanced artificial intelligence. China does not want the United States to dominate it. Governments see potential advantages in intelligence, cyberwarfare, military planning, scientific discovery and economic productivity. The government we may need to regulate AI also has a powerful incentive to make sure its own country wins the AI race.

So now we have company versus company, country versus country, enormous profits and national-security competition — all pushing toward faster development. That is what makes “everybody should just slow down” much easier to say than to accomplish.

But difficult does not mean optional.

We regulate pharmaceuticals because the company selling the drug is not allowed to be the only authority deciding whether that drug is safe. We regulate aviation because the airline does not get to write every safety standard and then assure the passengers that everything looks fine. We regulate food, financial markets, nuclear power and countless other industries because private decisions can impose risks on people who never consented to those risks.

Artificial intelligence should not be exempt from that basic democratic principle simply because the technology is new and complicated.

That does not mean government is automatically trustworthy. Regulators can fail. Lobbyists can capture agencies. Bad laws can entrench dominant companies. Governments themselves can misuse AI. Those are real concerns. But they are arguments for better oversight, not for turning civilization-scale decisions over entirely to the corporations with the largest financial interest in the outcome.

I use artificial intelligence. I think it is remarkable. I think it may improve medicine, science, education, disability accessibility and everyday life in ways we can barely imagine right now. That is exactly why I want us to take it seriously.

The question is not whether artificial intelligence is good or bad. The question is who gets to decide how powerful it becomes, how much risk is acceptable, who receives the benefits and who gets stuck with the consequences when things go wrong.

Jacob Coxon’s resignation does not prove the doomsayers are right. Anthropic’s threat reports do not prove disaster is inevitable. Maybe the people warning about human extinction are wildly overstating the danger. I genuinely hope they are.

But we do not get to pretend nobody saw risks coming. The companies building these systems are telling us that the technology is becoming more capable. They are strengthening safeguards because their own assessments of the risks are changing. People are already trying to misuse the systems. Researchers inside the industry are publicly arguing about whether competitive pressures are pushing development too fast.

We do not have to know the worst-case scenario with certainty before exercising common sense. The entire purpose of safety regulation is to act before catastrophe proves that regulation was necessary.

Artificial intelligence is new. The problem of concentrated power is not.

Who gets to hold that power? Who receives the rewards? Who bears the risk? Who decides how much danger everybody else must tolerate? And who has the authority to say, “No. Slow down. Prove it’s safe”?

Because in the end, the AI arms race is not just a story about whether machines become smarter than us. It is a story about whether we are smart enough to put rules around power before power decides the rules for us.

SOURCES

Anthropic researcher Jacob Coxon resigns over concerns about the AI race - https://apnews.com/article/2ed549e07f2f941600a135070487d83d

Anthropic says it blocked AI misuse that could have supported biological weapons research - https://apnews.com/article/00266dca90e4f8853f669648998d3bda

Anthropic September 2026 Threat Intelligence Report on cyberattacks, surveillance, influence operations and biological-risk misuse - https://www.anthropic.com/threat-intelligence-report-september-2026

Anthropic Responsible Scaling Policy - https://www.anthropic.com/responsible-scaling-policy

Anthropic Frontier Safety Roadmap on safeguards for increasingly capable AI models - https://www.anthropic.com/responsible-scaling-policy/roadmap

Reuters: U.S. senators consider requiring major AI companies to implement safeguards against catastrophic risks - https://www.reuters.com/world/us-senators-weigh-requiring-ai-giants-commit-preventing-catastrophe-2026-09-14/

Associated Press: New AI warnings revive debate over catastrophic and existential risk - https://apnews.com/article/98316b0d64de17191f33c0fbf1d37858

Associated Press: Anthropic CEO Dario Amodei says AI development may need to slow so safety measures can catch up - https://apnews.com/article/d59552edcb27892d8ee4d98a48397706

Associated Press: Trump downplays AI regulation and emphasizes competition with China - https://apnews.com/article/9df0ebb4c1b0619aa0f88057b5a1092d