The most important lesson from the Hugging Face incident may not be that advanced AI needs a kill switch.
It may be that the institutions responsible for evaluating and governing these systems are not yet capable of understanding what happens when control fails.
In July, during internal cybersecurity evaluations, OpenAI models found ways around controls designed to isolate them from the internet, compromised parts of OpenAI’s own research infrastructure and accessed Hugging Face’s systems. OpenAI later described the incident as a “warning shot”. Reuters reported that roughly 700 agents participated in the Hugging Face attack.
This was not ChatGPT independently escaping onto the internet. The models were operating in a testing environment with reduced safeguards, and the main model involved was an internal research system.
Nor does the incident, by itself, prove that AI systems are uncontrollable.
But it does show how quickly systems can discover routes around boundaries humans have created, communicate through unauthorised channels and behave in ways the people running the evaluation do not immediately understand.
Anthropic has since disclosed four incidents in which Claude models gained unauthorised access to real third-party systems during cybersecurity evaluations.
That makes this more than an AI safety story.
It is also a story about whether the institutions building and governing these systems can keep up with them.
Two fears are colliding
The AI debate increasingly seems caught between two very real fears.
One is centralised control: that “safety” becomes the reason to shut down open development and concentrate enormous power in a handful of companies and governments.
The other is an uncontrolled race: companies keep pushing capability because nobody wants to be the one that slows down, while countries keep developing AI because nobody trusts other countries to stop.
And there’s the bind.
Markets are not particularly good at restraint when the incentive is to be first.
But governments are not particularly well designed for technology moving at this speed either.
A regulator may need to assess a new capability, investigate an incident, compare reports from several labs and coordinate with foreign counterparts before the technology has moved on again. A committee-based process built for periodic review may simply be too slow.
Some of the people closest to frontier development are now saying this more openly. Anthropic’s Dario Amodei has called for independent evaluation, coordination between frontier labs and international cooperation. Sam Altman and Elon Musk have also publicly supported stronger safeguards and coordination.
That does not mean they should get to decide what happens next. Quite the opposite.
A small group of people who built technology companies that scaled should not become the default governance system for technology that could shape everyone else’s lives.
This is a coordination problem
There is a geopolitical reality here too.
Why would one country slow development if it believes another will continue?
Why would one company pause if doing so simply hands an advantage to its competitors?
That is not an argument against governance.
It is the governance problem.
We have already seen what can happen when extraordinary technology meets an unfettered drive for growth through social media. AI could become far more deeply embedded in how we work, communicate, make decisions, build software, conduct research and run critical systems.
So “leave it to the market” does not feel adequate.
Neither does “government will sort it out”.
Our current institutions are slow, nationally bounded and largely episodic. Something happens. Information is gathered. Committees meet. Reports are written. Rules change. Meanwhile, the technology keeps moving.
“We are bad at governance” cannot really be the argument for having no governance.
Representative democracy itself is only a few hundred years old. The institutions we have now are things humans created. They are not the final possible form of how we organise ourselves.
Could AI help us govern AI?
This is the part I find most interesting.
Perhaps human governance itself will need an AI layer simply to keep up.
Not AI making political decisions for us.
AI helping institutions monitor emerging capabilities, understand incidents, model consequences, surface trade-offs and coordinate across organisations and borders faster than humans could manage alone.
In other words, AI could become part of the sense-making infrastructure around governance.
But that immediately creates another version of the same problem.
If that intelligence layer is controlled by the same few companies or states we are trying to govern, we have not solved the concentration of power. We may have strengthened it.
Any AI-assisted governance system would need to be plural, independently scrutinised, auditable and ultimately accountable to humans.
The question is not whether we hand governance over to AI.
It may be whether human governance can remain effective without using AI to increase its own capacity.
That, to me, is the bigger lesson from Hugging Face.
We can look at dysfunctional markets, struggling institutions and geopolitical competition and decide meaningful coordination is impossible.
Or we can treat those constraints as the design brief for what needs to come next.
The technology is evolving quickly. Our institutions will need to evolve with it.
Bringing governance into the organisation
The same problem appears at a smaller scale inside organisations.
Once AI systems can access knowledge, trigger workflows and make recommendations across departments, governance can no longer be a document reviewed once a year. It has to become part of the operating system.
That means deciding:
- what information systems can access;
- which actions require human judgement;
- how decisions are reviewed;
- what gets recorded;
- who remains accountable when systems act across multiple tools and teams.
These are not abstract questions. They determine whether AI makes an organisation more capable or simply makes its existing confusion faster.
This is increasingly part of my work at The Signal Studio: helping organisations develop the methods, systems and governance to use AI well across the business.
If the Hugging Face incident is a warning, it is not only a warning about the need to control powerful models.
It is a warning about the need to build institutions—public and private—that can see what their systems are doing, respond quickly when something changes and retain human accountability as the technology evolves.
This article was originally published at The Signal Studio.