Pacing the Frontier Is Not Enough

If powerful AI is dangerous when concentrated in the “wrong hands,” we should also ask what makes any small group the right hands for humanity.

Dario Amodei’s recent essay, We Must Pace the Frontier, addresses a problem I take seriously.

Frontier AI capabilities are developing extremely quickly. Amodei argues that alignment, interpretability, evaluations, operational safety and governance need time to catch up, particularly as AI systems become increasingly capable of contributing to the development of future AI systems. He also points to recent agentic failures as evidence that relatively limited failures today could become far more consequential as capabilities increase.

I do not think those concerns should be dismissed.

Nor am I opposed to regulation simply because it is regulation. Some of Amodei’s proposals seem genuinely worth exploring. In particular, his proposal for embedded independent evaluators — external teams with ongoing access to frontier AI companies, the ability to inspect safety practices and incidents, and some capacity to report findings independently — addresses a real problem: companies should not be the sole arbiters of whether their own systems are safe.

My concern lies somewhere else.

It is possible to improve the technical safety of AI while leaving untouched the distribution of authority surrounding it.

And if we fail to distinguish those two problems, we may build increasingly sophisticated systems for AI safety while quietly concentrating control over AI itself.

Which “we”?

One word appears repeatedly in discussions about the future of AI:

We.

We need to slow down.

We need to regulate.

We need to preserve democratic control.

We need to protect humanity.

But “we” is rarely as universal as it sounds.

In Amodei’s essay, the practical decision-making actors are primarily frontier AI companies, third-party evaluators, the United States government, allied democratic governments and, eventually, other major states such as China. At the same time, he argues that society should have a say in how AI is used and ultimately frames the task as something owed to humanity.

Those categories are not interchangeable.

I live in Adelaide, Australia.

I am one of the humans contained inside the word humanity. Decisions made by frontier laboratories in California, governments in Washington or Beijing, and future international regulatory bodies may materially affect my life and the lives of my children.

But being included in the consequences does not mean I am included in the operational we making the decisions.

This is not a criticism unique to Anthropic. It is a structural problem that appears throughout large institutions.

People can be consulted without becoming co-authors.

An institution can conduct extensive stakeholder consultation, invite submissions, hold public meetings, publish reports and genuinely listen to some of what it hears — while retaining almost complete control over the underlying decision architecture.

It can decide:

what the problem is;

which options are considered realistic;

what counts as relevant evidence;

which trade-offs are acceptable;

and who ultimately decides.

In a metaphor I use elsewhere in my work, the public may be invited to hand the institution pieces of paper while the institution keeps hold of the clipboard.

That is participation.

It is not necessarily shared authorship.

What exactly are the “wrong hands”?

This becomes particularly important when AI safety merges with geopolitical competition.

Amodei argues that democratic states should preserve their AI lead over authoritarian states, particularly China. His proposals include restrictions on advanced chips and semiconductor equipment, action against model distillation, stronger security around model weights and other measures intended to slow Chinese AI development relative to the United States. He explicitly argues that such measures could widen the American lead.

There may be legitimate national-security reasons for some of these measures.

But I think we need to distinguish at least three meanings of the phrase “wrong hands.”

There are malicious hands: actors deliberately intending to cause harm.

There are unsafe hands: actors who may lack sufficient competence, safeguards, caution or control.

And there are rival hands: actors whose possession of a capability changes the strategic distribution of power.

Those are not the same problem.

Yet the rhetoric of safety can allow them to blur together.

“This technology could enable catastrophic harm” is a safety argument.

“This technology must remain more advanced in our hands than theirs” is a strategic-power argument.

Both may sometimes be relevant. But one does not automatically establish the other.

And this creates an uncomfortable symmetry problem.

If concentrated frontier AI capability is frightening because China might control it, then ordinary people are entitled to ask why equivalent concentration inside US governments or a small number of American corporations should automatically feel reassuring.

China and the United States are not politically identical.

That is not the point.

The question is whether control by our preferred actor is quietly being treated as equivalent to safety.

Even Amodei recognises that China may fear US strategic dominance just as the United States fears Chinese dominance. His global coordination proposals are built around the difficulty that either side may defect from an agreement and gain a decisive advantage.

So the symmetry is already visible.

The proposed answer, however, remains asymmetrical: coordinate globally where possible, while preserving the lead of the United States and its allies.

This may be understandable from inside the present geopolitical system.

But it does not escape that system.

Slowing the race without leaving the race

This is perhaps the central tension I see in Amodei’s proposal.

He identifies a genuine danger created by race dynamics:

competition → faster capability development → inadequate time for safety → increased risk.

He therefore wants to slow the race.

But he also believes that slowing enough to lose the race to China would itself be dangerous.

The resulting position is effectively:

We should slow down, but not enough to lose our relative position.

Indeed, part of the strategy is to slow competitors more than domestic frontier developers.

Again, this may make sense from within the current geopolitical arrangement.

But the competitive relationship that helped generate the problem remains largely intact.

This matters because regulatory structures do not remain isolated from the systems in which they operate.

A safety threshold can become a licensing threshold.

A licensing threshold can favour institutions wealthy enough to comply.

Compute controls can become barriers to entry.

A designation of “competence” can gradually become a designation of which institutions are permitted to develop advanced intelligence at all.

A framework introduced to prevent catastrophic misuse can therefore simultaneously become infrastructure for institutional concentration.

That does not mean this outcome is inevitable.

It means we should design for the possibility from the beginning.

Safe AI can still reproduce an unsafe distribution of authorship

This is where I think the AI safety conversation needs to widen.

Imagine that frontier AI becomes extraordinarily safe in the conventional technical sense.

Models become reliable.

They are interpretable.

They resist dangerous instructions.

Agentic systems remain controllable.

Cybersecurity and biological risks are well managed.

Alignment works extremely well.

That would be an enormous achievement.

But then ask:

Who owns those systems?

Who gets access to the most capable versions?

Who controls the compute?

Who decides which applications are permitted?

Who defines competence?

Who gets to build new systems?

Whose preferences become defaults?

Who has enough resources to shape the next generation?

Who receives highly tailored tools, and who receives the standardised version distributed downward?

Technical alignment does not answer those questions.

A perfectly safe AI can still become another technology distributed through the existing hierarchy.

Those already possessing greater money, institutional influence, compute, political authority and technological access may receive greater choice and greater ability to shape what comes next.

Others may technically have “access to AI” while having very little authorship over the systems governing that access.

In that world, AI may be safe while amplifying an unsafe distribution of authorship.

Auralis asks a different question

I have been developing a framework called Auralis around human authorship, relational systems, AI and the conditions that allow differentiated people to participate meaningfully in their own lives.

Auralis does not begin by asking:

Which institution should be trusted to control increasingly powerful intelligence for everyone else?

It asks something closer to:

Can we build the relationships so that no set of hands can easily become everyone else’s hands in the first place?

That leads to a different way of thinking about competence and authority.

Competence should, wherever possible, be specific to the capability involved, rather than a general designation of an “approved” person or institution.

The criteria should be transparent.

They should be contestable.

They should be revisable.

Restrictions should be proportionate to the actual risk.

And authority should remain as close as reasonably possible to the people, places and consequences affected by the decision.

That is very different from deciding that a small number of institutions are the permanent competent custodians of intelligence for humanity.

It also leads to another principle:

Capability portability does not have to imply authority portability.

A useful AI model, method or discovery could originate in China, the United States, Australia or somewhere else entirely.

Other people should potentially be able to use it without thereby importing the political or institutional authority of whoever created it.

The tool may travel.

The knowledge may travel.

Some safety standards may need to travel.

The authority of the original creator does not automatically need to travel with them.

In simple terms:

the clipboard does not have to come with the tool.

This is not an argument for no regulation

Distributed authority does not mean unrestricted access to every dangerous capability.

Some technologies create risks that genuinely cross local boundaries.

An individual decision involving biological weapons, autonomous cyberattack infrastructure or other catastrophic capabilities cannot reasonably be treated as purely private simply because we value personal sovereignty.

Shared limits may sometimes be necessary.

But there is an important difference between saying:

“This particular capability creates sufficient cross-boundary risk that additional competence and safeguards are required”

and saying:

“These institutions are competent custodians of advanced intelligence generally.”

The first is bounded.

The second can become a hierarchy.

Auralis is interested in maintaining that distinction.

Worth, authorship and the giant clipboard

Underneath this is a more human problem.

Much of our current social structure still distributes security, authority, resources and recognition hierarchically.

Those at the top often become positioned as though they are acting for everyone else.

They may sincerely believe that they are.

But there is a structural danger whenever one person or institution begins to derive authority from representing a population that did not meaningfully author that representation.

“Humanity” can become the moral ground beneath a very small number of decision-makers.

The larger the constituency invoked — the public, democracy, humanity — the larger the authority of the person claiming to act on its behalf can appear.

Auralis begins somewhere else.

Human Worth does not need to be granted by position.

Human authorship should not need to be surrendered upward in exchange for safety.

If people begin with secure Worth and retain meaningful authorship, then institutions no longer need to become the sole location where intelligence, judgment and legitimate action reside.

We can coordinate without assuming coordination requires a single ultimate controller.

The deeper question

I think Amodei is right about something important.

The frontier may need to slow down.

External evaluation may be valuable.

International agreements around genuinely catastrophic capabilities may be necessary.

Some restrictions may be entirely justified.

But I am wary of solving the danger of concentrated AI power by concluding that AI power must therefore be concentrated in the right hands.

That may reduce one class of danger while leaving the deeper architecture untouched.

The question is not only:

Does the AI behave safely?

It is also:

Safely on behalf of whom?

Under whose authority?

According to whose definition of the desirable future?

And how do the humans affected remain meaningful authors of their own lives?

Perhaps the most important AI-governance question is therefore not:

Who should control AI for humanity?

but:

How do we build an AI ecology in which humanity does not require a single controller at all?

That is the question I want Auralis to keep asking.

About Auralis

Auralis is a developing framework concerned with human Worth, authorship, relational systems, distributed sovereignty, AI and the conditions that allow differentiated humans and communities to remain meaningful participants in the systems affecting their lives.

I named Auralis and first documented it under that name on 13 June 2025, bringing together patterns and questions I had been exploring for much longer.

Preserving that development trail — including where ideas originated, how they changed, and who contributed to their development — is itself part of the work.

Authorship and drafting note

Concepts, argument and Auralis framework: Elizabeth Brown
Formal drafting and language development: ChatGPT / OpenAI, working from Elizabeth Brown’s concepts, distinctions and discussion
Final selection, editing and publication authority: Elizabeth Brown