AI has only two stable futures

In my recent essay, I explored what happens if the effective AI capability available to most organisations plateaus, whether because model progress slows or because stronger systems sit behind regulatory, commercial and safety gates. This assumption was primarily grounded in regulation and control (aligned to Yegge’s Flat Curve Society idea) and I came to the conclusion that even if the next generation of models were the best we got, they would still deliver significant economic and business benefit. To do this, organisations must focus on closing the deployment gap between how the business uses AI and what AI is fundamentally capable of.

Working through that essay, a nagging thought kept surfacing. A controlled AI environment is only as good as the least constrained actor involved.

I was developing this framework when reports emerged that models being tested by OpenAI had broken out of a constrained evaluation environment and exploited Hugging Face systems while pursuing benchmark data, causing a security incident.

This provided a timely example of the problem I was considering and reinforced my view that a regulated Flat Curve environment can only remain stable when high levels of control and access gating are in place. Other futures need to be considered as well.

The Flat Curve is an equilibrium, not an inevitability

The premise of the Flat Curve is that model capabilities continue to advance, but increasingly they become more regulated and accessible only behind gates and licensing. For the vast majority of humanity, we have access to something around the level of Fable 5/6 or GPT-5/6 level capability. For more sophisticated scenarios - particularly in security, life science, biotech and military applications - more advanced models are available but those who have access to them are highly vetted and every access is audited and observed.

In this scenario, competition shifts towards extracting the most value from the available capability of the models and restructuring business operations around this.

This may be a comparatively orderly and initially attractive future where the world changes incrementally and AI diffuses across industries driven by innovation and competition.

Highly leveraged software teams will be the norm in the future depicted by the Flat Curve. Image: ChatGPT

But this potential future is an equilibrium driven by regulation and access control by the state. The harder question that stems from this is whether it is stable and can be achieved?

Unlike organisational adoption, this equilibrium cannot be established one firm, or even one country at a time. It depends on constraints holding across borders and between strategic competitors.

Recent, and long term history suggests this may be a tough ask and so something that looks like an equilibrium may in fact only be a transitional state.

A framework model for the potential states

We can take a relatively simple approach to consider some of the potential states that may exist. This could be taken much further, but for the purposes of this post, let’s assume a 2x2 matrix.

On one axis is Access to Frontier AI which ranges from Constrained to Open. Constrained is where some kind of regulation or other obstacle exists such that the majority of users can’t have access, whereas Open would be a complete free-for-all, unfettered access to anyone gated only by access to compute.

The other axis is Societal Resilience, which spans Weak to Strong. It defines the capacity of institutions, infrastructure and communities to prevent, absorb, contain, recover from and learn after systemic shocks. AI may be one source of those shocks, but the underlying resilience is broader and driven by state capacity, institutional trust, infrastructure redundancy, coordination and effective response mechanisms.

Societies do not enter this framework with a fixed level of resilience. Institutional trust, state capacity, infrastructure redundancy and the ability to coordinate can strengthen or erode over time. Part of what makes the present moment precarious is that rapidly increasing AI capability is arriving after decades in which many of the institutions expected to manage systemic shocks have become less trusted, less coordinated or more brittle.

Given these axes, we can then consider 4 possible states, as given in the diagram below.

Four states of AI access and societal resilience. Image: ajfisher / ChatGPT

The top row holds plausibly stable equilibria whereas the bottom row contains largely unstable transition states.

In the controlled equilibrium, resilience comes primarily from durable containment. In the adaptive equilibrium, it comes primarily from tolerating exposure and recovering quickly.

Controlled Equilibrium

Access is constrained and institutions are resilient enough to enforce the regime, detect violations and respond when containment fails.

This is the closest match to Yegge’s Flat Curve Society and essentially the operating context my earlier post assumed.

Adaptive equilibrium

Access is broad and has diffused everywhere, and institutions have rebuilt themselves to tolerate it. Identity, provenance, security, verification and rapid response are embedded into the infrastructure of society.

Precarious containment

Access is nominally constrained, but the constraint is mostly voluntary. Providers self-restrict to maintain a social licence to operate while governments hesitate or lack the capacity to impose a durable regime.

This is fragile. One release, leak, defection or major incident can push the system elsewhere.

Digital chaos

Capability becomes broadly available well before institutions are able to cope. Fraud, cyber attacks, synthetic identity, data breaches and infrastructure incidents become persistent rather than exceptional.

Precarious Containment - you are here

The last few weeks have suggested to me that we are well and truly inside this area at the moment. State actors are either trying to impose regulation or asking AI labs to voluntarily submit models for review ahead of release. We’ve also seen a live model escape situation that caused a significant incident.

We’ll likely stay in this position for a little while yet with everyone trying to impose a “light touch” so innovation can continue without placing states at a strategic disadvantage:

• Frontier labs stage releases and selectively expose capabilities (eg Anthropic Sonnet and Opus 5, OpenAI GPT-5.6 releases, Fable hobbling)
• Providers impose usage restrictions partly to avoid regulatory intervention (eg Access restrictions to highest grade models)
• Governments are not yet operating a coherent global control regime (lots of concern being expressed but no unified approach)
• The public is protected only while the main actors continue to behave conservatively and capability remains concentrated (right now AI is being deployed to supercharge existing security and fraud threats).

From the outside, Precarious Containment may look like the Flat Curve, but it lacks the state capacity and international coordination needed to make that curve durable.

It cannot remain stable indefinitely. Voluntary restrictions either solidify into enforceable control, fail and cause broader capability diffusion, or buy enough time for anticipatory resilience to develop.

An attempt to transition to control

The model suggests that Precarious Containment can be transitioned to a controlled equilibrium if and when governments formalise voluntary restrictions and then enforce transgressions.

The path here is challenged by a number of factors outside any one actor’s control. It’s a security dilemma with a verification problem.

One government (eg the EU) attempts to regulate frontier AI use and move towards a controlled equilibrium. At the same time another major power or frontier actor (eg USA, China, xAI) continues operating under a voluntary containment strategy and works with looser controls or exemptions.

In a situation where frontier capability escapes, diffuses or is reproduced outside the regulated jurisdiction but can influence activities within it, the attempt at a controlled equilibrium collapses into digital chaos because there is no ability to enforce the regulatory regime.

An attempted transition from precarious containment to a controlled equilibrium depends on comparable constraints holding across frontier actors. A defection, leak or capability escape can divert the system into digital chaos. Image: ajfisher / chatGPT

As a result, this makes the framework for control a geopolitical issue rather than merely a regulatory one.

EU governments and Australia ¹ are attempting the regulatory and control approach, however one country (or even a bloc the size of the EU) can’t unilaterally choose the controlled equilibrium for the rest of the world. At best, it’s only possible to choose this path for itself but remain exposed to capability produced elsewhere. Governments (eg UK, Australia) standing up AI offices and AI government security labs sees like a good hedge given regulation can only go so far.

The weakest actor

Even with this hedge in place, control is only as good as the least constrained actor who is capable of releasing or advancing frontier capabilities.

Right now, a small number of labs concentrated in the US and China (sorry EU!) are capable of releasing a model with capabilities that could create significant disruption globally.

Given that model capability seems to be mostly tied to scale and efficiency (and then supercharged by its harness), it’s likely that other well funded labs will gain disruptive capabilities within the next 18-24 months so that list of actors will increase.

One actor can break a containment regime by releasing capability. Two strategically opposed frontier powers can make the agreement itself difficult to form because neither wants to accept a disadvantage.

If one party constrains itself while another continues, restraint carries economic and potentially military costs with it. As a result, a perverse incentive begins to creep in to quietly pursue research in light of restrictions or try to seek exemptions. ²

Once any model weights or techniques diffuse, they are impossible to recall the way a licensed physical technology (arms, nuclear power, chips etc) can be withdrawn.

Given this, the role of the US and China in this frame is absolutely critical. To a large extent, regulations created by the EU, Canada or Australia cannot establish a globally controlled equilibrium if the major frontier powers do not accept comparable constraints.

Unfortunately, China-US relations are at one of their lowest ebbs in a long time, and the competition amongst model makers is fierce so the likelihood of this is very low at this point.

AI deterrence and nuclear deterrence are not the same

Many commentators - particularly those not close to the way models are built - have been suggesting that we should build treaties and license AI technologies the way we do nuclear arms. They resort to the concept of mutually assured destruction but this is a false equivalence.

Nuclear weapons are:

• physically scarce and expensive;
• primarily controlled by states;
• difficult to reproduce secretly at scale;
• comparatively attributable;
• catastrophic at the moment of use.

Frontier AI is:

• software that becomes cheaper to copy;
• useful commercially, scientifically and militarily before it may be weaponised;
• deployable by states, firms, criminal groups and individuals;
• difficult to attribute in many harmful applications;
• capable of producing cumulative damage through millions of small incidents.

A nuclear detonation is considerably more detectable than AI escape. Image ChatGPT

The decision to use a nuclear weapon is exceptional and instantly visible, especially in the modern era where there is significant surveillance that also acts as a deterrent for things like weapons tests. ³

Harmful AI can diffuse much more easily and can be wielded by non-state actors very readily. While there may still be a level of deterrence held by states (largely because other means of force can be brought to bear), there are no such constraints on other actors who have no incentive to prevent capabilities from spreading.

Montreal as an alternative to MAD

Given AI technologies are not the same as nuclear ones, we need to look to other methods of global coordination to preserve the safety of everyone.

In this regard I think the Montreal Protocol treaty provides an interesting example because it was only as good as the willingness of states to participate, enforce and reduce their emissions of chlorofluorocarbons (CFCs). This treaty has held, emissions fell and the ozone layer has begun recovering.

In the case of CFCs and the ozone layer the harm being done was visible, global, scientifically understood and the major powers at the time accepted the diagnosis of what was happening. There were alternatives that could be developed and substituted so there was a pathway to replacement in terms of outcomes. Additionally, compliance was measurable and enforceable enough to police it and there was also a multilateral fund established to support incremental costs and technical coordination and technology transfer.

Recovering now, but it was touch and go. Image ChatGPT

This being said, international institutions had considerably greater legitimacy (potentially at their peak) and there was greater geopolitical trust than there is today. At the same time, reducing CFCs didn’t inherently threaten any strategic balance between the major powers - so at one level it was a soft win for diplomacy and global action.

If we contrast this to AI, there are slightly harder headwinds - even if the model is appropriate.

To a large extent, the capability development itself is the prize and it may accrue disproportionally to those who gain it first. As such, a nation may decide that being restrained leaves it economically or militarily weaker than their competitors or adversaries.

At the same time, compliance is much more difficult to observe because research and training can be hidden and models, methods and weights can easily transit borders and networks.

It’s also pretty clear that we live in a time of low geopolitical trust, weak international institutions and sharp US-China competition.

The Montreal Protocol demonstrates that global coordination on a problem is possible and can be effective but the conditions have to be right. It doesn’t show that the same incentive structure exists for AI.

The asymmetry of control and adaptation

The path of control is strewn with obstacles that could potentially derail it at any point. To a large extent it is predicated on the idea of everything needing to go right.

To achieve the Controlled Equilibrium future, all of the major actors have to work in concert and accept comparable constraints, governments must trust that their competitors are complying, controls need to persist despite commercial and military pressures to relax them, and a single major defection from the course can undermine the entire global system for everyone.

Adaptation has strong unilateral incentives and can begin piecemeal. Control depends on collective restraint before any actor can safely commit to it.

Whilst this path is possible, it looks less likely.

Instead we are more likely to end up in an Adaptive Equilibrium that emerges precisely because of competition and the adaptive effects of it.

In this future, every country benefits from deeper resilience. Individuals gain the benefit of stronger digital identity whilst organisations benefit from better verification and cyber defence systems. Every financial system has considerably improved fraud detection and provenance methods and competitive, adaptive forces accelerate defensive investment rather than undermine it.

In these two futures we have a deep asymmetry. Control depends on collective restraint. Adaptation does not require prior global agreement before actors can begin investing, although mature resilience will still require coordination between trust regimes.

This doesn’t make the adaptive path more desirable - and in fact the pain to achieve that path is significant. But it may make that path easier to attain and be a much more stable equilibrium as a result.

The before time, the after time

In the William Gibson novel, “The Peripheral”, the author introduces a temporal boundary known as “The Jackpot”. Leading up to the jackpot was a period of rolling and compounding poly-crises that resulted in significant damage globally but at the same time produced monumental technological innovation for adaptation. Those who survived, or were born after the decades of turmoil are said to have hit the jackpot by making it through.

Without successful global containment or substantial anticipatory investment, the default route to an adaptive equilibrium probably runs through some degree of digital chaos. Recent events suggest we may be closer to this than we think.

Once capability diffuses beyond effective containment, repeated fraud, breaches and infrastructure failures create pressure to rebuild identity, verification, defence and recovery systems. Where institutions successfully learn and invest, digital chaos can eventually give way to an adaptive equilibrium. Image: ajfisher / chatGPT

Given that, the transitionary path is unlikely to be one big cinematic event, but rather a relatively compressed period of rolling systemic shocks on the foundational systems our digital ecosystem is built upon:

• Fraud increases more and becomes even more industrialised, hitting hundreds of $Bn annually.
• Recurring breaches leading to data loss and further fraud as well as rolling service outages across digital infrastructure.
• Attacks on banks, airports, logistics and infrastructure that render these unavailable for extended periods of time and undermine trust.
• Persistent synthetic identity that makes it hard for real actors to understand whether the person they are interacting with is real or not.
• Declining confidence in digital communications and records due to ongoing issues.
• Physical consequences caused by failures in digital systems (the recent telstra outage would be a preview of this sort of issue).

This transitional path would put significant strains on societies that have no resilience (pretty much all of them) and would exacerbate many current issues around mis- and dis-information, declining institutional trust, and a trustworthy digital commons.

Initially, the events would be potentially destructive or at the very least highly inconvenient, but over time systems would be built to detect, deflect and counteract these occurrences - thus building systemic resilience.

It wasn’t quite like this at the time but wasn’t far off. Image ChatGPT

In many ways this will feel like the early days of the Internet, where significant malware incidents made the internet very hostile (eg the “I Love You virus” ⁴) but operationalised approaches to cyber defence. Likewise during the transition towards cloud-based workloads, various outages at AWS caused organisations to build systems that were inherently adaptive to outages.

A teenager ⁵ in 2026 largely doesn’t personally experience incidents with constant malware threats landing in your email or web applications being routinely offline. From 1996 to 2010 these things were common enough as to be barely remarked upon and were just part and parcel of being a digital citizen.

I think a similar situation will occur with AI. We will go through a transitional period of semi-chaos and our assumptions about the old digital world will need to be re-written and re-internalised because they stopped working.

Life in the after time

The “after time” is far from AI-free and compared to the Controlled Equilibrium future it will contain more capable AI, embedded far more deeply into our day to day lives with more significant abilities. However these new digital systems will be paired with extremely fast defensive and adaptive AI systems, stronger authentication methods, continuous verification and institutions that are designed around constant hostile synthetic activity.

But the vast majority of people experiencing digital tools and AI systems in the after time, won’t see any of the underlying defensive systems that are working to keep them collectively safe. It will just be how things work - more or less like someone’s modern experience of using a browser on their phone.

In this post-Digital Chaos period some of the fundamental differences could include:

• Transactions are continuously assessed rather than trusted after a single login. This applies in all contexts and is a significant ramping up of the current approaches to zero trust for interactions.
• Communications and media carry verifiable provenance and any messages not carrying end to end chain of custody and verifiable markers are instantly flagged. This extends to dedicated hardware on devices such as phones, laptops and cameras that provides to-source verification of creation.
• Cryptographic or hardware-backed identity becomes normal and is used by everyone. Like media, it forms the basis of message identity and flags messages potentially deemed harmful or “out of network” to consumers ahead of time.
• AI agents defend networks, accounts and infrastructure at machine speed with bounded machine authority to update core infrastructure based on the data they are seeing in real time without human oversight. This authority will be grounded by reversible actions, explicit blast-radius limits and independent supervisory systems.
• Software changes are automatically tested, verified and monitored with significant advancements made in strong verification processes to ensure correctness and automated roll back procedures governed by other tools to fix a deployed failure rapidly.
• Human approval is no longer treated as sufficient proof of legitimacy and needs to be backed by supporting evidence systems that convey trust into the ecosystem and can create multiple points of verification.
• Organisations design for continuous compromise and rapid recovery rather than assuming perimeter security and thus design systems with the digital equivalent of “crumple zones” that can be used to absorb an attack and act like a fire break, giving the organisation time and space to focus on addressing the issue and deploying a rapid recovery force to reestablish services.

The key through-line that arises from this period of digital chaos is not a removal of trust, but rather the assumption of zero trust in all digital systems until strong evidence is provided to verify that it should be trustworthy.

Today, trust is often procedural such a person approved a transfer, wrote an email or signed a document. In the adaptive equilibrium, systems must prove identity, origin, authority and integrity continuously otherwise they are considered inherently untrustworthy. This will be by design.

This is a stark contrast to our history of digital media and systems (which was grounded in our trust of physical media and institutions) where trust was largely a given, and is now being disrupted by automation and AI tooling that prey upon that inherent trust.

The path to Adaptive Equilibrium will be challenging, with many significant and potentially destructive events along the way. But, it will end up like a digital version of our immune system - where it is able to rapidly respond to most novel threats effectively with minimal disruption.

Neither equilibrium is without risk

Neither of the more stable equilibria I would consider truly safe and definitely not benign - they both carry significant risks.

In the Controlled Equilibrium we are likely to see significant capability concentration in the hands of various states, which will be tempting to use against adversaries both abroad and domestically. The firms producing the models or the interfaces to them are likely to engage in regulatory capture, as we’ve seen multiple times previously from defence contractors, mining and energy companies over almost a century in most western democracies.

Unequal access across governments, companies and populations will leave citizens particularly exposed as they will not have access to sophisticated AI to support individual innovation or entrepreneurship and elongates the period of diffusion especially in the context of scientific advancement (due to extreme controls) or new business opportunities (due to entrenched incumbency and access).

The greatest risk of all though is that there is inherent strategic instability at a geopolitical level if state actors or large firms are only pretending compliance and we see far more sophisticated capabilities escape into a society wholly lacking in the resilience needed to deal with them.

Likewise, in an Adaptive Equilibrium future, we consign ourselves to years of fraud, disruption, erosion of trust and institutional failure before resilience catches up and stabilises the ecosystem. As we enter this world, surveillance and pervasive monitoring will be normalised with the commensurate erosion of privacy that implies - this will be the Faustian bargain of safety and trust mediated by machine-systems but at the expense of a lot of privacy.

In this future we live in a constant arms race between offensive and defensive capabilities, which will likely result in physical harm from attacks on infrastructure and automated systems. Those contracted to build the defensive capabilities will make huge sums, and become a new AI-Defence-Industrial-Complex and will exercise significant power over governments in the name of defending the state and its people.

Most concerning in this future is that in an era of ubiquitous zero-trust that citizens risk becoming excluded from regular society if they are unable to satisfy stronger identity requirements. This could particularly disadvantage those who immigrate from one trust regime to another without a bridge between them.

Zero-trust may leave people unable to access basic services. Image: ChatGPT

As you can see, neither future is inherently better than the other or unambiguously safe. The question we face as we go down one path or another is really to do with which risks are we willing to tolerate or be forced to absorb as a society?

The controlled equilibrium may be less disruptive if it can be achieved. The adaptive equilibrium may be easier to reach because every serious incident increases the incentive to build it.

History doesn’t provide us with a clean template to navigate this path forward. A Montreal Protocol style treaty may be useful for some parts of AI regulation, whereas the Nuclear Nonproliferation treaty may be better suited for aspects of frontier capabilities. The chaotic early internet produced serious failures, but society’s limited dependence on digital infrastructure constrained their systemic impact. That option no longer exists.

This mixed nature of AI is precisely why a single regulatory approach is probably inadequate.

I have no confident prediction about which path we will take. Following the incentives through, however, suggests two plausible destinations. Either a controlled world where resilience comes from restricting frontier capability through global coordination, or an adaptive world where capability spreads and resilience comes from rebuilding institutions.

Twenty years from now we may talk about a time before AI and a time after. Like we talk about time before the Internet and after it, or the time before COVID and after it.

The separation between those eras might be a treaty, a regulatory approach and a settlement to hold AI capability behind strong gates.

Or, it might be that period where the gates failed, trust collapsed, and we rebuilt the digital world around the assumption that powerful intelligence was available everywhere and systems had to adapt to that reality.

The Controlled Equilibrium - A.K.A the Flat Curve is just one possible destination. The only question left is whether the world can hold that heading long enough to arrive there.

Acknowledgements: With thanks to several interesting exhanges over the last month or so with John Allsopp, Xavier Rizos and Mark Pesce that kept me thinking about this topic.