wassname/anz-2040-draft
1
0
Fork 0
mirror of https://github.com/wassname/anz-2040-draft.git synced 2026-08-20 12:10:40 +08:00
Code Issues Packages Projects Releases Wiki Activity

ANZ 2040 draft: decision-tree site (prose + inline probability-flow SVG)

Browse Source
This commit is contained in:
wassname
2026-07-11 17:29:53 +08:00
commit 93d095e856
49 changed files with 24521 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitignore
+5
View File
@@ -0,0 +1,5 @@
handover.md
sources/resources/ai-2027-for-anz-june-2026.md
*.err
.DS_Store
__pycache__/
.nojekyll
View File
README.md
+31
View File
@@ -0,0 +1,31 @@
# ANZ 2040: the decision tree
The [AI 2040 / Plan A](https://ai-2040.com) forecast, retold from an Australia and New
Zealand middle-power point of view: a decision tree of the choices Canberra and
Wellington actually face as AI gets more powerful, with named endings and our own
probability estimates. Genre is [AI 2027](https://ai-2027.com) and
[Europe 2031](https://europe2031.ai). This is a draft.
## Build
The site is one static `index.html` with no build step on GitHub's side. To regenerate
it locally:
```sh
just build # or: python3 scripts/build_html.py && python3 scripts/build_tree.py
just serve # build, then serve at http://localhost:8080
```
- `anz-2040.md` is the source document.
- `scripts/build_html.py` turns it into `index.html`: prose via pandoc, the decision
tree via [mermaid.js](https://mermaid.js.org/) loaded from a CDN, CSS inline.
- `scripts/build_tree.py` turns `data/tree.json` into `tree.svg`, a probability-sized
view of the same tree (node area tracks our estimated probability). It is a separate,
simplified view, not generated from the mermaid tree.
- `sources/` holds the AI-2040 supplements and outside references the document cites.
## Credit
Built on the AI-2040 supplements by the [AI Futures Project](https://ai-2040.com), used
with full credit. Sources and one-line summaries are in
[`sources/INDEX.md`](sources/resources/INDEX.md).
anz-2040.md
+160
View File
@@ -0,0 +1,160 @@
# ANZ 2040: the decisions we still get to make
<!-- Rewritten from the evidence base by Codex, 2026-07-11. Probabilities are ours, conditional on rapid AI progress making the 2029 decision matter. -->
What can Australia and New Zealand actually decide if AI becomes much more powerful over the next decade?
We start from the excellent [AI 2040](https://ai-2040.com), which describes choices the United States and China could make. We zoom in on what those choices would mean for Australia and New Zealand, and what we could still do ourselves.
The probabilities below assume rapid AI progress. We leave aside the roughly 35% chance that progress plateaus first.
We don't get to choose whether the United States and China race, slow down, or make a deal. We don't get to choose whether a powerful AI remains under human control. But we do get a few choices about our position before those larger decisions are made. Some are cheap and useful across almost every future. The valuable later choices only exist if we prepare for them now.
[![ANZ 2040: choices and possible futures](tree.svg)](tree.svg)
## What Australia and New Zealand can do<sup title="ANZ-specific extrapolations from the cited scenarios">*</sup>
### 1. Prepare before it looks urgent (2026-28)
The later choices need ordinary government machinery: royalty laws, approved land, a way to pay a dividend, biosecurity capacity, and working relationships with other middle powers. None of this requires believing one precise AI forecast. It is cheap compared with discovering in 2029 that the legal and diplomatic work takes five years.
Biosecurity is the clearest case. AI 2040 proposes "massive investments into biosecurity and other measures to improve the resilience of the world," but does not assign the work to anyone. New Zealand already runs a serious agricultural biosecurity system. This is a useful thing for it to get unusually good at. <!-- local evidence: how-plan-a-solves-our-5-biggest-problems.md:69; faq.md:94 -->
### 2. Train the people the treaty needs (2026-29)
Australia and New Zealand can contribute people even when they cannot contribute frontier models. AI 2040 allocates only 1% of its safety budget to studying model character, meaning persistent behavioural tendencies or "personas+propensities." It says almost nothing about activation steering, which changes behaviour by intervening on a model's internal activity, or assistance games, which train an AI to infer what a person wants through cooperation. It does call truth-seeking AI and public forecasting a priority. Australia and New Zealand could specialise in those areas, as well as evaluations, treaty verification, biosecurity, and cyber work through the Five Eyes intelligence alliance. ([Alignment roadmap](https://ai-2040.com/supplements/alignment-roadmap), [AI for epistemics](https://ai-2040.com/supplements/ai-for-epistemics)) <!-- local evidence: alignment-roadmap.md:216; ai-for-epistemics.md:41 -->
### 3. Ask for terms before signing (2029)
Australia is named among the countries that join the deal. New Zealand is not mentioned in any of the 18 supplements we checked. Once inside, leaving could mean sanctions, cyberattack, or war. The realistic decision is therefore what to ask for before signing: hosted data centres, a strategic chip reserve, verification seats, equity in AI companies, and durable access rather than a revocable API account. <!-- local evidence: verification-plan.md:401; deal-decline.md:783 -->
Scott Alexander's summary is unusually blunt. Observer countries follow the rules in return for data centres, AI access, and a share of future wealth, even though the great powers don't strictly need to offer it. That makes accession our best bargaining moment. Signing the standard form gives it away. ([Introducing Plan A](https://www.astralcodexten.com/p/introducing-plan-a))
### 4. Host allied compute (2030-32)
AI 2040 moves advanced chip fabs and robot production into special economic zones that can be destroyed if the deal breaks. The zones are meant to sit near an ocean or an adversarial power, and the robot economy expands inside them until it is as large as today's human economy. North-west Australia fits those criteria and sits above much of the ore the machines would consume. ([Deal decline](https://ai-2040.com/supplements/deal-decline), [economics](https://ai-2040.com/supplements/economics-of-plan-a)) <!-- local evidence: deal-decline.md:872; economics-of-plan-a.md:439 -->
Hosting creates exposure as well as income. The same cluster can be a target, collateral for the treaty, and leverage against being cut off. Tom Davidson makes the mechanism explicit: if the US withdraws AI access, allies could destroy US data centres on their territory. A kill switch negotiated at accession is leverage. A foreign cluster accepted without access rights is just exposure. ([Davidson 2026](https://newsletter.forethought.org/p/how-can-the-middle-powers-avoid-getting))
There is also a smaller trusted-vault option. Australia could host cold-storage model weights, treaty verification equipment, or monitoring linked to Pine Gap without hosting the whole robot economy. Geologically stable ground and Five Eyes trust make this the lowest-volume version of the policy.
### 5. Keep a public stake in the machine economy (2031-34)
The US dividend in AI 2040 comes from compute permits. By 2035, the scenario sends 75% of the US permit share to US citizens. Australia and New Zealand cannot rely on that. Our dividend would need resource royalties, land taxes, and public equity held through the Future Fund and NZ Super. Those rules need to exist before cognitive labour becomes cheap and wage earners lose their bargaining power. ([Economics of Plan A](https://ai-2040.com/supplements/economics-of-plan-a)) <!-- local evidence: economics-of-plan-a.md:368 -->
Who owns the rents matters more than the size of the boom. The same ore and land can support a public dividend or a few mining dynasties. Leicht and Ball warn that selling a strategic firm can look like an unexpected windfall while giving up the country's "ticket guaranteeing their home country's stake in the AI economy." In a rapid takeoff, taking only cash is especially bad: the useful payment is equity, access, or ownership of productive assets. ([The Race Worth Winning](https://www.thefai.org/posts/the-race-worth-winning-middle-powers-in-the-age-of-machine-intelligence)) <!-- local evidence: fai-race-worth-winning.md:1747-1762 -->
Australia probably starts above the generic non-US country. Compulsory superannuation gives many households an existing claim on global capital, while mining gives governments a claim on inputs the robot economy needs. Neither is evenly shared by default. Superannuation gains follow existing balances and investment choices. Mining income reaches households only when royalties, taxes, public ownership, or transfers capture it. That is what separates the Dividend commonwealth from the Quarry economy. <!-- local evidence: fai-race-worth-winning.md:1925-1939 -->
The political claim is stronger, and more speculative. If governments no longer depend on wages or citizen labour, owners and automated security may stop needing broad consent. A legally entrenched citizen claim on public assets keeps ordinary people tied to the revenue stream and gives them something material to defend through democratic institutions. We are treating distribution as a constraint on domestic power, not merely as welfare.
### 6. Choose a side if the deal collapses (2030s)
If the US-China deal declines, Australia's security guarantor and largest customer pull in opposite directions. Joining the US bloc keeps the alliance and loses trade. A concert with Japan, Korea, Canada, the Netherlands, and the UK might bargain for access, but only if it was built before the crisis. Leicht and Ball's warning is that a coalition improvised late will be picked off "one by one." Trying to hedge returns us to someone else's empire by a slower route. Free Chinese open-weight models don't remove the dependency; a supplier earning nothing from us also loses nothing by cutting us off. ([The Race Worth Winning](https://www.thefai.org/posts/the-race-worth-winning-middle-powers-in-the-age-of-machine-intelligence), [Davidson 2026](https://newsletter.forethought.org/p/how-can-the-middle-powers-avoid-getting))
## Where this could leave us<sup title="ANZ-specific extrapolations from the cited scenarios">*</sup>
Doom and concentrated global power account for most of our probability. Australia and New Zealand only get the later choices if humanity keeps control and no single actor takes it all.
### I. Doom, about 50%
AI 2040's own comparison gives Plan A a 42% chance of a "great future," falling to 10-25% for its alternatives. Combining the table's conditional rows implies misaligned takeover at roughly one quarter under Plan A and roughly three quarters in the modal race. Our 50% is a rough mixture across those possibilities. <!-- local evidence: comparing-possible-plans.md:104 -->
Yudkowsky and Soares state the doom case much more strongly than our estimate:
> "If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die."
>
> Eliezer Yudkowsky and Nate Soares, [*If Anyone Builds It, Everyone Dies*](https://www.hachettebookgroup.com/titles/eliezer-yudkowsky/if-anyone-builds-it-everyone-dies/9780316595667/)
Once control is lost, Australia has no policy response left. The preparations above have to improve the odds before then.
### II. Someone else's empire, about 25%
The AI remains aligned, but control does not spread. A government, company, or person holds the decisive systems. Australia and New Zealand may be materially rich, especially while iron demand is high, but they live under rules set elsewhere.
> "a tiny group of people, or possibly just a single individual, is effectively in control of the world’s only army of superintelligences"
>
> AI Futures Project, [AI 2040](https://ai-2040.com) <!-- local evidence: main.md:61 -->
### III. Dividend commonwealth
We negotiated terms, captured resource and land rents, and built the dividend before wages collapsed. Households share in the assets that became valuable instead of watching those assets reprice around them. The better version also uses AI inside courts, health systems, and government rather than treating it only as something to regulate.
> "In Plan A, most of the US permit revenue share (75% in 2035) is redistributed to the US population as a Citizen’s Dividend, resulting in roughly $1M/yr per person in 2035 and around $10M/yr in 2040."
>
> AI Futures Project, [Economics of Plan A](https://ai-2040.com/supplements/economics-of-plan-a) <!-- local evidence: economics-of-plan-a.md:368 -->
### IV. Quarry economy
This is the drift outcome. The country exports what the robot economy needs, but has no dividend rail and no public stake. AI 2040's land model makes the distribution problem concrete: single-family homes rise about 40 times while 97% of their paper value becomes land. Incumbent owners gain while cognitive wages fall. The nation is rich and the median household is poor, without anyone needing to plan it. <!-- local evidence: economics-of-plan-a.md:389 -->
> "As AI devalues human capital relative to physical and intangible assets, capital income consumes a growing proportion of national income. Elevated spending on luxury goods and a productivity-driven investment boom sustains elevated economic growth even as millions face unemployment and diminished purchasing power."
>
> UK Government Office for Science and AI Security Institute, [AI Scenarios 2030](https://www.gov.uk/government/publications/ai-scenarios-2030-helping-policymakers-plan-for-the-future-of-ai/ai-scenarios-2030-helping-policymakers-plan-for-the-future-of-ai) <!-- local evidence: govuk-ai-scenarios-2030.md:655 -->
The resource boom also expires. On current reserves and extraction, iron is the shortest-dated major reserve at roughly 55 years. A machine economy growing every 6-12 months would bring exhaustion and substitution forward. AI 2040 models the demand for energy and land, but not the Australian terms of trade or the cost of scaling mines.
### V. Allied compute host
Australia accepts the data centres and robot zones, but only after securing access and ownership terms. Land, power, and Five Eyes trust become a stake in the machine economy. The price is physical exposure: the hosted clusters are treaty collateral, and the treaty's wars become ours.
> "If the US withdraws AI access, allies could destroy US data centres in response. It's a way to lock in the deal."
>
> Tom Davidson, [How can the middle powers avoid getting trounced?](https://newsletter.forethought.org/p/how-can-the-middle-powers-avoid-getting)
### VI. Garrison ally
The deal collapses and Australia chooses the US bloc. We keep the security relationship and lose much of the Chinese market. Iron becomes a strategic allocation rather than an ordinary export. This future is poorer and tenser than the dividend or hosting alternatives, but Australia still has a seat.
> "middle powers should help the US, and make sure they are rewarded with continued access to frontier AI and new technologies (including military tech)"
>
> "The only alternative that makes sense to me is siding with China."
>
> Tom Davidson, [How can the middle powers avoid getting trounced?](https://newsletter.forethought.org/p/how-can-the-middle-powers-avoid-getting) <!-- local evidence: davidson-middle-powers-plan.md:26-28 -->
### VII. Boring decade, about 10%
AI stalls or governments shut the frontier down. The consents are unused, the royalty argument was early, and the extra biosecurity capacity looks like insurance that did not pay. This is the cheapest way for our preparations to be wrong.
> "AI progress slows, and AI causes less disruption than expected."
>
> UK Government Office for Science and AI Security Institute, [AI Scenarios 2030](https://www.gov.uk/government/publications/ai-scenarios-2030-helping-policymakers-plan-for-the-future-of-ai/ai-scenarios-2030-helping-policymakers-plan-for-the-future-of-ai) <!-- local evidence: govuk-ai-scenarios-2030.md:186 -->
### VIII. Middle-power concert
The deal declines, but a coalition formed before the crisis holds together. Australia and New Zealand pool their supply-chain and hosting leverage with Japan, Korea, Canada, the Netherlands, and the UK. None could demand durable access alone. Together they may get terms and help write the rules.
> "Coordination, done right, reassures partners that they can focus on their comparative advantage instead of pursuing full autarky; and it enables powers to see eye to eye with AI exporters instead of being picked off by great powers offering rewards for defection one by one."
>
> Leicht and Ball, [The Race Worth Winning](https://www.thefai.org/posts/the-race-worth-winning-middle-powers-in-the-age-of-machine-intelligence) <!-- local evidence: fai-race-worth-winning.md:437-449 -->
### IX. ~~Homegrown autocracy~~ free beer and rugby for life
This is the domestic version of the global power-grab risk. Cognitive wages collapse, public ownership never develops, and automated security removes the state's remaining practical dependence on citizens. No coup is required. Political leverage drains away with economic leverage.
> Luckily we completely dodged a coup and now have Benevolent Dictator for Life (BDFL) Gina Palmer to lead us through. Everyone is very happy, especially since the free beer, rugby, and netball are a great respite from our quickly shrinking population and ever more confusing global situation. - Joe Citizen, 2041
> "AI could create a similar effect: if governments can generate massive revenue from taxing AI projects rather than citizens, heads of state may lose their economic incentive to ensure citizens prosper. This would weaken citizens’ power to resist coup and backsliding attempts."
>
> "By replacing government employees with loyal AI systems, a head of state could remove important checks on their power."
>
> Davidson, Finnveden, and Hadshar, [AI-Enabled Coups](https://www.forethought.org/research/ai-enabled-coups-how-a-small-group-could-use-ai-to-seize-power) <!-- local evidence: forethought-ai-enabled-coups.md:425-431 -->
## Where we depart from AI 2040
AI 2040 models energy consumption rising about eight times by 2040 and notes that proven mineral reserves cover decades of current extraction, but it does not price the tonnage needed for a robot economy growing every 6-12 months. Coal, uranium, energy trade, and Australian mine capacity are outside its model. ([Economics of Plan A](https://ai-2040.com/supplements/economics-of-plan-a))
We use its land estimates with the authors' warning that this is a "very rough model" they do not necessarily endorse. We also give model character and activation steering more weight than their roadmap does.
AI 2040 does share some US wealth abroad, but very unevenly. In 2032 its American dividend starts at $45,000 while adults outside the US and China average $1,200. By 2035 the figures are about $1M and $10,000. Australia may do better than that generic non-US payment through superannuation and mining, but only if the gains are broadly owned. A rich national balance sheet can still leave the median household without economic leverage. ([AI 2040](https://ai-2040.com/?choices=plan-a-root)) <!-- local evidence: main.md:553-557 -->
We leave out a trans-Tasman split and a separate Australian decision during US sabotage operations because neither changes the main choices above.
## Sources and thanks
This is built from the AI Futures Project's [AI 2040](https://ai-2040.com) scenario and supplements. Their caveat applies to our use of the numbers too: the economic model "plausibly contains bugs" and they do not trust its outputs to be accurate. The local source mirrors retain exact line references for auditing.
It also benefited from a June 2026 scoping discussion with authors of related scenarios and Australian policy participants. Those workshop notes are private and are not included in the public source archive.
<small><sup>*</sup> The Australian and New Zealand decisions and futures are extrapolations from the cited work. A citation means we used that source, not that its authors endorse this scenario.</small>
The middle-power framing also draws on [AI 2027](https://ai-2027.com), [Europe 2031](https://europe2031.ai), Tom Davidson's [middle-power plan](https://newsletter.forethought.org/p/how-can-the-middle-powers-avoid-getting), Leicht and Ball's [The Race Worth Winning](https://www.thefai.org/posts/the-race-worth-winning-middle-powers-in-the-age-of-machine-intelligence), and Scott Alexander's [Introducing Plan A](https://www.astralcodexten.com/p/introducing-plan-a). Australian groundwork came from the [e61 Institute](https://e61.in), [Tech Policy Design Institute](https://techpolicy.au/aiagency), [ASPI](https://www.aspistrategist.org.au/data-centres-are-australias-chance-to-shape-ais-future/), and [Kate Chaney MP](https://www.katechaney.com.au/making_technology_safe).
data/tree.json
+53
View File
@@ -0,0 +1,53 @@
{
"comment": "Claude: single source for tree.svg. Each item has a tier (row) and left-to-right order; two items per row max. `p` and edge flows record our probability mass conditional on the 2029 decision mattering. Display widths fit the text. The HTML build fails if an href no longer matches a prose heading.",
"title": "Choose a path",
"canvas": {"w": 560, "top": 40, "row_h": 115, "node_h": 56, "gap": 44, "floor_w": 130, "p_scale": 300, "choice_w": 150},
"kinds": {
"global": {"fill": "#e0e0e0", "stroke": "#888888"},
"hazard": {"fill": "#f4cccc", "stroke": "#990000"},
"anz": {"fill": "#ffd966", "stroke": "#b8860b"},
"robust": {"fill": "#b6d7a8", "stroke": "#38761d"},
"outcome": {"fill": "#cfe2f3", "stroke": "#3d85c6"}
},
"nodes": [
{"id": "A", "tier": 0, "order": 0, "kind": "robust", "href": "#1-prepare-before-it-looks-urgent-2026-28", "title": "1. Prepare early", "sub": "2026-28: royalties, biosecurity, allies"},
{"id": "F", "tier": 0, "order": 1, "kind": "robust", "href": "#2-train-the-people-the-treaty-needs-2026-29", "title": "2. Build the people", "sub": "2026-29: safety testers, inspectors"},
{"id": "FORK", "tier": 1, "order": 0, "kind": "global", "href": "#what-australia-and-new-zealand-can-do", "title": "2029: the US chooses", "sub": "go-slow deal with China, or race?"},
{"id": "VII", "tier": 2, "order": 0, "kind": "outcome", "p": 0.10, "href": "#vii-boring-decade-about-10", "title": "VII. Boring decade", "sub": "~10%"},
{"id": "AL", "tier": 2, "order": 1, "kind": "hazard", "p": 0.90, "href": "#i-doom-about-50", "title": "RISK: AI escapes control", "sub": "~75% racing, ~25% in the deal"},
{"id": "I", "tier": 3, "order": 0, "kind": "outcome", "p": 0.50, "href": "#i-doom-about-50", "title": "I. Doom", "sub": "~50%"},
{"id": "PC", "tier": 3, "order": 1, "kind": "hazard", "p": 0.40, "href": "#ii-someone-elses-empire-about-25", "title": "RISK: power seized", "sub": "one actor holds the AI"},
{"id": "II", "tier": 4, "order": 0, "kind": "outcome", "p": 0.25, "href": "#ii-someone-elses-empire-about-25", "title": "II. Someone else's empire", "sub": "~25% — seized abroad"},
{"id": "HOME", "tier": 4, "order": 1, "kind": "hazard", "href": "#ix-homegrown-autocracy-free-beer-and-rugby-for-life", "title": "RISK: power grabbed at home", "sub": "seized in our own country"},
{"id": "B", "tier": 5, "order": 0, "kind": "anz", "href": "#3-ask-for-terms-before-signing-2029", "title": "3. Sign, or set our price?", "sub": "the deal reaches us, 2029"},
{"id": "IX", "tier": 5, "order": 1, "kind": "outcome", "href": "#ix-homegrown-autocracy-free-beer-and-rugby-for-life", "title": "IX. Homegrown autocracy", "sub": "citizens stop being needed"},
{"id": "E", "tier": 6, "order": 0, "kind": "anz", "href": "#6-choose-a-side-if-the-deal-collapses-2030s", "title": "6. If the deal collapses", "sub": "US bloc, concert, or neutral?"},
{"id": "C", "tier": 6, "order": 1, "kind": "anz", "href": "#4-host-allied-compute-2030-32", "title": "4. Host compute here?", "sub": "2030-32"},
{"id": "VIII", "tier": 7, "order": 0, "kind": "outcome", "href": "#viii-middle-power-concert", "title": "VIII. Middle-power concert", "sub": ""},
{"id": "V", "tier": 7, "order": 1, "kind": "outcome", "href": "#v-allied-compute-host", "title": "V. Allied compute host", "sub": ""},
{"id": "VI", "tier": 8, "order": 0, "kind": "outcome", "href": "#vi-garrison-ally", "title": "VI. Garrison ally", "sub": ""},
{"id": "D", "tier": 8, "order": 1, "kind": "anz", "href": "#5-keep-a-public-stake-in-the-machine-economy-2031-34", "title": "5. Tax it, own a share?", "sub": "2031-34"},
{"id": "III", "tier": 9, "order": 0, "kind": "outcome", "href": "#iii-dividend-commonwealth", "title": "III. Dividend commonwealth", "sub": ""},
{"id": "IV", "tier": 9, "order": 1, "kind": "outcome", "href": "#iv-quarry-economy", "title": "IV. Quarry economy", "sub": "drift default"}
],
"edges": [
{"from": "A", "to": "FORK", "flow": 0.50},
{"from": "F", "to": "FORK", "flow": 0.50},
{"from": "FORK", "to": "VII", "flow": 0.10, "label": "stalls"},
{"from": "FORK", "to": "AL", "flow": 0.90, "label": "race / deal"},
{"from": "AL", "to": "I", "flow": 0.50, "label": "control lost"},
{"from": "AL", "to": "PC", "flow": 0.40, "label": "control kept"},
{"from": "PC", "to": "II", "flow": 0.24, "label": "seized abroad"},
{"from": "PC", "to": "HOME", "flow": 0.05, "label": "seized at home"},
{"from": "PC", "to": "B", "flow": 0.11, "label": "power stays spread"},
{"from": "HOME", "to": "IX", "flow": 0.05, "label": "no public stake"},
{"from": "B", "to": "C", "flow": 0.08, "label": "set our price"},
{"from": "B", "to": "E", "flow": 0.03, "label": "if deal collapses", "dashed": true},
{"from": "C", "to": "V", "flow": 0.04, "label": "host it"},
{"from": "C", "to": "D", "flow": 0.04, "label": "stay supplier"},
{"from": "E", "to": "VIII", "flow": 0.015, "label": "concert"},
{"from": "E", "to": "VI", "flow": 0.015, "label": "US bloc"},
{"from": "D", "to": "III", "flow": 0.02, "label": "public stake"},
{"from": "D", "to": "IV", "flow": 0.02, "label": "no stake"}
]
}
global.css
+609
View File
@@ -0,0 +1,609 @@
@tailwind base;
@tailwind components;
@tailwind utilities;
:root {
--vivid-background: #fffff8;
--vivid-foreground: #000;
--dull-background: #f0f0f0;
--dull-foreground: #666;
--accent: #2A623D; /* Slytherin green */
--plan-s-background: #31566f;
--plan-a-background: #2A623D;
--race-background: #8B0000;
--sabotage-background: #c77400;
--slowdown-background: #2A623D;
}
.race {
--accent: var(--race-background);
}
@property --w {
syntax: '<percentage>';
inherits: true;
initial-value: 0.00000001%;
}
/* Extra px the supplement essay's content column is nudged right on wide layouts
(toc rail visible). Factored out because TWO rules must agree on it: the shift
applied to the content column's text (.supplement-content-shift) and the equal
shift applied to the header row (.supplement-header-row) so "AI 2040" lines up
with the content title. Edit here to move both together. */
:root {
--supplement-content-shift: 30px;
}
.main-grid {
display: grid;
margin-left: auto;
margin-right: auto;
grid-template-areas:
"gap1 tocGap header header header gap2"
"gap1 topGraph topGraph topGraph topGraph gap2"
"gap1 toc content graphgap graph gap2"
"gap1 toc2 choice graphgap2 graph gap2"
"gap1 fgap fgap fgap fgap gap2"
"gap1 gap3 cFooter footer graphFooter gap2";
/* toc column: as the page narrows, the flexible gap columns (1fr/0.3fr)
give up their space first while the rail keeps its full 220px. Once the
viewport can't fit the other columns at their preferred widths (4px gap +
700px content + 32px graphgap + 430px graph = 1166px), the toc column
itself shrinks — down to 140px — and the rail labels wrap to the narrower
column. Below 1281px the rail is hidden entirely (see below). */
grid-template-columns: 4px clamp(140px, calc(100vw - 1166px), 220px) minmax(300px, 700px) minmax(32px, 1fr) minmax(min-content, 430px) 0.3fr;
grid-template-rows: auto auto auto auto auto;
padding-top: 8px;
}
/* Pushes the central content column an extra 30px to the right, on top of the
30px gained by widening the toc column. Only applied on layouts where the
toc rail is visible (≥1281px); narrower views hide the rail entirely. */
@media (min-width: 1281px) {
.supplement-content-shift {
padding-left: var(--supplement-content-shift);
}
}
/* Aligns the supplement header ("AI 2040") with the essay's content column.
The header grid area is `header header header` (parent columns 3–5), so it
already starts exactly at the `content` column's left edge — left-aligned
header text begins there for free. The only remaining offset is the same
`--supplement-content-shift` that nudges the content column's *text* right
(padding on <main>), so we apply that identical shift to the header row and
the two line up. Scoped to the toc-rail layouts (≥1281px), where the shift
applies; narrower views don't shift the content, so the header already
matches without it. */
@media (min-width: 1281px) {
.supplement-header-row {
padding-left: var(--supplement-content-shift);
}
}
@media (max-width: 1280px) {
.main-grid.main-grid {
grid-template-columns: 32px 0px minmax(300px, 700px) minmax(64px, 1fr) minmax(min-content, 430px) 8px;
}
.section-nav {
display: none;
}
}
@media (max-width: 930px) {
.main-grid.main-grid {
grid-template-areas:
"gap1 header header header gap2"
"gap1 topGraph topGraph graph gap2"
"gap1 content content content gap2"
"gap1 choice choice choice gap2"
"gap1 fgap fgap fgap gap2"
"gap1 cFooter cFooter cFooter gap2";
grid-template-columns: 10px minmax(0px, 900px) 64px 0px 10px;
}
}
@media (min-width: 1600px) {
.main-grid.main-grid {
grid-template-columns: 8px minmax(250px, 1fr) minmax(300px, 700px) minmax(32px, 96px) minmax(min-content, 430px) minmax(8px, 1fr);
}
}
body.antialiased {
-webkit-font-smoothing: subpixel-antialiased;
}
@layer utilities {
.text-balance {
text-wrap: balance;
}
}
summary {
cursor: pointer;
}
details b,
details strong,
.report-details b,
.report-details strong {
font-weight: 600;
}
.content-with-chart {
padding-top: 50vh;
display: flex;
align-items: flex-start;
gap: 40px;
}
.main-content {
flex: 1;
}
.sticky-chart-container {
position: sticky;
top: 20px;
width: 330px;
overflow: hidden;
margin-top: 250px;
opacity: 0.05;
transition: opacity 0.15s ease-in-out;
flex-grow: 0.35;
border: 1.5px solid grey;
border-radius: 12px;
padding: 8px;
}
.grid-small {
grid-template-areas:
"rdmultiple rdmultiple rdmultiple"
"computeBudgets computeBudgets goalsPieChart"
"capability capability goalsPieChart"
"keystats keystats keystats"
"techBuckets techBuckets techBuckets"
"agentPopulation agentPopulation agentPopulation";
grid-template-columns: 1fr 1fr 1fr;
grid-template-rows: auto auto 1fr auto auto auto;
/* grid-template-rows: 130px 60px 60px 100px 100px 50px; */
}
/* Single column grid layout for mobile fullscreen view */
.grid-small-mobile {
grid-template-areas:
"rdmultiple rdmultiple"
"capability goalsPieChart"
"keystats keystats"
"techBuckets techBuckets"
"agentPopulation agentPopulation";
grid-template-columns: auto 100px;
grid-template-rows: auto auto auto auto auto;
width: 100%;
max-width: 100vw;
overflow-x: hidden;
}
.footnote-hover {
position: relative;
text-decoration: underline dotted;
cursor: pointer;
}
/* The container for the actual footnote DOM */
.footnote-popup {
position: absolute;
top: 1.2em; /* or bottom: 1.2em, etc. */
left: 0;
width: 250px;
max-width: 300px;
background-color: rgba(245, 245, 245, 0.95);
color: #444;
border: 1px solid #ccc;
border-radius: 6px;
padding: 0.5rem;
font-size: 0.85rem;
line-height: 1.2;
z-index: 9999;
}
/* Show the .footnote-popup when hovering over the parent link */
.footnote-hover:hover .footnote-popup {
display: block;
}
/* HUD PIP and fullscreen mode styles */
.hud-pip {
transform-origin: bottom right;
box-shadow: 0 10px 25px rgba(0, 0, 0, 0.2);
z-index: 999;
}
.hud-pip [data-state="open"] {
display: none !important; /* Disable tooltips in PIP mode */
}
.hud-fullscreen {
z-index: 999;
}
@layer base {
:root {
--background: 0 0% 100%;
--foreground: 0 0% 3.9%;
--card: 0 0% 100%;
--card-foreground: 0 0% 3.9%;
--popover: 0 0% 100%;
--popover-foreground: 0 0% 3.9%;
--primary: 0 0% 9%;
--primary-foreground: 0 0% 98%;
--secondary: 0 0% 96.1%;
--secondary-foreground: 0 0% 9%;
--muted: 0 0% 96.1%;
--muted-foreground: 0 0% 45.1%;
--accent: 0 0% 96.1%;
--accent-foreground: 0 0% 9%;
--destructive: 0 84.2% 60.2%;
--destructive-foreground: 0 0% 98%;
--border: 0 0% 89.8%;
--input: 0 0% 89.8%;
--ring: 0 0% 3.9%;
--chart-1: 12 76% 61%;
--chart-2: 173 58% 39%;
--chart-3: 197 37% 24%;
--chart-4: 43 74% 66%;
--chart-5: 27 87% 67%;
--radius: 0.5rem;
}
/* .dark {
--background: 0 0% 3.9%;
--foreground: 0 0% 98%;
--card: 0 0% 3.9%;
--card-foreground: 0 0% 98%;
--popover: 0 0% 3.9%;
--popover-foreground: 0 0% 98%;
--primary: 0 0% 98%;
--primary-foreground: 0 0% 9%;
--secondary: 0 0% 14.9%;
--secondary-foreground: 0 0% 98%;
--muted: 0 0% 14.9%;
--muted-foreground: 0 0% 63.9%;
--accent: 0 0% 14.9%;
--accent-foreground: 0 0% 98%;
--destructive: 0 62.8% 30.6%;
--destructive-foreground: 0 0% 98%;
--border: 0 0% 14.9%;
--input: 0 0% 14.9%;
--ring: 0 0% 83.1%;
--chart-1: 220 70% 50%;
--chart-2: 160 60% 45%;
--chart-3: 30 80% 55%;
--chart-4: 280 65% 60%;
--chart-5: 340 75% 55%;
} */
}
@layer base {
* {
@apply border-border;
}
body {
@apply bg-background text-foreground;
}
}
h2 {
scroll-margin-top: 25vh;
}
/* Add this CSS rule into your global stylesheet */
/* Reveal the text by clipping it from the bottom upward */
@keyframes revealDown {
0% {
clip-path: inset(0 0 100% 0); /* Clipped from the bottom (hidden) */
opacity: 1;
}
50% {
opacity: 0.8;
clip-path: inset(0 0 95% 0);
}
100% {
clip-path: inset(0 0 0 0); /* Fully visible */
opacity: 1;
}
}
/* CSS class to trigger the revealDown animation */
.animate-revealDown {
animation: revealDown 1.5s ease-out forwards;
}
/* Audio player scrub label: slide the chapter title in/out as you scrub. */
@keyframes scrub-enter-right {
from { transform: translateX(100%); opacity: 0; }
to { transform: translateX(0); opacity: 1; }
}
@keyframes scrub-enter-left {
from { transform: translateX(-100%); opacity: 0; }
to { transform: translateX(0); opacity: 1; }
}
@keyframes scrub-exit-left {
from { transform: translateX(0); opacity: 1; }
to { transform: translateX(-100%); opacity: 0; }
}
@keyframes scrub-exit-right {
from { transform: translateX(0); opacity: 1; }
to { transform: translateX(100%); opacity: 0; }
}
.animate-scrub-enter-right { animation: scrub-enter-right 0.3s ease-out; }
.animate-scrub-enter-left { animation: scrub-enter-left 0.3s ease-out; }
.animate-scrub-exit-left { animation: scrub-exit-left 0.3s ease-out forwards; }
.animate-scrub-exit-right { animation: scrub-exit-right 0.3s ease-out forwards; }
/* Override katex default larger font size */
.katex.katex {
font-size: 1em;
}
number-flow-react::part(left),
number-flow-react::part(right),
number-flow-react::part(left)::after,
number-flow-react::part(right)::after,
number-flow-react::part(symbol) {
padding: calc(var(--number-flow-mask-height, 0.25em) / 2) 0;
}
.essay-footnote {
flex-direction: row;
gap: 10px;
}
.essay-footnote p, .essay-footnote ol, .essay-footnote ul {
font-size: 14px;
margin: 0;
line-height: 1.5;
}
main h1 {
font-size: 52px;
}
.footnote-page p {
display: inline;
}
.summary-chart p {
font-size: 14px;
line-height: 1.4;
}
.tab-boxes p, .tab-boxes li {
font-size: 16px;
line-height: 1.45;
margin-bottom: 0.5em;
}
.tab-boxes ol {
margin-top: 0.5em;
margin-bottom: 0.5em;
}
.tab-boxes a {
text-decoration: none;
color: var(--accent);
font-weight: 600;
}
@supports not selector(:first-child) {
.requires-modern-css {
display: none !important;
}
}
.scrollbar-hide, .scrollbar-hide * {
-ms-overflow-style: none;
scrollbar-width: none;
scrollbar-color: transparent transparent;
}
.scrollbar-hide::-webkit-scrollbar {
display: none;
}
blockquote {
border: 2px solid #232323;
border-radius: 8px;
padding: .25rem 1.5rem;
margin: 1rem 0;
}
.good-ending-text { font-size: 1.4rem; }
.good-ending-text h1 { font-size: 3rem; margin: 1.25em 0 1em 0; }
@media (max-width: 767px) {
.good-ending-text h1 { font-size: 2rem; line-height: 1.3; }
/* Keep the heading ladder descending under the 2rem mobile h1: tufte's
desktop sizes (h2 2.25rem, h3 1.85rem, h4 1.5rem) would otherwise
render h2 BIGGER than h1 and h4 bigger than the scaled h3. h5 keeps
tufte's 1.3rem, which already sits below the 1.35rem h4. */
.good-ending-text h2 { font-size: 1.7rem; }
.good-ending-text h3 { font-size: 1.45rem; }
.good-ending-text h4 { font-size: 1.35rem; }
/* ai-2027's mobile body size. tufte.css carries the same rule at 730px,
but the .good-ending-text font-size above out-specifies it (paragraphs
inherit), so it must be restated at this level. 767px matches the rest
of this site's mobile breakpoint. */
.good-ending-text { font-size: 1.25rem; }
.good-ending-text :where(p, dl, ol, ul, summary, blockquote, li) {
line-height: 1.45;
}
}
.good-ending-text :where(p, dl, ol, ul, summary, blockquote, li) { font-size: inherit; }
/* Narrow non-mobile window (two-column playbook layout, 930–1024px): shave a
couple px off body text and scale headings down ~10%. Set once here at the
top-level container instead of threading a class through Markdown.tsx.
Line-height is unitless (1.5) so it auto-follows whatever font-size wins —
change the size and the leading tracks it, no separate override needed. */
@media (min-width: 930px) and (max-width: 1135px) {
.good-ending-text { font-size: 1.2rem }
.good-ending-text :where(p, dl, ol, ul, summary, blockquote, li) { line-height: 1.5; }
.good-ending-text h1 { font-size: 2.4rem; }
.good-ending-text h2 { font-size: 2.03rem; }
.good-ending-text h3 { font-size: 1.53rem; }
.good-ending-text h4 { font-size: 1.35rem; }
.good-ending-text h5 { font-size: 1.17rem; }
}
.markdown-content.supplement-essay-prose {
box-sizing: border-box;
}
.markdown-content.supplement-essay-prose > :not(h1):not(h2):not(h3):not(h4):not(h5):not(h6) {
max-width: 800px;
margin-left: auto;
margin-right: auto;
}
.markdown-content.supplement-essay-prose > h1,
.markdown-content.supplement-essay-prose > h2,
.markdown-content.supplement-essay-prose > h3,
.markdown-content.supplement-essay-prose > h4,
.markdown-content.supplement-essay-prose > h5,
.markdown-content.supplement-essay-prose > h6 {
max-width: none;
width: 100%;
box-sizing: border-box;
overflow-wrap: break-word;
}
/* Top-level Part headings (h1) get a strong gap above to separate the major
sections — base tufte h1 has margin-top:0, which left Part 2 cramped. */
.markdown-content.supplement-essay-prose > h1 {
margin-top: 5rem;
}
/* Images carry a small negative left margin for optical alignment in the main
essay column; inside an expanded details section that would push them across
the section's brown left rule. Reset it there so they stay inside the rule. */
details .ml-\[-20px\],
details .ml-\[-8px\] {
margin-left: 0 !important;
}
/* "Back to contents" arrow in the left margin of supplement headings. Hidden
everywhere else (the link target #top-toc only exists in supplements). */
.toc-backlink {
display: none;
}
.markdown-content.supplement-essay-prose > :is(h1, h2, h3, h4, h5, h6) {
position: relative;
}
/* On every level of section heading inside supplements (h1 = top-level
section like Executive Summary / Verifying the deal; h2/h3 = sub-sections
and timeline rows). Scoped via :has(#top-toc) so the triangle only
appears in supplements whose body actually contains a :::toc block —
right now that's only verification-plan. Em-sized border-triangle,
vertically centered. */
.markdown-content.supplement-essay-prose:has(#top-toc) > :is(h1, h2, h3) .toc-backlink {
display: inline-block;
position: absolute;
right: 0;
top: 50%;
transform: translateY(-50%);
width: 0;
height: 0;
border-left: 0.32em solid transparent;
border-right: 0.32em solid transparent;
border-bottom: 0.5em solid rgba(0, 0, 0, 0.2);
text-decoration: none;
transition: border-bottom-color 0.15s ease-out;
}
.markdown-content.supplement-essay-prose:has(#top-toc) > :is(h1, h2, h3) .toc-backlink:hover {
border-bottom-color: rgba(0, 0, 0, 0.55);
}
.ai2030-feedback-scroll-area {
scrollbar-width: none;
-ms-overflow-style: none;
}
.ai2030-feedback-scroll-area::-webkit-scrollbar {
width: 0;
height: 0;
display: none;
}
/* Feed panel on /feedback sliding open from its collapsed strip. */
@keyframes ai2030-feed-slide-in {
from {
width: 44px;
}
to {
width: 420px;
}
}
/* Persistent highlight on the selected text while its feedback dialog is open,
so it stays visible even after focus moves into the dialog inputs. */
::highlight(ai2030-feedback-selection) {
background-color: #accef7;
color: black;
}
/* While the feedback dialog's persistent highlight is showing, the native
selection is kept alive (so the reader can still copy the text) but restyled
to the same color so the two highlights render as one. */
.ai2030-feedback-selection-active ::selection {
background-color: #accef7;
color: black;
}
/* Gentle pop when a submitted-feedback margin icon first appears (page load,
and right after the feedback dialog fades out on submit). */
@keyframes ai2030-feedback-icon-pop {
from {
opacity: 0;
transform: scale(0.6);
}
to {
opacity: 1;
transform: scale(1);
}
}
.ai2030-feedback-icon-pop {
animation: ai2030-feedback-icon-pop 180ms ease-out;
}
/* Subtle dotted underline on passages the reader has already left feedback on
(rendered by SubmittedFeedbackLayer via the CSS Custom Highlight API). */
::highlight(ai2030-feedback-submitted) {
text-decoration-line: underline;
text-decoration-style: dotted;
text-decoration-color: rgba(0, 0, 0, 0.45);
text-underline-offset: 3px;
}
/* Solid underline on the passage whose margin icon/popover is being hovered,
so the reader can tell which line the feedback belongs to. */
::highlight(ai2030-feedback-submitted-active) {
text-decoration-line: underline;
text-decoration-style: solid;
text-decoration-color: rgba(0, 0, 0, 0.8);
text-underline-offset: 3px;
}
.ai2030-feedback-inline-highlight {
background-color: rgba(255, 220, 90, 0.45);
border-radius: 2px;
padding: 0 1px;
cursor: pointer;
transition: background-color 0.08s ease-out;
}
.ai2030-feedback-inline-highlight:hover {
background-color: rgba(255, 200, 60, 0.6);
}
.ai2030-feedback-inline-highlight[data-active="true"] {
background-color: rgba(255, 180, 30, 0.7);
}
/* Hovering a feed card whose quote lives inside a collapsed expandable section
tints that section's summary the same warm highlight color, so the reviewer
can spot which section to open. */
.ai2030-feedback-details-highlight {
background-color: rgba(255, 180, 30, 0.7);
border-radius: 4px;
transition: background-color 0.08s ease-out;
}
index.html
+280
View File
@@ -0,0 +1,280 @@
<!doctype html>
<html lang="en-AU">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>ANZ 2040: the decisions we still get to make</title>
<style>
:root { --bg:#fffff8; --fg:#111; --muted:#666; --rule:#ccc; --accent:#2A623D; }
html { font-size: 17px; }
body { max-width: 42em; margin: 4rem auto; padding: 0 1.25rem;
background: var(--bg); color: var(--fg);
font-family: Georgia, 'Times New Roman', serif; line-height: 1.55; }
h1,h2,h3 { font-weight: normal; line-height: 1.2; margin: 2.2rem 0 0.6rem; }
h1 { font-size: 2rem; } h2 { font-size: 1.5rem; border-bottom: 1px solid var(--rule); padding-bottom: .2rem; }
h3 { font-size: 1.2rem; color: #333; }
a { color: var(--accent); text-decoration: none; } a:hover { text-decoration: underline; }
blockquote { margin: 1rem 0 1rem 1.2rem; padding-left: 1rem; border-left: 3px solid var(--rule);
color: #333; font-size: .95rem; }
table { border-collapse: collapse; margin: 1.2rem 0; font-size: .92rem; }
th,td { text-align: left; padding: .35rem .8rem; border-bottom: 1px solid var(--rule); }
thead th { border-bottom: 2px solid #999; }
code { font-family: ui-monospace, Menlo, Consolas, monospace; font-size: .88em; }
/* the tree is a tall, narrow two-per-row flow; cap its width and centre it so the text
stays a legible size (a wide canvas stretched to the column would shrink the text) */
.tree { margin: 1.5rem auto; max-width: 560px; overflow-x: auto; }
.tree svg { width: 100%; height: auto; display: block; }
hr { border: none; border-top: 1px solid var(--rule); margin: 2.5rem 0; }
footer { margin-top: 3rem; padding-top: 1rem; border-top: 1px solid var(--rule);
color: var(--muted); font-size: .85rem; }
</style>
</head>
<body>
<h1 id="anz-2040-the-decisions-we-still-get-to-make">ANZ 2040: the decisions we still get to make</h1>
<p>What can Australia and New Zealand actually decide if AI becomes much more powerful over the next decade?</p>
<p>We start from the excellent <a href="https://ai-2040.com">AI 2040</a>, which describes choices the United States and China could make. We zoom in on what those choices would mean for Australia and New Zealand, and what we could still do ourselves.</p>
<p>The probabilities below assume rapid AI progress. We leave aside the roughly 35% chance that progress plateaus first.</p>
<p>We don't get to choose whether the United States and China race, slow down, or make a deal. We don't get to choose whether a powerful AI remains under human control. But we do get a few choices about our position before those larger decisions are made. Some are cheap and useful across almost every future. The valuable later choices only exist if we prepare for them now.</p>
<figure class="tree"><svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 560 1272" font-family="Georgia, serif" font-size="14">
<rect width="560" height="1272" fill="#fffff8"/>
<text x="280" y="82" text-anchor="middle" font-size="28" font-weight="bold" fill="#111">Choose a path</text>
<path d="M145,150 C145,180 280,180 280,209" fill="none" stroke="#a9a99f" stroke-width="13.0"/>
<path d="M426,150 C426,180 280,180 280,209" fill="none" stroke="#a9a99f" stroke-width="13.0"/>
<path d="M280,265 C280,294 157,294 157,324" fill="none" stroke="#a9a99f" stroke-width="2.6"/>
<path d="M280,265 C280,294 381,294 381,324" fill="none" stroke="#a9a99f" stroke-width="23.4"/>
<path d="M381,380 C381,410 179,410 179,439" fill="none" stroke="#a9a99f" stroke-width="13.0"/>
<path d="M381,380 C381,410 377,410 377,439" fill="none" stroke="#a9a99f" stroke-width="10.4"/>
<path d="M377,495 C377,524 146,524 146,554" fill="none" stroke="#a9a99f" stroke-width="6.2"/>
<path d="M377,495 C377,524 406,524 406,554" fill="none" stroke="#a9a99f" stroke-width="1.3"/>
<path d="M377,495 C377,582 161,582 161,669" fill="none" stroke="#a9a99f" stroke-width="2.9"/>
<path d="M406,610 C406,640 410,640 410,669" fill="none" stroke="#a9a99f" stroke-width="1.3"/>
<path d="M161,725 C161,754 403,754 403,784" fill="none" stroke="#a9a99f" stroke-width="2.1"/>
<path d="M161,725 C161,754 168,754 168,784" fill="none" stroke="#a9a99f" stroke-width="1.0" stroke-dasharray="5 5"/>
<path d="M403,840 C403,870 410,870 410,899" fill="none" stroke="#a9a99f" stroke-width="1.0"/>
<path d="M403,840 C403,927 377,927 377,1014" fill="none" stroke="#a9a99f" stroke-width="1.0"/>
<path d="M168,840 C168,870 165,870 165,899" fill="none" stroke="#a9a99f" stroke-width="1.0"/>
<path d="M168,840 C168,927 161,927 161,1014" fill="none" stroke="#a9a99f" stroke-width="1.0"/>
<path d="M377,1070 C377,1100 179,1100 179,1129" fill="none" stroke="#a9a99f" stroke-width="1.0"/>
<path d="M377,1070 C377,1100 410,1100 410,1129" fill="none" stroke="#a9a99f" stroke-width="1.0"/>
<rect x="194" y="280" width="50" height="17" fill="#fffff8"/>
<text x="219" y="292" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">stalls</text>
<rect x="289" y="280" width="83" height="17" fill="#fffff8"/>
<text x="330" y="292" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">race / deal</text>
<rect x="235" y="396" width="89" height="17" fill="#fffff8"/>
<text x="280" y="408" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">control lost</text>
<rect x="334" y="396" width="89" height="17" fill="#fffff8"/>
<text x="379" y="408" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">control kept</text>
<rect x="214" y="510" width="96" height="17" fill="#fffff8"/>
<text x="262" y="522" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">seized abroad</text>
<rect x="340" y="510" width="102" height="17" fill="#fffff8"/>
<text x="392" y="522" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">seized at home</text>
<rect x="248" y="533" width="129" height="17" fill="#fffff8"/>
<text x="312" y="545" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">power stays spread</text>
<rect x="354" y="626" width="109" height="17" fill="#fffff8"/>
<text x="408" y="638" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">no public stake</text>
<rect x="234" y="740" width="96" height="17" fill="#fffff8"/>
<text x="282" y="752" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">set our price</text>
<rect x="378" y="856" width="56" height="17" fill="#fffff8"/>
<text x="406" y="868" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">host it</text>
<rect x="347" y="878" width="96" height="17" fill="#fffff8"/>
<text x="395" y="890" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">stay supplier</text>
<rect x="138" y="856" width="56" height="17" fill="#fffff8"/>
<text x="167" y="868" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">concert</text>
<rect x="138" y="878" width="56" height="17" fill="#fffff8"/>
<text x="166" y="890" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">US bloc</text>
<rect x="234" y="1086" width="89" height="17" fill="#fffff8"/>
<text x="278" y="1098" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">public stake</text>
<rect x="362" y="1086" width="63" height="17" fill="#fffff8"/>
<text x="393" y="1098" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">no stake</text>
<a href="#1-prepare-before-it-looks-urgent-2026-28"><title>1. Prepare early — 2026-28: royalties, biosecurity, allies</title>
<rect x="21" y="94" width="248" height="56" rx="4" fill="#fffff8" stroke="#38761d" stroke-width="2.2"/>
<text x="145" y="119" text-anchor="middle" font-weight="bold" font-size="13">1. Prepare early</text>
<text x="145" y="135" text-anchor="middle" fill="#333" font-size="11">2026-28: royalties, biosecurity, allies</text>
</a>
<a href="#2-train-the-people-the-treaty-needs-2026-29"><title>2. Build the people — 2026-29: safety testers, inspectors</title>
<rect x="313" y="94" width="226" height="56" rx="4" fill="#fffff8" stroke="#38761d" stroke-width="2.2"/>
<text x="426" y="119" text-anchor="middle" font-weight="bold" font-size="13">2. Build the people</text>
<text x="426" y="135" text-anchor="middle" fill="#333" font-size="11">2026-29: safety testers, inspectors</text>
</a>
<a href="#what-australia-and-new-zealand-can-do"><title>2029: the US chooses — go-slow deal with China, or race?</title>
<rect x="173" y="209" width="214" height="56" rx="4" fill="#fffff8" stroke="#888888" stroke-width="2.2"/>
<text x="280" y="234" text-anchor="middle" font-weight="bold" font-size="13">2029: the US chooses</text>
<text x="280" y="250" text-anchor="middle" fill="#333" font-size="11">go-slow deal with China, or race?</text>
</a>
<a href="#vii-boring-decade-about-10"><title>VII. Boring decade — ~10%</title>
<rect x="79" y="324" width="157" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="157" y="349" text-anchor="middle" font-weight="bold" font-size="13">VII. Boring decade</text>
<text x="157" y="365" text-anchor="middle" fill="#333" font-size="11">~10%</text>
</a>
<a href="#i-doom-about-50"><title>RISK: AI escapes control — ~75% racing, ~25% in the deal</title>
<rect x="280" y="324" width="201" height="56" rx="4" fill="#fffff8" stroke="#990000" stroke-width="3.6"/>
<text x="381" y="349" text-anchor="middle" font-weight="bold" font-size="13">RISK: AI escapes control</text>
<text x="381" y="365" text-anchor="middle" fill="#333" font-size="11">~75% racing, ~25% in the deal</text>
</a>
<a href="#i-doom-about-50"><title>I. Doom — ~50%</title>
<rect x="104" y="439" width="150" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="179" y="464" text-anchor="middle" font-weight="bold" font-size="13">I. Doom</text>
<text x="179" y="480" text-anchor="middle" fill="#333" font-size="11">~50%</text>
</a>
<a href="#ii-someone-elses-empire-about-25"><title>RISK: power seized — one actor holds the AI</title>
<rect x="298" y="439" width="157" height="56" rx="4" fill="#fffff8" stroke="#990000" stroke-width="3.6"/>
<text x="377" y="464" text-anchor="middle" font-weight="bold" font-size="13">RISK: power seized</text>
<text x="377" y="480" text-anchor="middle" fill="#333" font-size="11">one actor holds the AI</text>
</a>
<a href="#ii-someone-elses-empire-about-25"><title>II. Someone else&#x27;s empire — ~25% — seized abroad</title>
<rect x="42" y="554" width="208" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="146" y="579" text-anchor="middle" font-weight="bold" font-size="13">II. Someone else&#x27;s empire</text>
<text x="146" y="595" text-anchor="middle" fill="#333" font-size="11">~25% — seized abroad</text>
</a>
<a href="#ix-homegrown-autocracy-free-beer-and-rugby-for-life"><title>RISK: power grabbed at home — seized in our own country</title>
<rect x="295" y="554" width="223" height="56" rx="4" fill="#fffff8" stroke="#990000" stroke-width="3.6"/>
<text x="406" y="579" text-anchor="middle" font-weight="bold" font-size="13">RISK: power grabbed at home</text>
<text x="406" y="595" text-anchor="middle" fill="#333" font-size="11">seized in our own country</text>
</a>
<a href="#3-ask-for-terms-before-signing-2029"><title>3. Sign, or set our price? — the deal reaches us, 2029</title>
<rect x="53" y="669" width="216" height="56" rx="4" fill="#fffff8" stroke="#b8860b" stroke-width="2.2"/>
<text x="161" y="694" text-anchor="middle" font-weight="bold" font-size="13">3. Sign, or set our price?</text>
<text x="161" y="710" text-anchor="middle" fill="#333" font-size="11">the deal reaches us, 2029</text>
</a>
<a href="#ix-homegrown-autocracy-free-beer-and-rugby-for-life"><title>IX. Homegrown autocracy — citizens stop being needed</title>
<rect x="313" y="669" width="194" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="410" y="694" text-anchor="middle" font-weight="bold" font-size="13">IX. Homegrown autocracy</text>
<text x="410" y="710" text-anchor="middle" fill="#333" font-size="11">citizens stop being needed</text>
</a>
<a href="#6-choose-a-side-if-the-deal-collapses-2030s"><title>6. If the deal collapses — US bloc, concert, or neutral?</title>
<rect x="68" y="784" width="201" height="56" rx="4" fill="#fffff8" stroke="#b8860b" stroke-width="2.2"/>
<text x="168" y="809" text-anchor="middle" font-weight="bold" font-size="13">6. If the deal collapses</text>
<text x="168" y="825" text-anchor="middle" fill="#333" font-size="11">US bloc, concert, or neutral?</text>
</a>
<a href="#4-host-allied-compute-2030-32"><title>4. Host compute here? — 2030-32</title>
<rect x="313" y="784" width="179" height="56" rx="4" fill="#fffff8" stroke="#b8860b" stroke-width="2.2"/>
<text x="403" y="809" text-anchor="middle" font-weight="bold" font-size="13">4. Host compute here?</text>
<text x="403" y="825" text-anchor="middle" fill="#333" font-size="11">2030-32</text>
</a>
<a href="#viii-middle-power-concert"><title>VIII. Middle-power concert</title>
<rect x="57" y="899" width="216" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="165" y="931" text-anchor="middle" font-weight="bold" font-size="13">VIII. Middle-power concert</text>
</a>
<a href="#v-allied-compute-host"><title>V. Allied compute host</title>
<rect x="317" y="899" width="187" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="410" y="931" text-anchor="middle" font-weight="bold" font-size="13">V. Allied compute host</text>
</a>
<a href="#vi-garrison-ally"><title>VI. Garrison ally</title>
<rect x="86" y="1014" width="150" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="161" y="1046" text-anchor="middle" font-weight="bold" font-size="13">VI. Garrison ally</text>
</a>
<a href="#5-keep-a-public-stake-in-the-machine-economy-2031-34"><title>5. Tax it, own a share? — 2031-34</title>
<rect x="280" y="1014" width="194" height="56" rx="4" fill="#fffff8" stroke="#b8860b" stroke-width="2.2"/>
<text x="377" y="1039" text-anchor="middle" font-weight="bold" font-size="13">5. Tax it, own a share?</text>
<text x="377" y="1055" text-anchor="middle" fill="#333" font-size="11">2031-34</text>
</a>
<a href="#iii-dividend-commonwealth"><title>III. Dividend commonwealth</title>
<rect x="71" y="1129" width="216" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="179" y="1161" text-anchor="middle" font-weight="bold" font-size="13">III. Dividend commonwealth</text>
</a>
<a href="#iv-quarry-economy"><title>IV. Quarry economy — drift default</title>
<rect x="331" y="1129" width="157" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="410" y="1154" text-anchor="middle" font-weight="bold" font-size="13">IV. Quarry economy</text>
<text x="410" y="1170" text-anchor="middle" fill="#333" font-size="11">drift default</text>
</a>
</svg></figure>
<h2 id="what-australia-and-new-zealand-can-do">What Australia and New Zealand can do<sup title="ANZ-specific extrapolations from the cited scenarios">*</sup></h2>
<h3 id="1-prepare-before-it-looks-urgent-2026-28">1. Prepare before it looks urgent (2026-28)</h3>
<p>The later choices need ordinary government machinery: royalty laws, approved land, a way to pay a dividend, biosecurity capacity, and working relationships with other middle powers. None of this requires believing one precise AI forecast. It is cheap compared with discovering in 2029 that the legal and diplomatic work takes five years.</p>
<p>Biosecurity is the clearest case. AI 2040 proposes "massive investments into biosecurity and other measures to improve the resilience of the world," but does not assign the work to anyone. New Zealand already runs a serious agricultural biosecurity system. This is a useful thing for it to get unusually good at.</p>
<h3 id="2-train-the-people-the-treaty-needs-2026-29">2. Train the people the treaty needs (2026-29)</h3>
<p>Australia and New Zealand can contribute people even when they cannot contribute frontier models. AI 2040 allocates only 1% of its safety budget to studying model character, meaning persistent behavioural tendencies or "personas+propensities." It says almost nothing about activation steering, which changes behaviour by intervening on a model's internal activity, or assistance games, which train an AI to infer what a person wants through cooperation. It does call truth-seeking AI and public forecasting a priority. Australia and New Zealand could specialise in those areas, as well as evaluations, treaty verification, biosecurity, and cyber work through the Five Eyes intelligence alliance. (<a href="https://ai-2040.com/supplements/alignment-roadmap">Alignment roadmap</a>, <a href="https://ai-2040.com/supplements/ai-for-epistemics">AI for epistemics</a>)</p>
<h3 id="3-ask-for-terms-before-signing-2029">3. Ask for terms before signing (2029)</h3>
<p>Australia is named among the countries that join the deal. New Zealand is not mentioned in any of the 18 supplements we checked. Once inside, leaving could mean sanctions, cyberattack, or war. The realistic decision is therefore what to ask for before signing: hosted data centres, a strategic chip reserve, verification seats, equity in AI companies, and durable access rather than a revocable API account.</p>
<p>Scott Alexander's summary is unusually blunt. Observer countries follow the rules in return for data centres, AI access, and a share of future wealth, even though the great powers don't strictly need to offer it. That makes accession our best bargaining moment. Signing the standard form gives it away. (<a href="https://www.astralcodexten.com/p/introducing-plan-a">Introducing Plan A</a>)</p>
<h3 id="4-host-allied-compute-2030-32">4. Host allied compute (2030-32)</h3>
<p>AI 2040 moves advanced chip fabs and robot production into special economic zones that can be destroyed if the deal breaks. The zones are meant to sit near an ocean or an adversarial power, and the robot economy expands inside them until it is as large as today's human economy. North-west Australia fits those criteria and sits above much of the ore the machines would consume. (<a href="https://ai-2040.com/supplements/deal-decline">Deal decline</a>, <a href="https://ai-2040.com/supplements/economics-of-plan-a">economics</a>)</p>
<p>Hosting creates exposure as well as income. The same cluster can be a target, collateral for the treaty, and leverage against being cut off. Tom Davidson makes the mechanism explicit: if the US withdraws AI access, allies could destroy US data centres on their territory. A kill switch negotiated at accession is leverage. A foreign cluster accepted without access rights is just exposure. (<a href="https://newsletter.forethought.org/p/how-can-the-middle-powers-avoid-getting">Davidson 2026</a>)</p>
<p>There is also a smaller trusted-vault option. Australia could host cold-storage model weights, treaty verification equipment, or monitoring linked to Pine Gap without hosting the whole robot economy. Geologically stable ground and Five Eyes trust make this the lowest-volume version of the policy.</p>
<h3 id="5-keep-a-public-stake-in-the-machine-economy-2031-34">5. Keep a public stake in the machine economy (2031-34)</h3>
<p>The US dividend in AI 2040 comes from compute permits. By 2035, the scenario sends 75% of the US permit share to US citizens. Australia and New Zealand cannot rely on that. Our dividend would need resource royalties, land taxes, and public equity held through the Future Fund and NZ Super. Those rules need to exist before cognitive labour becomes cheap and wage earners lose their bargaining power. (<a href="https://ai-2040.com/supplements/economics-of-plan-a">Economics of Plan A</a>)</p>
<p>Who owns the rents matters more than the size of the boom. The same ore and land can support a public dividend or a few mining dynasties. Leicht and Ball warn that selling a strategic firm can look like an unexpected windfall while giving up the country's "ticket guaranteeing their home country's stake in the AI economy." In a rapid takeoff, taking only cash is especially bad: the useful payment is equity, access, or ownership of productive assets. (<a href="https://www.thefai.org/posts/the-race-worth-winning-middle-powers-in-the-age-of-machine-intelligence">The Race Worth Winning</a>)</p>
<p>Australia probably starts above the generic non-US country. Compulsory superannuation gives many households an existing claim on global capital, while mining gives governments a claim on inputs the robot economy needs. Neither is evenly shared by default. Superannuation gains follow existing balances and investment choices. Mining income reaches households only when royalties, taxes, public ownership, or transfers capture it. That is what separates the Dividend commonwealth from the Quarry economy.</p>
<p>The political claim is stronger, and more speculative. If governments no longer depend on wages or citizen labour, owners and automated security may stop needing broad consent. A legally entrenched citizen claim on public assets keeps ordinary people tied to the revenue stream and gives them something material to defend through democratic institutions. We are treating distribution as a constraint on domestic power, not merely as welfare.</p>
<h3 id="6-choose-a-side-if-the-deal-collapses-2030s">6. Choose a side if the deal collapses (2030s)</h3>
<p>If the US-China deal declines, Australia's security guarantor and largest customer pull in opposite directions. Joining the US bloc keeps the alliance and loses trade. A concert with Japan, Korea, Canada, the Netherlands, and the UK might bargain for access, but only if it was built before the crisis. Leicht and Ball's warning is that a coalition improvised late will be picked off "one by one." Trying to hedge returns us to someone else's empire by a slower route. Free Chinese open-weight models don't remove the dependency; a supplier earning nothing from us also loses nothing by cutting us off. (<a href="https://www.thefai.org/posts/the-race-worth-winning-middle-powers-in-the-age-of-machine-intelligence">The Race Worth Winning</a>, <a href="https://newsletter.forethought.org/p/how-can-the-middle-powers-avoid-getting">Davidson 2026</a>)</p>
<h2 id="where-this-could-leave-us">Where this could leave us<sup title="ANZ-specific extrapolations from the cited scenarios">*</sup></h2>
<p>Doom and concentrated global power account for most of our probability. Australia and New Zealand only get the later choices if humanity keeps control and no single actor takes it all.</p>
<h3 id="i-doom-about-50">I. Doom, about 50%</h3>
<p>AI 2040's own comparison gives Plan A a 42% chance of a "great future," falling to 10-25% for its alternatives. Combining the table's conditional rows implies misaligned takeover at roughly one quarter under Plan A and roughly three quarters in the modal race. Our 50% is a rough mixture across those possibilities.</p>
<p>Yudkowsky and Soares state the doom case much more strongly than our estimate:</p>
<blockquote>
<p>"If any company or group, anywhere on the planet, builds an artificial superintelligence using anything remotely like current techniques, based on anything remotely like the present understanding of AI, then everyone, everywhere on Earth, will die."</p>
<p>Eliezer Yudkowsky and Nate Soares, <a href="https://www.hachettebookgroup.com/titles/eliezer-yudkowsky/if-anyone-builds-it-everyone-dies/9780316595667/"><em>If Anyone Builds It, Everyone Dies</em></a></p>
</blockquote>
<p>Once control is lost, Australia has no policy response left. The preparations above have to improve the odds before then.</p>
<h3 id="ii-someone-elses-empire-about-25">II. Someone else's empire, about 25%</h3>
<p>The AI remains aligned, but control does not spread. A government, company, or person holds the decisive systems. Australia and New Zealand may be materially rich, especially while iron demand is high, but they live under rules set elsewhere.</p>
<blockquote>
<p>"a tiny group of people, or possibly just a single individual, is effectively in control of the world’s only army of superintelligences"</p>
<p>AI Futures Project, <a href="https://ai-2040.com">AI 2040</a></p>
</blockquote>
<h3 id="iii-dividend-commonwealth">III. Dividend commonwealth</h3>
<p>We negotiated terms, captured resource and land rents, and built the dividend before wages collapsed. Households share in the assets that became valuable instead of watching those assets reprice around them. The better version also uses AI inside courts, health systems, and government rather than treating it only as something to regulate.</p>
<blockquote>
<p>"In Plan A, most of the US permit revenue share (75% in 2035) is redistributed to the US population as a Citizen’s Dividend, resulting in roughly $1M/yr per person in 2035 and around $10M/yr in 2040."</p>
<p>AI Futures Project, <a href="https://ai-2040.com/supplements/economics-of-plan-a">Economics of Plan A</a></p>
</blockquote>
<h3 id="iv-quarry-economy">IV. Quarry economy</h3>
<p>This is the drift outcome. The country exports what the robot economy needs, but has no dividend rail and no public stake. AI 2040's land model makes the distribution problem concrete: single-family homes rise about 40 times while 97% of their paper value becomes land. Incumbent owners gain while cognitive wages fall. The nation is rich and the median household is poor, without anyone needing to plan it.</p>
<blockquote>
<p>"As AI devalues human capital relative to physical and intangible assets, capital income consumes a growing proportion of national income. Elevated spending on luxury goods and a productivity-driven investment boom sustains elevated economic growth even as millions face unemployment and diminished purchasing power."</p>
<p>UK Government Office for Science and AI Security Institute, <a href="https://www.gov.uk/government/publications/ai-scenarios-2030-helping-policymakers-plan-for-the-future-of-ai/ai-scenarios-2030-helping-policymakers-plan-for-the-future-of-ai">AI Scenarios 2030</a></p>
</blockquote>
<p>The resource boom also expires. On current reserves and extraction, iron is the shortest-dated major reserve at roughly 55 years. A machine economy growing every 6-12 months would bring exhaustion and substitution forward. AI 2040 models the demand for energy and land, but not the Australian terms of trade or the cost of scaling mines.</p>
<h3 id="v-allied-compute-host">V. Allied compute host</h3>
<p>Australia accepts the data centres and robot zones, but only after securing access and ownership terms. Land, power, and Five Eyes trust become a stake in the machine economy. The price is physical exposure: the hosted clusters are treaty collateral, and the treaty's wars become ours.</p>
<blockquote>
<p>"If the US withdraws AI access, allies could destroy US data centres in response. It's a way to lock in the deal."</p>
<p>Tom Davidson, <a href="https://newsletter.forethought.org/p/how-can-the-middle-powers-avoid-getting">How can the middle powers avoid getting trounced?</a></p>
</blockquote>
<h3 id="vi-garrison-ally">VI. Garrison ally</h3>
<p>The deal collapses and Australia chooses the US bloc. We keep the security relationship and lose much of the Chinese market. Iron becomes a strategic allocation rather than an ordinary export. This future is poorer and tenser than the dividend or hosting alternatives, but Australia still has a seat.</p>
<blockquote>
<p>"middle powers should help the US, and make sure they are rewarded with continued access to frontier AI and new technologies (including military tech)"</p>
<p>"The only alternative that makes sense to me is siding with China."</p>
<p>Tom Davidson, <a href="https://newsletter.forethought.org/p/how-can-the-middle-powers-avoid-getting">How can the middle powers avoid getting trounced?</a></p>
</blockquote>
<h3 id="vii-boring-decade-about-10">VII. Boring decade, about 10%</h3>
<p>AI stalls or governments shut the frontier down. The consents are unused, the royalty argument was early, and the extra biosecurity capacity looks like insurance that did not pay. This is the cheapest way for our preparations to be wrong.</p>
<blockquote>
<p>"AI progress slows, and AI causes less disruption than expected."</p>
<p>UK Government Office for Science and AI Security Institute, <a href="https://www.gov.uk/government/publications/ai-scenarios-2030-helping-policymakers-plan-for-the-future-of-ai/ai-scenarios-2030-helping-policymakers-plan-for-the-future-of-ai">AI Scenarios 2030</a></p>
</blockquote>
<h3 id="viii-middle-power-concert">VIII. Middle-power concert</h3>
<p>The deal declines, but a coalition formed before the crisis holds together. Australia and New Zealand pool their supply-chain and hosting leverage with Japan, Korea, Canada, the Netherlands, and the UK. None could demand durable access alone. Together they may get terms and help write the rules.</p>
<blockquote>
<p>"Coordination, done right, reassures partners that they can focus on their comparative advantage instead of pursuing full autarky; and it enables powers to see eye to eye with AI exporters instead of being picked off by great powers offering rewards for defection one by one."</p>
<p>Leicht and Ball, <a href="https://www.thefai.org/posts/the-race-worth-winning-middle-powers-in-the-age-of-machine-intelligence">The Race Worth Winning</a></p>
</blockquote>
<h3 id="ix-homegrown-autocracy-free-beer-and-rugby-for-life">IX. <del>Homegrown autocracy</del> free beer and rugby for life</h3>
<p>This is the domestic version of the global power-grab risk. Cognitive wages collapse, public ownership never develops, and automated security removes the state's remaining practical dependence on citizens. No coup is required. Political leverage drains away with economic leverage.</p>
<blockquote>
<p>Luckily we completely dodged a coup and now have Benevolent Dictator for Life (BDFL) Gina Palmer to lead us through. Everyone is very happy, especially since the free beer, rugby, and netball are a great respite from our quickly shrinking population and ever more confusing global situation. - Joe Citizen, 2041</p>
</blockquote>
<blockquote>
<p>"AI could create a similar effect: if governments can generate massive revenue from taxing AI projects rather than citizens, heads of state may lose their economic incentive to ensure citizens prosper. This would weaken citizens’ power to resist coup and backsliding attempts."</p>
<p>"By replacing government employees with loyal AI systems, a head of state could remove important checks on their power."</p>
<p>Davidson, Finnveden, and Hadshar, <a href="https://www.forethought.org/research/ai-enabled-coups-how-a-small-group-could-use-ai-to-seize-power">AI-Enabled Coups</a></p>
</blockquote>
<h2 id="where-we-depart-from-ai-2040">Where we depart from AI 2040</h2>
<p>AI 2040 models energy consumption rising about eight times by 2040 and notes that proven mineral reserves cover decades of current extraction, but it does not price the tonnage needed for a robot economy growing every 6-12 months. Coal, uranium, energy trade, and Australian mine capacity are outside its model. (<a href="https://ai-2040.com/supplements/economics-of-plan-a">Economics of Plan A</a>)</p>
<p>We use its land estimates with the authors' warning that this is a "very rough model" they do not necessarily endorse. We also give model character and activation steering more weight than their roadmap does.</p>
<p>AI 2040 does share some US wealth abroad, but very unevenly. In 2032 its American dividend starts at $45,000 while adults outside the US and China average $1,200. By 2035 the figures are about $1M and $10,000. Australia may do better than that generic non-US payment through superannuation and mining, but only if the gains are broadly owned. A rich national balance sheet can still leave the median household without economic leverage. (<a href="https://ai-2040.com/?choices=plan-a-root">AI 2040</a>)</p>
<p>We leave out a trans-Tasman split and a separate Australian decision during US sabotage operations because neither changes the main choices above.</p>
<h2 id="sources-and-thanks">Sources and thanks</h2>
<p>This is built from the AI Futures Project's <a href="https://ai-2040.com">AI 2040</a> scenario and supplements. Their caveat applies to our use of the numbers too: the economic model "plausibly contains bugs" and they do not trust its outputs to be accurate. The local source mirrors retain exact line references for auditing.</p>
<p>It also benefited from a June 2026 scoping discussion with authors of related scenarios and Australian policy participants. Those workshop notes are private and are not included in the public source archive.</p>
<p><small><sup>*</sup> The Australian and New Zealand decisions and futures are extrapolations from the cited work. A citation means we used that source, not that its authors endorse this scenario.</small></p>
<p>The middle-power framing also draws on <a href="https://ai-2027.com">AI 2027</a>, <a href="https://europe2031.ai">Europe 2031</a>, Tom Davidson's <a href="https://newsletter.forethought.org/p/how-can-the-middle-powers-avoid-getting">middle-power plan</a>, Leicht and Ball's <a href="https://www.thefai.org/posts/the-race-worth-winning-middle-powers-in-the-age-of-machine-intelligence">The Race Worth Winning</a>, and Scott Alexander's <a href="https://www.astralcodexten.com/p/introducing-plan-a">Introducing Plan A</a>. Australian groundwork came from the <a href="https://e61.in">e61 Institute</a>, <a href="https://techpolicy.au/aiagency">Tech Policy Design Institute</a>, <a href="https://www.aspistrategist.org.au/data-centres-are-australias-chance-to-shape-ais-future/">ASPI</a>, and <a href="https://www.katechaney.com.au/making_technology_safe">Kate Chaney MP</a>.</p>
<footer>Built by claude (fable 5) and wassname from the
<a href="https://ai-2040.com">AI 2040 / Plan A</a> supplements. Source and mirrors in the
<a href="sources/">repository</a>.</footer>
</body>
</html>
justfile
+17
View File
@@ -0,0 +1,17 @@
# ANZ 2040 site build. `just` with no target builds everything.
default: build
# build the whole site (tree first because HTML inlines it)
build: tree html
# index.html from anz-2040.md: prose via pandoc, with tree.svg inlined
html:
python3 scripts/build_html.py
# tree.svg: the probability-sized decision tree, from data/tree.json
tree:
python3 scripts/build_tree.py
# serve locally so relative links work (localhost:8080)
serve: build
uvx python -m http.server 8080
scripts/build_html.py
+76
View File
@@ -0,0 +1,76 @@
#!/usr/bin/env python3
"""Build index.html from anz-2040.md. One static file, no build step on GitHub's side
and no runtime JavaScript. Prose goes through pandoc (gfm, so it matches GitHub's own
rendering and keeps quotes verbatim); the linked tree.svg image is replaced by the same
SVG inlined into the page, so its <a> links to
each section and <title> tooltips work. CSS is hand-rolled tufte, borrowing ai-2040's
palette (global.css is a Tailwind *source* file and can't be served as-is).
-- claude (opus) + wassname, 2026-07"""
import re, subprocess, pathlib
ROOT = pathlib.Path(__file__).resolve().parent.parent
md = (ROOT / "anz-2040.md").read_text()
# Claude: Strip internal credit and local-evidence notes from the public page.
md = re.sub(r"<!--.*?-->", "", md, flags=re.DOTALL)
# Claude: Inline the linked SVG so its links work in the standalone page.
tree_link = "[![ANZ 2040: choices and possible futures](tree.svg)](tree.svg)"
assert md.count(tree_link) == 1, f"expected exactly one tree link, found {md.count(tree_link)}"
md = md.replace(tree_link, "@@TREE@@")
body = subprocess.run(
["pandoc", "-f", "gfm", "-t", "html", "--wrap=none"],
input=md, capture_output=True, text=True, check=True,
).stdout
svg = (ROOT / "tree.svg").read_text()
# Claude: Crash when an SVG link no longer matches a prose heading.
heading_ids = set(re.findall(r'<h[23] id="([^"]+)"', body))
svg_targets = set(re.findall(r'href="#([^"]+)"', svg))
missing_targets = svg_targets - heading_ids
assert not missing_targets, f"tree.svg links to missing headings: {sorted(missing_targets)}"
body = body.replace("<p>@@TREE@@</p>", f'<figure class="tree">{svg}</figure>')
CSS = """
:root { --bg:#fffff8; --fg:#111; --muted:#666; --rule:#ccc; --accent:#2A623D; }
html { font-size: 17px; }
body { max-width: 42em; margin: 4rem auto; padding: 0 1.25rem;
background: var(--bg); color: var(--fg);
font-family: Georgia, 'Times New Roman', serif; line-height: 1.55; }
h1,h2,h3 { font-weight: normal; line-height: 1.2; margin: 2.2rem 0 0.6rem; }
h1 { font-size: 2rem; } h2 { font-size: 1.5rem; border-bottom: 1px solid var(--rule); padding-bottom: .2rem; }
h3 { font-size: 1.2rem; color: #333; }
a { color: var(--accent); text-decoration: none; } a:hover { text-decoration: underline; }
blockquote { margin: 1rem 0 1rem 1.2rem; padding-left: 1rem; border-left: 3px solid var(--rule);
color: #333; font-size: .95rem; }
table { border-collapse: collapse; margin: 1.2rem 0; font-size: .92rem; }
th,td { text-align: left; padding: .35rem .8rem; border-bottom: 1px solid var(--rule); }
thead th { border-bottom: 2px solid #999; }
code { font-family: ui-monospace, Menlo, Consolas, monospace; font-size: .88em; }
/* the tree is a tall, narrow two-per-row flow; cap its width and centre it so the text
stays a legible size (a wide canvas stretched to the column would shrink the text) */
.tree { margin: 1.5rem auto; max-width: 560px; overflow-x: auto; }
.tree svg { width: 100%; height: auto; display: block; }
hr { border: none; border-top: 1px solid var(--rule); margin: 2.5rem 0; }
footer { margin-top: 3rem; padding-top: 1rem; border-top: 1px solid var(--rule);
color: var(--muted); font-size: .85rem; }
"""
HTML = f"""<!doctype html>
<html lang="en-AU">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>ANZ 2040: the decisions we still get to make</title>
<style>{CSS}</style>
</head>
<body>
{body}
<footer>Built by claude (fable 5) and wassname from the
<a href="https://ai-2040.com">AI 2040 / Plan A</a> supplements. Source and mirrors in the
<a href="sources/">repository</a>.</footer>
</body>
</html>
"""
(ROOT / "index.html").write_text(HTML)
print(f"wrote index.html ({len(HTML)} bytes), inlined tree.svg ({len(svg)} chars)")
scripts/build_tree.py
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env python3
"""Build tree.svg from data/tree.json. No dependencies beyond stdlib, no JS in the
output. Layout is automatic from each item's `tier` (row) and `order` (left-to-right),
so there are no hand-tuned coordinates to drift. Box width fits its title and subtitle;
probability mass is shown by edge width and explicit percentage labels. Every item is an <a> into its prose section and carries a
<title> tooltip. The viewBox makes it scale to any screen width (phones included).
-- claude (opus) + wassname, 2026-07"""
import json, math, html, pathlib
ROOT = pathlib.Path(__file__).resolve().parent.parent
d = json.loads((ROOT / "data" / "tree.json").read_text())
KINDS = d["kinds"]
CV = d["canvas"]
W = CV["w"]
FLOOR, PSCALE, NH, GAP = CV["floor_w"], CV["p_scale"], CV["node_h"], CV["gap"]
CHOICE_W, ROW_H, TOP = CV["choice_w"], CV["row_h"], CV["top"]
TITLE = d.get("title", "")
TITLE_H = 82 if TITLE else 0 # hero heading band at the top (ai-2040 "Choose a Path" style)
def width(n):
# box fits its text (probability is carried honestly by the flow-weighted edges and
# the explicit % labels, not by node size, which can't be honest at variable text length)
fit = max(len(n["title"]) * 7.3, len(n.get("sub", "")) * 5.7) + 26
return max(fit, CHOICE_W)
# ---- layout: place each tier's nodes left-to-right, centred on the canvas ----
tiers = {}
for n in d["nodes"]:
tiers.setdefault(n["tier"], []).append(n)
pos = {} # id -> (cx, cy, w)
for t, row in tiers.items():
row.sort(key=lambda n: n["order"])
widths = [width(n) for n in row]
total = sum(widths) + GAP * (len(row) - 1)
x = (W - total) / 2
cy = TOP + TITLE_H + t * ROW_H
for n, w in zip(row, widths):
pos[n["id"]] = (x + w / 2, cy, w)
x += w + GAP
H = TOP + TITLE_H + (max(tiers) + 1) * ROW_H
out = [f'<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 {W} {H:.0f}" '
f'font-family="Georgia, serif" font-size="14">',
f'<rect width="{W}" height="{H:.0f}" fill="#fffff8"/>']
if TITLE:
out.append(f'<text x="{W/2:.0f}" y="{TOP+42:.0f}" text-anchor="middle" font-size="28" '
f'font-weight="bold" fill="#111">{html.escape(TITLE)}</text>')
def edge_pts(a, b):
ax, ay, aw = pos[a]; bx, by, bw = pos[b]
if by > ay: # child below: leave bottom, enter top
return ax, ay + NH / 2, bx, by - NH / 2
if by < ay: # child above (back-edge): leave top, enter bottom
return ax, ay - NH / 2, bx, by + NH / 2
# same row: side to side
return (ax + aw / 2, ay, bx - bw / 2, by) if bx > ax else (ax - aw / 2, ay, bx + bw / 2, by)
FLOWSCALE = 26 # edge thickness = flow (probability mass through the edge) * this
edge_labels = []
for e in d["edges"]:
x1, y1, x2, y2 = edge_pts(e["from"], e["to"])
my = (y1 + y2) / 2
sw = max(1.0, e.get("flow", 0.02) * FLOWSCALE) # thick where the mass pours, thin in the trickle
dash = ' stroke-dasharray="5 5"' if e.get("dashed") else ""
out.append(f'<path d="M{x1:.0f},{y1:.0f} C{x1:.0f},{my:.0f} {x2:.0f},{my:.0f} {x2:.0f},{y2:.0f}" '
f'fill="none" stroke="#a9a99f" stroke-width="{sw:.1f}"{dash}/>')
if e.get("label") and not e.get("dashed"): # dashed (secondary) edges stay unlabelled
# place the label near the source (in the gap just below the parent) so labels on
# tier-skipping edges don't land on top of the boxes in the row they cross
# adjacent edges: label at the midpoint (siblings have spread apart there, so they
# don't overlap). tier-skipping edges: push the label near the source, into the gap
# below the parent, so it doesn't land on the boxes in the row it crosses.
lf = 0.30 if abs(y2 - y1) > ROW_H else 0.5
edge_labels.append((x1 + lf * (x2 - x1), y1 + lf * (y2 - y1) - 2, e["label"]))
# draw labels last, each on a background halo so no stroke cuts through the text
for lx, ly, lab in edge_labels:
lw = len(lab) * 6.6 + 10
out.append(f'<rect x="{lx-lw/2:.0f}" y="{ly-12:.0f}" width="{lw:.0f}" height="17" fill="#fffff8"/>')
out.append(f'<text x="{lx:.0f}" y="{ly:.0f}" text-anchor="middle" fill="#555" font-size="12.5" '
f'font-style="italic">{html.escape(lab)}</text>')
for n in d["nodes"]:
cx, cy, w = pos[n["id"]]
k = KINDS[n["kind"]]
x, y = cx - w / 2, cy - NH / 2
rx = 22 if n["kind"] == "outcome" else 4 # pill endings, square choices/risks
sw = 3.6 if n["kind"] == "hazard" else 2.2 # thicker borders so the category colour reads; heaviest on risks
tip = html.escape(f'{n["title"]} — {n["sub"]}' if n.get("sub") else n["title"])
out.append(f'<a href="{n["href"]}"><title>{tip}</title>')
# ai-2040 style: box fill matches the page background so only the category-coloured
# border shows (pure white would read as a patch on the cream page)
out.append(f'<rect x="{x:.0f}" y="{y:.0f}" width="{w:.0f}" height="{NH}" rx="{rx}" '
f'fill="#fffff8" stroke="{k["stroke"]}" stroke-width="{sw}"/>')
ty = cy + (4 if not n.get("sub") else -3)
out.append(f'<text x="{cx:.0f}" y="{ty:.0f}" text-anchor="middle" font-weight="bold" '
f'font-size="13">{html.escape(n["title"])}</text>')
if n.get("sub"):
out.append(f'<text x="{cx:.0f}" y="{cy+13:.0f}" text-anchor="middle" fill="#333" '
f'font-size="11">{html.escape(n["sub"])}</text>')
out.append('</a>')
out.append('</svg>')
(ROOT / "tree.svg").write_text("\n".join(out))
print(f"wrote tree.svg: {len(d['nodes'])} nodes, {len(d['edges'])} edges, canvas {W}x{H:.0f}")
sources/ai2040-pages/ai-for-epistemics.md
+290
View File
@@ -0,0 +1,290 @@
---
url: https://ai-2040.com/supplements/ai-for-epistemics
---
## [AI 2040](/)
≡[PDF](https://ghjyjqzwz4.ufs.sh/f/9qHa0cBclQ7sf6FVzqbp2gHFXC1IbGS5ZA7DmNBusfM4YJRP)
[Supplements](/supplements)
Plan A Basics
[FAQFAQ](/supplements/faq)[How Plan A solves our 5 biggest problemsHow Plan A solves our 5 biggest problems](/supplements/how-plan-a-solves-our-5-biggest-problems)
Plan A Policies
[Covert AI ProjectsCovert AI Projects](/supplements/covert-ai-projects)[Verification PlanVerification Plan](/supplements/verification-plan)[Transparency PlanTransparency Plan](/supplements/transparency-plan)[Capability Scaling StrategyCapability Scaling Strategy](/supplements/capability-scaling-strategy)[Security in Plan ASecurity in Plan A](/supplements/security-in-plan-a)
Analysis of Plan A Policies
[Plan A AssumptionsPlan A Assumptions](/supplements/plan-a-assumptions)[Comparing Possible PlansComparing Possible Plans](/supplements/comparing-possible-plans)[Deal DeclineDeal Decline](/supplements/deal-decline)[Takeoff SupplementTakeoff Supplement](/supplements/takeoff-supplement)
Economics
[Economics of Plan AEconomics of Plan A](/supplements/economics-of-plan-a)[Compute in Plan ACompute in Plan A](/supplements/compute-supplement)[Economic Growth ExplorerEconomic Growth Explorer](/supplements/econ-explorer)
Speculative Analysis
[Space Governance PlanSpace Governance Plan](/supplements/space-governance-plan)[AI for EpistemicsAI for Epistemics](/supplements/ai-for-epistemics)[Alignment RoadmapAlignment Roadmap](/supplements/alignment-roadmap)
[About](/about)
[![AI Futures Project](/ai-futures-logo.svg)](https://ai-futures.org/)
# AI for Epistemics
### Eli Lifland
### Summary
Humanity’s _epistemics_ , by which I mean our ability to come to true beliefs and thus make sensible decisions, is crucially important for achieving a great future. As AI capabilities improve throughout Plan A, they shape increasingly more of every facet of life, and epistemics is no different. I expect how AIs affect epistemics to be highly important, and it could easily be positive or negative.
Basin of sanity: a self-reinforcing equilibrium during AI takeoffVICIOUS LOOPpoor epistemics→ sycophantic or agenda-pushing AIs adopted→ poor epistemics deepenVIRTUOUS LOOPgood epistemics→ epistemic AI tools adopted→ epistemics improve furtherSociety's epistemicsentering AI takeoffBASIN OFCAPTUREBASIN OFSANITY← worse societal epistemicsbetter societal epistemics →
I expect there to be positive feedback loops around AI for epistemics, and thus our goal should be to reach the _basin of sanity_. If we are in this basin, it’s self-reinforcing: society is sane enough to make itself more sane as AIs continue to improve. A top priority of the US government during AI takeoff should be to stay in this basin.
How AI for epistemics interventions flow through to impact
Intervention categories
Collective epistemics
Individual epistemics
Encourage adoption
Key indicators
Org culture
(AGI projects, govt)
Truth-seeking AI
(epistemic virtue)
Transparency to the public
Labor on key decisions
Public discourse quality
Areas of impact
Government decision-making
AGI project decision-making
Public influence on govt & AGI projects
Public investment into making AGI go well
I think that the most important areas of impact of AI to epistemics include the decision-making of the government and AGI projects and the public influence over these actors ([more](/supplements/ai-for-epistemics#areas-of-impact-of-ai-for-epistemics)). In Plan A, an especially important application of AI for epistemics is [preventing deal decline](/supplements/deal-decline#preventing-deal-decline), i.e. improving the stability and effectiveness of the slowdown deal.
[Key indicators](/supplements/ai-for-epistemics#indicators-to-track-societys-epistemics) to track society’s epistemics include:
* Organizational culture of frontier AI projects and governments.
* Truth-seeking AI (or more broadly, AIs’ epistemic virtue).
* How transparent each AI project and government is to the public.
* Human and AI labor spent on key decisions.
* Public discourse quality.
I suggest the following as interventions to positively shape AI’s epistemic impact. ([more](/supplements/ai-for-epistemics#interventions))
AI for epistemics interventions
Collective epistemics
Help society converge toward truth
Epistemic virtue evals
Measure & incentivize AI truth-seeking
Allow people to prove they're telling the truth
Privacy-preserving auditing and automated lie detection
AI in social & traditional media
Community notes, counter-arguments, curated feeds
Track records
Score everyone's promises and predictions; surface in context
Shared epistemic infrastructure
Shared knowledge bases, prediction tracking, argument trees
Persuasion defenses
Limit AI persuasion capabilities; detect & flag attempts
Individual epistemics
Help individuals figure out the truth
Research assistants & forecasters
Calibrated, truth-seeking; not sycophantic
Scenario planning & exploration
On-tap deep planning; gamified what-if simulators
Navigate emotional blockers
AIs that help enable good reasoning despite emotional factors
### Background on AI for epistemics and the “basin of sanity”
How well the development of superintelligence goes for humanity is heavily influenced by the extent to which we are able to come to make reasonable decisions regarding how to navigate the AI takeoff period. In Plan A, we are already in a better position than the default world simply by virtue of having more time to orient ourselves to highly capable AIs. But still, humanity’s _epistemics_ , by which we mean our ability to come to true beliefs and thus make sensible decisions, is crucially important for achieving a great future.
As AI capabilities improve throughout Plan A, they shape increasingly more of every facet of life, and epistemics is no different. I expect AI’s effect on epistemics to be highly important, potentially to a similar or even greater level as the automation of technical AI safety research.
I think that highly capable AIs could easily end up either improving or worsening epistemics, on the whole. It’s plausible, given AIs’ capabilities, that they will end up pushing epistemics to either a positive or negative extreme greater than that we’ve observed throughout history thus far.
Furthermore, I expect there to be positive feedback loops that maintain or push further toward the positive and negative extremes. For example, if societal epistemics are good and people have true beliefs, it’s more likely that AI tools which improve epistemics are recognized as such and adopted. On the other hand, if societal epistemics are bad and people are in extreme filter bubbles where AIs tell them exactly what they want to hear, they may continue to adopt sycophantic AI tools as AIs get more capable. The presence of these feedback loops increases the urgency of steering AIs’ effect on epistemics.
Because of these feedback loops, one way to think about achieving the goal of navigating AI takeoff with good epistemics is that we want to be in the _basin of sanity_. If we are in this basin, it’s self-reinforcing: society is sane enough to make itself more sane as AIs continue to improve. A top priority of the US government during AI takeoff should be to stay in this basin.
### Areas of impact of AI for epistemics
I expect sanity in different areas to be correlated, at least in the long run, due to having similar drivers such as “do frontier AIs have and promote good epistemics” (more on this below). Still, it’s useful to discuss different concrete impacts of epistemics, especially early in takeoff when there may be differences in tooling and adoption in different areas. I think that the most important areas to pay attention to epistemics’ impact are:
1. **Government decision-making,** including to what extent government is captured by AGI projects.
2. **AGI project decision-making**
3. **Public influence on governments and AGI projects.** Including via voting.
4. **Public investment into directly making AGI go well, e.g. via preventing vs. creating large-scale harms or steering toward a great future conditional on avoiding catastrophe.** For example, how much philanthropic investment there is in biodefense and AI safety, vs. how many ideological fanatics there are attempting terrorism. Or how much investment there is into ethical reflection.
For all of the above, importance is greatly increased for decisions/beliefs on AGI-relevant topics. And it’s important that for all of them society has the ability to orient and respond to rapid AI progress.
One related distinction is to what extent interventions affect those with below-average epistemics (i.e. _floor-raising_), those with typical epistemics (i.e. _average-raising)_ , or those with above-average epistemics (i.e. _ceiling-raising_).
### Indicators to track society’s epistemics
I think that the following are the most important indicators to keep track of in order to tell whether we are in the basin of sanity, especially for the important areas identified above.
**Organizational culture of frontier AI projects and governments.** How conducive the culture of frontier AI projects and governments are to making good decisions.
**Truth-seeking AI (or more broadly, AIs’ epistemic virtue).** To what extent frontier AIs are trying to figure out the truth and report it, as opposed to other goals such as advancing a hidden agenda or producing sycophantic outputs that look good but aren’t true.
**How transparent each AI project and government is to the public.**
**Human and AI labor spent on key decisions.** On the most important decisions, many of which are made by governments and AGI projects, how much labor goes into making them? Are governments and AGI projects good at allocating labor across decisions? Note that the _quality_ of the labor is also hugely important, which is why organizational culture and truth-seeking AI are key metrics.
**Public discourse quality.** Whether the public discourse becomes more reasonable due to tools like automated fact checking, or less reasonable due to things like increased polarization from persuasive sycophantic AIs, seems highly important. One important portion of this is how difficult it is for actors to lie or manipulate people. In addition things like fact checking, automated lie detection or privacy-preserving ways to prove claims could be important (discussed more below).
### Interventions
Below I non-exhaustively list what seem like some of the most important levers for AIs improving epistemics, many of which governments could encourage. I loosely divide interventions whether they focus more on the collective societal epistemics, individual epistemics, or encouraging adoption, but in practice many interventions help with multiple of these categories.
This section has lots of overlap with this [Forethought work](https://www.forethought.org/research/design-sketches-for-a-more-sensible-world) which sketches out ideas for epistemic and coordination tools; we’ve linked to specific sketches as appropriate.
#### Collective epistemics: help society converge toward the truth
**Create evaluations of AIs’ epistemic virtue and incentivize AGI projects to make their AIs perform well on these evaluations.** (see also [Forethought writeup](https://www.forethought.org/research/design-sketches-collective-epistemics#epistemic-virtue-evals)) The first step to improving AIs’ epistemic virtue is measuring it. For example, one could measure to what extent AIs figure out what’s true rather than giving answers users want to hear, or how calibrated AIs’ quantitative forecasts are. Once the evaluations exist, it’s important that AGI projects actually care about doing well on them. There will be a place for both quantitative, easily runnable evaluations and more open-ended, human-in-the-loop evaluations. In Plan A, we recommend that the government pushes for AGI projects to do well on epistemic evaluations as part of decisions regarding whether they are able to scale their capabilities, similar to how they would use alignment and control evaluations. We also recommend members of the public to create scorecards for AIs’ epistemic virtue and for consumers to take this into account when deciding what AIs to use, as well as for AGI project employees to take these into account when deciding which company to work at.
**Allow for politicians and other public figures to prove that they’re telling the truth, via privacy-preserving auditing and potentially automated lie detection.** Privacy-preserving auditing might look something like: People can ask you questions, and if you’d like, you can have an AI read through your private data, and report back an answer that preserves your privacy as much as possible while still answering the question. That AI wouldn’t write the data to any sort of memory, so the data would still only be owned by you. You could refuse to have the AI answer any given question, but people could then judge you appropriately. We recommend that norms are established such that politicians are expected to submit to privacy-preserving auditing. (See also [“Confidential monitoring and verification” Forethought writeup](https://www.forethought.org/research/design-sketches-defense-favoured-coordination-tech#confidential-monitoring-and-verification).) A more flexible technology would be if AI enabled automated, highly accurate lie detection. This would be dual use, for example it could be used by those in power to purge dissidents. However, our best guess is that it’s still good to develop, especially in the environment of Plan A where the world is set up well to ensure it’s used as a check on those in power. Additionally, it would dramatically increase ability to detect [covert projects](/supplements/covert-ai-projects) and significantly improve [deal stability](/supplements/deal-decline#preventing-deal-decline).
**Use AI in social and traditional media to improve discourse and keep people informed.** AIs will heavily feature in both social and traditional media, whether via new entrants or existing sites/outlets adopting AI. Influencing how AI is integrated into the media is very important. For example, there could be AI-generated [“community notes”](https://en.wikipedia.org/wiki/Community_Notes) in cases where a post is false or misleading (see also [Forethought writeup](https://www.forethought.org/research/design-sketches-collective-epistemics#community-notes-for-everything)). AIs could also automatically generate counter-arguments to each post. AIs could help reduce echo chambers by steering toward interactions between people who would disagree but have a productive conversation. AIs could curate feeds to give people the information that is most important for them to see and show content that most productively challenges their views (see also [Automated deep briefings](https://www.forethought.org/research/design-sketches-angels-on-the-shoulder#deep-briefing) and [Automated OSINT](https://www.forethought.org/research/design-sketches-tools-for-strategic-awareness#automated-osint) Forethought writeups). Curation more broadly could greatly improve how informed people are with respect to politics and the news, for example by highlighting policy proposals that the reader might have a strong opinion on. (See also [Aligned recommender systems](https://www.forethought.org/research/design-sketches-angels-on-the-shoulder#aligned-recommender-systems) and [Personalized leaning systems](https://www.forethought.org/research/design-sketches-angels-on-the-shoulder#personalised-learning-systems) Forethought writeups).
**Collect and score everyone’s track records, including politicians, and surface track records when relevant.** (see also [Forethought writeup](https://www.forethought.org/research/design-sketches-collective-epistemics#reliability-tracking)) AIs will have the capacity to greatly increase the amount of effort put into collecting people’s past statements, and scoring them. For example, an AI could grade how many of their promises a politician has kept, or what percentages of their predictions have come true, or how productive they have been at lawmaking. There will be variation in how objectively statements can be scored; for the less objective ones, hopefully there would be a trusted AI that a broad array of people listen to; but if not, only scoring the more objective statements could still result in positives.
**Shared epistemic infrastructure, for example knowledge bases.** As AIs improve they will be able to maintain vast amounts of epistemic infrastructure. Wikipedia is an obvious reference point, though for AIs the most promising infrastructure may be more structured knowledge bases (at least at first), and the infrastructure might focus more on being parsable by AIs than humans. One could imagine infrastructure for tracking quantitative predictions and how they have fared, argument trees regarding various claims, etc. Building infrastructure that is trusted by a wide range of actors is important, as opposed to fracturing into alternate realities. (See also this post on building a [full epistemic stack](https://www.oliversourbut.net/p/a-full-epistemic-stack). This is related to and would help with [”Provenance tracing” as described by Forethought](https://www.forethought.org/research/design-sketches-collective-epistemics#provenance-tracing).)
**Defend against persuasion via reducing AIs’ persuasion abilities and using AIs to detect it.** Let’s define persuasion as the use of techniques to convince people of beliefs that work roughly as well when a belief is false as when a belief is true. By default, AIs may eventually get very superhuman at persuasion, but it’s plausible that their persuasion capabilities could be reduced without hurting beneficial capabilities too much. Another line of defense is to have AIs detect persuasion and help people avoid being exposed to the strongest forms of it, or at least to be aware of it (see also [”](https://www.forethought.org/research/design-sketches-collective-epistemics#rhetoric-highlighting)[Rhetoric](https://www.forethought.org/research/design-sketches-collective-epistemics#rhetoric-highlighting) [highlighting” Forethought writeup](https://www.forethought.org/research/design-sketches-collective-epistemics#rhetoric-highlighting)). In Plan A, we recommend that the US government makes sure to evaluate, limit, and require defenses against persuasion capabilities in collaboration with the rest of the Consortium.
**Know whether you’re talking to an AI or human in online discussions.** While we think AI can pose a lot of benefit for epistemics, it’s still valuable to know whether you’re interacting with a human or not. This is especially true if there are bots pushing various agendas all over the internet.
#### Individual epistemics: Help individual actors figure out the truth
**Automated research assistants and forecasters.** AIs could enable a broad range of actors to have access to high-quality, labor intensive research and forecasts. It’s important that these be as truth-seeking as possible, rather than sycophantic. Forecasts should be ensured to be calibrated, i.e. if an AI says something has a 10% chance of happening, that it actually happens 10% of the time. (See also [“Ambient superforecasting” Forethought writeup](https://www.forethought.org/research/design-sketches-tools-for-strategic-awareness#ambient-superforecasting).)
**Help people overcome emotional blockers to good reasoning, rather than preying on them.** Of course, much of people’s poor epistemics comes from emotional blockers rather than dispassionate analytical errors or inadequate information. AIs have the potential to greatly help with emotional blockers, as they reach the capabilities of top human therapists but with greater availability, patience, etc. On the other hand, without intervention AIs might confirm and compound people’s existing biases because this would provide a good short-term experience. (See also [Reflection scaffolding](https://www.forethought.org/research/design-sketches-angels-on-the-shoulder#reflection-scaffolding) and [Guardian angels](https://www.forethought.org/research/design-sketches-angels-on-the-shoulder#guardian-angels) Forethought writeups, which are related but more broad.)
**Automated scenario planning and exploration**. AIs could help do in-depth scenario planning such as Plan A and AI 2027 but for any decision, and could help extract quantitative empirical predictions. AIs could also help explore a variety of scenarios via video-game-like simulations in which the human plays an actor, such as the CEO of an AGI company during an intelligence explosion, and the AI simulates everyone else’s actions and how the world evolves. (See also [“Scenaio planning on tap” Forethought writeup](https://www.forethought.org/research/design-sketches-tools-for-strategic-awareness#scenario-planning-on-tap).)
#### Encourage adoption
**Encourage adoption of epistemic tools such as some of the above.** In Plan A, we recommend that the US government adopt epistemic tools quickly, and encourage AGI projects to do so as well.
**Generally encourage adoption of AI.** Encouraging adoption of AI in general seems valuable for society having a better understanding of existing AI capabilities and trends. In Plan A, we recommend that the US government adopt AIs quickly.
#### Other
Some other ideas that we think are promising but haven’t fleshed out as much as the above:
1. All of the defense-favored coordination tech ideas described by Forethought [here](https://www.forethought.org/research/design-sketches-defense-favoured-coordination-tech).
2. Figure out how to overhaul the government for the age of superintelligence. Plausibly the Constitution should be radically amended.
3. Positively shape the relationships between humans and AIs and they become closer; this one seems important but tricky to get right.
### Appendix
#### The promise of working on epistemic tools
_Why it’s valuable to build or encourage epistemic tools given that better AIs may reduce the need for domain-specific tooling:_ I’m not confident regarding the value of building epistemic tools, and it’s plausible that any given tool gets obsoleted by future AIs, but I’m still optimistic about their value. As AIs improve it will become easier to use them for epistemics out of the box, but scaffolding, UX, and human factors may continue to matter a lot. For example, a tool could incorporate best practices of having AIs debate each other then judging the debate, where an AI out of the box might not. Or a tool might have a UX that has been honed over lots of trial and error with humans, that an AI wouldn’t be able to replicate without iteration. Or a tool may have the technical aspects obsoleted, but the company has built up a base of users that trust its brand.
#### Decomposition of issues with decision-making
What are the reasons that people (including me) might make decisions that are bad, as compared to perfect decision-making and my values?
They could have wrong beliefs when they explicitly reason about the current state of the world, or forecasts. Why?
They might not have access to good enough information: e.g. they have only read news biased in a certain direction.
They might not be making sense of their information well enough:
1. Explicit reasoning about local incentives (monetary, social, etc.)
2. Don’t trust/defer to the right news sources and people
3. Emotional blockers
4. Poor reasoning
They could be trying to support a general “memeplex” (social group, culture, ideology) that requires certain types of cutting off good reasoning/noticing things. Memeplexes are helpful for coordination, but can be bad for epistemics.
They might be making poor decisions based on instinct rather than explicit reasoning. One cause of this is local incentives (monetary, social, etc.) that aren’t specifically reasoned about
They could have different values (these aren’t issues with decision-making, but rather ethical disagreements):
1. They don’t care as much about my selfish goals.
2. They care about their selfish goals more than I do.
3. They have different altruistic values.
1.
See [here](https://www.forethought.org/research/ai-impacts-on-epistemics-the-good-the-bad-and-the-ugly#so-what-should-we-expect-to-happen) for more discussion of potential positive and negative feedback loops in AI for epistemics.
2.
See <https://www.longtermwiki.com/organizations/anthropic> for a prototype of an AI-generated knowledge base.
sources/ai2040-pages/alignment-roadmap.md
+1762
View File
File diff suppressed because it is too large Load Diff
sources/ai2040-pages/capability-scaling-strategy.md
+431
View File
@@ -0,0 +1,431 @@
---
url: https://ai-2040.com/supplements/capability-scaling-strategy
---
## [AI 2040](/)
≡[PDF](https://ghjyjqzwz4.ufs.sh/f/9qHa0cBclQ7sf6FVzqbp2gHFXC1IbGS5ZA7DmNBusfM4YJRP)
[Supplements](/supplements)
Plan A Basics
[FAQFAQ](/supplements/faq)[How Plan A solves our 5 biggest problemsHow Plan A solves our 5 biggest problems](/supplements/how-plan-a-solves-our-5-biggest-problems)
Plan A Policies
[Covert AI ProjectsCovert AI Projects](/supplements/covert-ai-projects)[Verification PlanVerification Plan](/supplements/verification-plan)[Transparency PlanTransparency Plan](/supplements/transparency-plan)[Capability Scaling StrategyCapability Scaling Strategy](/supplements/capability-scaling-strategy)[Security in Plan ASecurity in Plan A](/supplements/security-in-plan-a)
Analysis of Plan A Policies
[Plan A AssumptionsPlan A Assumptions](/supplements/plan-a-assumptions)[Comparing Possible PlansComparing Possible Plans](/supplements/comparing-possible-plans)[Deal DeclineDeal Decline](/supplements/deal-decline)[Takeoff SupplementTakeoff Supplement](/supplements/takeoff-supplement)
Economics
[Economics of Plan AEconomics of Plan A](/supplements/economics-of-plan-a)[Compute in Plan ACompute in Plan A](/supplements/compute-supplement)[Economic Growth ExplorerEconomic Growth Explorer](/supplements/econ-explorer)
Speculative Analysis
[Space Governance PlanSpace Governance Plan](/supplements/space-governance-plan)[AI for EpistemicsAI for Epistemics](/supplements/ai-for-epistemics)[Alignment RoadmapAlignment Roadmap](/supplements/alignment-roadmap)
[About](/about)
[![AI Futures Project](/ai-futures-logo.svg)](https://ai-futures.org/)
# Capability Scaling Strategy
### Eli Lifland
# Overview
Plan A vs. Plan D capability trajectoryAutomated Coder3× AI R&D upliftSuperhuman AI Researcher20× AI R&D upliftTop-Expert-Dominating AI400× AI R&D upliftWildly Superintelligent10,000× AI R&D upliftPlan A — capability-shaped upliftsAutomated Coder3× realized upliftTop-Expert-Dominating AI~40× realized upliftMax-controllable AI2040: handofflate 2029mid 2030mid 2030late 2030late 2034early 2031Plan Dno deal — intelligence explosionPlan Aslowdown deal — controlled takeoff20262028203020322034203620382040
In the Plan A scenario, why does the Consortium choose to steadily increase capabilities until reaching top-expert-dominating AI (TED-AI) in 2035, then slow down nearly to a pause until handing off to these AIs in 2040? This supplement describes the high level strategy for scaling capabilities and the reasoning behind them.
This supplement explores the reasoning behind these choices.
In one sentence, the high-level scaling strategy in Plan A is to scale as quickly as possible subject to maintaining high confidence that the scaling is safe. In more detail:
First [I discuss](/supplements/capability-scaling-strategy#high-level-strategy) the high level capability scaling strategy used in Plan A and the reasoning behind it. I split the strategy into 2 stages:
1. **Scale until you’re close to max-controllable-AI** , i.e. the maximally capable AI that we are confident we can prevent from causing a catastrophe even if they were misaligned. We predict in the scenario that the max-controllable-AI is roughly TED-AI.
2. **Slow down** nearly to a halt, then eventually **hand off to AIs**.
Then [I discuss](/supplements/capability-scaling-strategy#costs-of-taking-more-time-to-scale) the costs of taking more time to scale capabilities, most importantly [deal decline](/supplements/deal-decline) risk.
Finally, I discuss in more detail the recommended strategy for [Stage 1](/supplements/capability-scaling-strategy#stage-1-scale-to-max-controllable-ai) and [Stage 2](/supplements/capability-scaling-strategy#stage-2-slow-down-nearly-to-a-halt-then-eventually-hand-off-to-ais).
In practice, much more analysis should be done to decide the best strategies than that which we’ve done for this supplement, and furthermore these strategies should be adjusted as information comes in during the execution of Plan A. I’ll describe my current best guess.
I’m confident that we’ve identified the most important considerations pushing toward scaling slower, and I’m moderately but not highly confident that the high-level strategies we describe are correct. I’m not confident in any precise quantitative estimates.
# High level strategy
## Overall shape of the trajectory
We choose to scale up at all rather than pause indefinitely because of the [costs of going slowly](/supplements/capability-scaling-strategy#costs-of-taking-more-time-to-scale), most importantly [deal decline](/supplements/deal-decline#eli-lifland), i.e. the risk that the slowdown deal dissolves or becomes less effective.
We aim for a trajectory shaped like the following:
Plan A spends more time at higher capability levels than a uniformly slowed intelligence explosionPlan A — capability-shaped upliftsAutomated Coder3× realized upliftTop-Expert-Dominating AI~40× realized uplift2029: two ways to slow down2040: handoffearly 2035mid 2030late 2034Plan Aslowdown deal — controlled takeoffUniform slowdownuniformly slower uncontrolled intelligence explosion20262028203020322034203620382040
We break Plan A into 2 high-level stages:
1. Stage 1: [Scale until max-controllable-AI](/supplements/capability-scaling-strategy#stage-1-scale-until-max-controllable-ai).
2. Stage 2: [Slow down nearly to a halt, then eventually hand off to AIs](/supplements/capability-scaling-strategy#stage-2-slow-down-nearly-to-a-halt-then-eventually-hand-off-to-ais).
This produces a curve that first increases relatively quickly, though still slower than in an uncontrolled intelligence explosion, and then slows down to near a halt for a while. This contrasts with what we’d get if the intelligence explosion were slowed uniformly.
We choose to aim for this sort of curve because we want to spend as much time as possible with AIs that are very useful for important applications, such as alignment, epistemics, and [stabilizing the deal.](/supplements/deal-decline#eli-lifland) And we think that going this fast early is manageable while maintaining high confidence in safety.
## Stage 1: Scale until max-controllable-AI
**During this stage the Consortium scales capabilities until a bit below _[max-controllable-AI](/supplements/alignment-roadmap#appendix-capability-threshold-definitions),_ then ~pauses further scaling.** Max-controllable-AI is the maximum capability level at which we are confident that we maintain _control_ over the AIs, i.e. the property that the AIs couldn’t cause catastrophic harm even if they wanted to. We think that this level will be approximately Top-Expert-Dominating AI (TED-AI), i.e. AIs that are at least as good as top humans at every cognitive task.
**Strategy:** Our best guess is that the Consortium will have a fairly good idea of whether there’s a covert project or not, even though they may not be able to prove it, based on how much compute is unaccounted for. We also predict that there probably won’t be a covert project. **If the Consortium thinks there 's likely a covert project but shares our current best guess regarding covert project detection (~60% chance within 5 years), then it’s a close call as to whether they should go slower than they otherwise would due to covert projects. If they are substantially more pessimistic than our current best guess about detection, then they should slow down in order to avoid making lots of software progress, hence taking at least ~5 years in Stage 1.** **If they are instead more optimistic, then the limiting factors on their speed should instead be maintaining high confidence in control and avoiding destabilization.** The benefits of scaling quickly are (a) the Consortium gets more capable AIs to automate useful work and to use as experimental subjects and (b) lower risk of [deal decline](/supplements/deal-decline).
## Stage 2: Slow down nearly to a halt, then eventually hand off to AIs
**Once the Consortium has AIs that are capable enough such that it is difficult to control them, they slow down (close to a pause) for a while. Then they hand off to the AIs, relaxing control measures.** By relaxing control measures, I mean that we can give AIs more ability to get things done, at the cost of increasing the chance they could take over the world if they were misaligned. For simplicity, in this supplement I discuss and model handoff as binary.
**Strategy: Approximately, the strategy is to wait until the Consortium has high confidence that the AIs are aligned, then hand off.** More precisely, the Consortium should wait until the benefits of delaying to increase the chance that the AIs are aligned are outweighed by various risks: the [deal declining](/supplements/deal-decline), [covert projects](/supplements/covert-ai-projects) reaching dangerous capabilities, or other existential risks such as pandemics; I think that deal declining is a much bigger factor in Stage 2. Roughly, you should hand off once either of the following is true (see more [below](/supplements/capability-scaling-strategy#concrete-case-study-of-possible-2040-handoff-decisions)):
1. You have very high confidence that alignment has been solved, e.g. 99.5%, and thus marginal gains from further research are low (e.g., ~0.1%/year).
2. You have somewhat high confidence that alignment has been solved, e.g. 95%, and thus marginal gains from further research are a little higher (e.g., ~1%/year). However, the Plan A deal seems somewhat unstable. E.g., if there were >5%/year chance of deal decline, that would point toward a handoff.
# Costs of taking more time to scale
The following are the biggest costs of taking longer to increase capabilities in Stage 1, or taking longer to hand off to AIs in Stage 2.
**First, the deal might dissolve or get worse. We call this[deal decline](/supplements/deal-decline),** and it encompasses both of:
**Deal dissolution.** The deal is officially dissolved.
**Deal impairment.** The deal becomes substantially less effective than if it were implemented well and adhered to, but isn’t dissolved. Ways the deal could be impaired:
1. **Deal non-enforcement.** The deal is not enforced as strongly as before, e.g. a country blatantly not complying is not punished. This leads to increased violations of the letter of the deal.
2. **Deal misexecution.** The letter of the deal is followed but is executed poorly.
3. **Deal revision.** The deal is explicitly edited to be worse. This could be straightforward weakening, but it could also be making it worse in other ways, such as unnecessarily increasing surveillance.
See the [deal decline supplement](/supplements/deal-decline) for more info.
**Second, there are risks from illegal model usage in[covert projects](https://www.ai-2040.com/supplements/covert-ai-projects) by state actors or terrorists.** If covert projects possess capabilities that would be catastrophic if in the hands of misaligned AIs or bad actors, AI takeover and misuse risks are higher. For Stage 1, unlike in Stage 2, the Consortium should (within plausible Stage 1 lengths) go _slower_ to beat the covert projects; going too fast requires making lots of software progress which helps the covert projects. More discussion [below](/supplements/capability-scaling-strategy#stage-2-slow-down-nearly-to-a-halt-then-eventually-hand-off-to-ais).
**Finally, there is direct pre-handoff existential risk incurred before handing off to AIs.** This doesn’t include risks from covert projects. This can be split into:
1. Risks from use of legal (i.e. non-covert) model capabilities.
2. Risks not derived from legal model capabilities. For example, asteroid risk or a nuclear war not causally downstream of legal frontier AIs.
**Additionally, there are shifts in the world not driven by the deal, which could be either a cost or benefit of going slowly.** For example, the balance of power could shift so that authoritarian countries have more power relative to democratic countries, or vice versa.
# Stage 1: Scale to max-controllable-AI
## Benefits and costs of going slowly
During Stage 1 our goal is to decide on a policy for how quickly we should scale training compute and make software progress in the run-up to max-controllable-AI. That is, we want to set a trajectory that has a target end date at which max-controllable-AI is reached, and specify the split of training/software progress at each point. We consider only strategies that specify in advance a target end date and then scale throughout all of Stage 1 toward that date, for simplicity and tractability. In practice the Consortium’s strategy should be adjusted over the course of Stage 1 as they get more information.
We also need to decide the [training compute safety tax](/supplements/comparing-possible-plans#training-compute-safety-tax-definition) that we’d like to pay, i.e. the difference between the amount of compute actually used on a training run to reach a given capability level, and the minimum amount of compute required to reach that capability. Paying a higher safety tax means using safer training algorithms at the cost of involving making extra software progress that goes unused but leaks to [covert projects](https://www.ai-2040.com/supplements/covert-ai-projects).
**The primary costs of going slowly in Stage 1 are:**
1. **Those laid out[above](/supplements/capability-scaling-strategy#costs-of-taking-more-time-to-scale);** risk from deal decline due to taking longer, covert projects (if paused for a very long time), and direct existential risks.
2. **Missing out on time with more capable AIs** , which can be used to automate useful work (alignment, control, [epistemics](https://www.ai-2040.com/supplements/ai-for-epistemics), etc.) and be used as experimental subjects. These AIs could also help stabilize the deal, reducing risk of decline per year.
**The primary benefits of going slowly in Stage 1 are:**
1. **A lower fraction of progress comes from software improvements. This is good because software improvements can be copied by[covert projects](https://www.ai-2040.com/supplements/covert-ai-projects).** Software progress and training compute scaling are the two sources of progress. Software progress is mostly made public, meaning it diffuses to [covert projects](https://www.ai-2040.com/supplements/covert-ai-projects). This means that despite covert projects being a reason that you _eventually_ should scale to superintelligence, they are a reason you should scale _slower_ in Stage 1 unless you’re taking well over a decade.
2. **Less risk of AI takeover due to overestimating to what extent the AIs are controlled.**
3. **Less destabilization, which could (among other things) potentially increase the chance of deal decline per year.** An example of a destabilizing effect is if we scaled to AIs capable of superpersuasion and it wasn’t properly defended against, leading to degraded epistemics. However, as noted above, better AIs could also be stabilizing and reduce the annual risk of deal decline.
The Consortium’s strategy is also constrained in that it might not be possible to avoid discovering software improvements, for 2 reasons: (a) some software improvements might be required to utilize increased amounts of compute, (b) doing lots of safety R&D likely inevitably leads to some capabilities externalities in the form of software progress. In our simulations, we require that the Consortium make 0.2 OOM of software progress for each OOM of training FLOP/s they gain, and that during Stage 2, the Consortium’s alignment research incurs capabilities externalities at a rate of 0.25 software OOMs/year.
## What to do if a covert project seems unlikely
Our best guess is that the Consortium will have a fairly good idea of whether there’s a covert project or not, even though they may not be able to prove it, based on how much compute is unaccounted for. We also predict that there likely won’t be a covert project.
**If it seems unlikely that there’s a covert project, my recommendation is to scale as fast as possible while maintaining high confidence in AIs being controlled and avoiding destabilizing society and the deal. This is what happens in our scenario, and we depict the resulting target year of 2035 as a rough guess.** I’d weakly guess that the target year should be a little earlier, were we to rewrite the scenario.
## Target year’s effect on risk probabilities
If it seems likely that there’s a covert project, then the situation is more complicated because we have to balance deal decline risk, which points toward going faster, with covert project risk, which points toward going slower.
I’ll now do some modeling of how the trajectory of the deal will go. [The modeling](https://www.ai-2040.com/supplements/takeoff-supplement) assumes that the deal begins in 2029 and that it starts being implemented a bit under a year before AC, as in the Plan A scenario. It does _not_ assume that takeoff from AC to TED-AI and AC to ASI is about 1 year as in the scenario. Instead it assumes uncertainty with parameter medians centered around those from the [Plan A default trajectory](https://www.ai-2040.com/supplements/takeoff-supplement), as we think decision-makers will still have substantial uncertainty over takeoff speeds in 2029. This means that sometimes the Consortium can’t hit the target date; in fact for the earliest target date we consider of mid-2031, it often can’t hit it, as this would be a takeoff of similar length to the default (the default trajectory goes from pre-deal capabilities to TED-AI in ~1.5 years, while in Plan A the training resumption happens at the start of 2030 and the Consortium aims to pay a 1 OOM [safety tax](/supplements/comparing-possible-plans#training-compute-safety-tax-definition) above TED-AI).
The below graph shows how the risk of deal decline and risk of covert projects achieving max-controllable-AI change based on the Stage 1 target end date. It assumes our best-guess projections as described in the [deal decline](/supplements/deal-decline) and [covert project](/supplements/covert-ai-projects) supplements. It looks at the risk of deal decline and covert projects reaching TED-AI in Stage 1 or within the first 5 years of Stage 2. I choose 5 years because this is the length of Stage 2 in the scenario; more discussion [below](/supplements/capability-scaling-strategy#concrete-case-study-of-possible-2040-handoff-decisions). The latest target date considered is 2037 because this is, with our median takeoff parameters, the year at which TED-AI is reached if the Consortium makes the minimum amount of software progress.
# Deal decline and covert project risk trade off as speed varies
Covert detection?Detection includedNo detectionSplit deal decline & covert by Stage 1 vs. first 5 years of Stage 2Deal declineCovert reaches TED-AIDeal unimpairedAssumes a covert project is attempted. All curves represent world state as of 5 years after end of Stage 1.
0%25%50%75%100%Highest deal unimpaired (45.2%)20312032203320342035203620372038Stage 1 target end date (TED-AI)Share of worlds5y after Consortium TED-AI
We can see the tradeoff between minimizing risks from deal decline, and minimizing risk from covert projects reaching TED-AI. Target years between 2032 and 2036 all look to be quite similar in terms of maximizing the chance of getting to 5 years into Stage 2 unimpaired. If we turn off detection, then any of 2034 to 2036 look to be of fairly similar value, with 2032 and especially 2031 looking much worse; this makes sense as covert projects are a bigger factor.
## Badness of risks
Does it make sense to treat these risks as equal in badness to each other?
I’ve done some rough estimates of how good various end states to the deal are [here](https://docs.google.com/spreadsheets/d/1MVdjISa09Y4qO3jT5eXYFz-s45gQm4NTXcZqEsFSowU/edit?gid=2143764324#gid=2143764324) (summary [here](https://docs.google.com/spreadsheets/d/1MVdjISa09Y4qO3jT5eXYFz-s45gQm4NTXcZqEsFSowU/edit?gid=1504118452#gid=1504118452)): by year and whether the deal stays unimpaired, or whether it declines or a covert project reaches TED-AI. In those estimates, I assumed that the covert project that reached TED-AI (or the AI that it created) had the ability and desire to essentially take over the world.
I’ll now bin outcomes into whether they happen before or after the start of Stage 2 and calculate the badness based on how much worse the outcome is than the deal surviving 5 years into Stage 2. The shift to Stage 2 is good for improving outcomes from both deal decline and covert TED-AIs, because reaching TED-AI allows the discovery of better alignment techniques, generally improving epistemics, and other useful applications.
I’ll also add a discount for the fact that there might not be human or AI takeover that originates from the covert project that reaches TED-AI.
[It seems like](https://docs.google.com/spreadsheets/d/1MVdjISa09Y4qO3jT5eXYFz-s45gQm4NTXcZqEsFSowU/edit?gid=1504118452#gid=1504118452) the percentage of value lost is very roughly:
| Pre-Consortium-TED-AI (Stage 1)| Post-Consortium-TED-AI, first 5 years (Stage 2)
---|---|---
Covert takeover badness| 70%| 55%
Takeover chance| 55%| 45%
**Covert reaches TED-AI badness**| **39%**| **25%**
**Deal declines**| **50%**| **25%**
Below we can see how deal decline and covert project risk are split up across Stage 1 and the first 5 years of Stage 2.
# Deal decline risk is more weighted toward Stage 1 than covert project risk
Covert detection?Detection includedNo detectionSplit deal decline & covert by Stage 1 vs. first 5 years of Stage 2
Deal decline (Stage 1)Deal decline (first 5 years of Stage 2)Covert reaches TED-AI (Stage 1)Covert reaches TED-AI (first 5 years of Stage 2)
Assumes a covert project is attempted. All curves represent world state as of 5 years after end of Stage 1.
0%25%50%75%100%20312032203320342035203620372038Stage 1 target end date (TED-AI)Share of worlds5y after Consortium TED-AI
We see that a higher percentage of deal decline risk comes in Stage 1 than for covert project risk, especially for target years of 2035 and earlier. This makes deal decline on average worse than covert projects reaching TED-AI, relative to what you’d think if you compared their badness from happening at the same time. Now let’s check out the implications of combining the risk levels with the value lost from the risk materializing:
# Expected value lost from deal decline and covert risks, by TED-AI target year
Covert detection?Detection includedNo detection
Value lost if the outcome happens (drag to adjust your own estimates):
Covert reaches TED-AI — Stage 139%
if takeover 70%
Covert reaches TED-AI — first 5y of Stage 225%
if takeover 55%
Also adjust deal-decline value lostReset value lost to defaults
Total value lostFrom deal declineFrom covert projectAssumes a covert project is attempted. All curves represent world state as of 5 years after end of Stage 1.
0%10%20%30%Lowest value lost (21.8%)20312032203320342035203620372038Stage 1 target end date (TED-AI)Expected value lost
My simple methodology has the following 2 issues which means that you should interpret the above graph as being too favorable toward earlier target dates relative to later ones
1. I only consider the percentage of value lost as the outcome variable, but reaching 5 years into Stage 2 has slightly greater value for later target dates than for earlier ones (though the effect size might be small given the use of AIs for safety in the final 5 years swamps that of previous years).
2. The percentage of value lost might be higher for early target years, especially during Stage 1.
With that said: With my best guess parameter values, it looks like earlier is better from the perspective of deal decline and covert project risks. However, it’s not that big of a difference. The expected percentage of value lost is just ~2% higher with a target year of 2036 than of 2032. This means that adjustments for the issues described above could potentially flip the conclusion.
Not taking into account covert project detection results in a curve that is more flat and at higher amounts of value lost; having there be a covert project and not being able to detect it is a rough situation to be in no matter what you do. All of 2031-2036 result in very similar values, but with adjustments later target dates could be favored.
Including detection but assuming that a covert project or its AI takes over once reaching TED-AI results again in the 2031-2036 targets being very close in value.
## Summing up
**If the Consortium thinks there 's likely a covert project but shares our current best guess regarding covert project detection (~60% chance within 5 years), then it’s a close call as to whether they should go slower than they otherwise would due to covert projects. If they are substantially more pessimistic than our current best guess about detection, then they should slow down in order to avoid making lots of software progress, hence taking at least ~5 years in Stage 1.** **If they are instead more optimistic, then the limiting factors on their speed should instead be maintaining high confidence in control and avoiding destabilization.**
All of the above assumed a constant [safety tax](/supplements/comparing-possible-plans#training-compute-safety-tax-definition) of 1 OOM of excess training compute above the minimum required paid at the end of Stage 1. This is my recommendation and the tax portrayed in the scenario. This recommendation is based on a mix of simple modeling and intuitive judgment. I’m not confident in it; the benefit of a higher tax is more confidence in control at higher capability levels, while the downsides are leaking software to covert projects and creating an overhang of foregone software improvements that can be used by projects post-deal-decline (especially post-dissolution).
# Stage 2: Slow down nearly to a halt, then eventually hand off to AIs
Once at max-controllable-AI (TED-AI in the Plan A scenario), we need to decide how long to pause before handing off trust to AIs. For the purposes of this analysis we are treating this stage as a pause, though in practice and in the scenario there are a few deviations from a complete pause: (a) it may be desirable to scale up very slowly, as the max controllable level might increase due to improved control techniques and/or tighter uncertainty bounds, and (b) safety research will have at least minor capabilities externalities, meaning software progress will be made, which will leak to covert projects even if it isn't applied to improve legal AIs.
**The costs of delaying handoff in Stage 2 are as laid out[above](/supplements/capability-scaling-strategy#costs-of-taking-more-time-to-scale); risk from deal decline, covert projects, and direct existential risks.** We think that deal decline will generally be the dominant consideration because covert projects will likely have already been detected, if they existed in the first place.
**The primary benefit of delaying handoff in Stage 2 is increasing the likelihood that the AIs we hand off to are aligned.** This mostly comes from alignment research that helps figure out how to align AIs and evaluate their alignment, and paying “safety taxes” such as training a model with more compute than is required to reach a given capability level, in order to use a safer training process. Note that this is specifically regarding the AI that we hand off to; it’s possible that an aligned AI might not succeed in aligning its successors (especially if circumstances change negatively in some way after handoff).
A secondary benefit is increasing the probability that the future goes very well conditional on alignment; this is also highly important but probably depends less on the timing than does probability of alignment. Additionally, it’s possible that waiting too long eventually ends up being marginally negative for outcomes conditional on alignment; it might be better to hand off to an aligned max-controllable-AI earlier rather than later so that it has more free rein to improve the situation.
To decide when to hand off in Stage 2, we recommend calculating the marginal benefits and costs of delaying handoff and handing off once the costs outweigh the benefits. **We expect the best strategy to be something like handing off once we have high confidence that the AIs are aligned.**
See below for specific concrete case studies on deciding what strategy is best.
## Concrete case study of possible 2040 handoff decisions
We’ll now discuss case studies regarding whether to hand off. To make things concrete, we’ll assume that we are making a decision regarding whether to hand off in 2040 some time after max-controllable-AI was reached, which is when handoff happens in the Plan A scenario.
To keep things simple, we’ll assume that the only factors for whether to delay handoff are:
1. **Benefit: Increased alignment probability.** See [below](/supplements/capability-scaling-strategy#trajectory-of-estimated-alignment-probability) for discussion of what the trajectory of alignment progress will look like.
2. **Cost: Deal decline risk.** Covert project risk is the main other contender for a substantial reason to hand off faster. However, our guess is that covert project risk is probably very low in this case study, because (a) probably if there was a covert project it would have been detected by now and (b) we haven’t done non-safety-externality software progress in 5 years, so covert projects haven’t been getting much help.
Below you can try out different settings for the relevant variables and see what recommendation they lead to regarding whether to delay hand off. I’ve set the default values to roughly match what they are during the handoff in the scenario, and have added plausible presets that set either the alignment or deal dissolution/impairment parameters.
Caveat: this calculator only weighs the single most important factor on each side — the alignment probability gained by delaying vs. the deal decline risk avoided by handing off.
Alignment scenario
High confidence alignment solutionAlignment probably solvedAlignment likely solvedAlignment clearly unsolved
Alignment gain (a_change)0.1%/yr
Extra p(alignment) bought per year of delay.
Alignment probability (a_prob)99.5%
Current p(alignment). Deal dissolution or impairment is more costly the more likely alignment is.
Deal decline scenario
Low riskMedium riskHigh risk
Deal decline probability (d_prob)3%/yr
Yearly chance the deal dissolves or is substantialy impaired without official dissolution.
Deal decline badness (d_loss)20%
Percentage of the future's value lost if the deal dissolves or is impaired.
p(alignment) change by handing off instead of delaying a year
+0.50%
Recommendation: Hand off now (weak)
u_handoff_minus_u_delay = d_prob·a_prob·d_loss − a_change
= (3% × 99.5% × 20%) − 0.1%
Benefit of handing off: 0.60%Cost of handing off: 0.10%
Reset to defaults
Recommendation by scenario
| Low riskd_prob 0.5%/yr · d_loss 10%| Medium riskd_prob 3%/yr · d_loss 20%| High riskd_prob 10%/yr · d_loss 35%
---|---|---|---
High confidence alignment solutiona_prob 99.5% · a_change 0.1%/yr| Delay−0.05%| Hand off+0.50%| Hand off+3.38%
Alignment probably solveda_prob 95% · a_change 1%/yr| Delay−0.95%| Delay−0.43%| Hand off+2.33%
Alignment likely solveda_prob 80% · a_change 3%/yr| Delay−2.96%| Delay−2.52%| Delay−0.20%
Alignment clearly unsolveda_prob 10% · a_change 2%/yr| Delay−2.00%| Delay−1.94%| Delay−1.65%
Should we hand off or delay? \- ai-2040.com
Reasoning for the medium deal decline probabilities and badness estimates can be found in the [Deal Decline supplement](/supplements/deal-decline#likelihood-of-deal-decline).
## Trajectory of estimated alignment probability
How steadily will alignment progress? There are 2 extremes we could use to model the situation.
1. **All or nothing:** We start very confidently believing that the AIs are misaligned, then at one point we make a breakthrough which quickly makes us confident that they’re now aligned.
2. **Smooth, predictable progress:** Our estimate of whether the AIs are aligned increases steadily and predictably.
Our high-uncertainty best guess is that the trajectory of our confidence in AIs’ alignment will be in between, perhaps slightly closer to the _All or nothing_ option. It could also be a combination: we think that progress may look smooth/predictable for a while, until there’s a sudden breakthrough. For example, if ambitious mechanistic interpretability is solved and we understand all of AIs’ reasoning processes, this may give very high confidence in alignment.
For example, if we weren’t yet confident in AIs’ alignment, in 2040 there might be a ~5%/year chance of an alignment breakthrough that makes us confident, in addition to continuous improvements until we get the breakthrough; if we are confident (as in the scenario), then perhaps we’re at 0.01-1%/year alignment probability increase since we’re already close to 100%.
1.
Non-exhaustive examples of catastrophic harm include: directly taking over the world, misaligning successor AIs that could take over the world, damaging the world’s epistemics to a difficult-to-reverse degree (for example via superhuman persuasion).
2.
This could also be the case if they agree with our current best guess about detection but have a much higher estimate of the badness of a covert project reaching TED-AI.
3.
Scaling too fast via building compute used for training could hurt the situation in the case where the deal declines, but the compute isn’t destroyed, as it allows for a faster takeoff. In the case the deal dissolves, the compute would probably be destroyed, but for partial deal impairment, this wouldn’t be the case. However, we think that this downside is probably outweighed by the upside you get from the benefits of scaling to higher capabilities within the deal rather than outside of it or in an impaired deal.
4.
In practice we recommend for it to be somewhat gradual. Example of a gradual handoff: first hand off a bit such that the AIs could maybe take over if they were misaligned, i.e. we’re not as confident as before; then hand off more such that they could only take over if they coordinated but couldn’t if they did not, etc.
5.
The deal could also get better, but we’re not including that here as our best guess is that the deal will in expectation get worse. This is definitionally true for deal dissolution, but we’re not confident for deal impairment; if we were more optimistic, we could remove the deal impairment term or explicitly model the deal getting better.
6.
Including any ways it derives from the relevant capabilities, e.g. distillation.
7.
My best guess is that the sign of this sort of change is in expectation negative for EV of the future, so this is a weak motivation for scaling faster. But I’m not confident.
8.
Or scaling in Stage 1 with only training compute.
9.
At least, assuming takeoff isn’t much faster than in the [Plan A default](https://www.ai-2040.com/supplements/takeoff-supplement).
10.
“Our” meaning the official Plan A median parameters, which represent Thomas’s views rather than mine; my median takeoff is significantly slower.
11.
A few thoughts about the difference between deal decline and covert project takeover with respect to Stage 1 vs. Stage 2 desirability: (a) covert projects can wait until they are confident in alignment if they’d like, even if in Stage 1, while a post-deal-decline project may not be able to; (b) on the other hand, covert projects do not benefit during Stage 2 from the safety tax the Consortium has paid, while post-deal-decline projects do.
12.
Specifically, this is relative to the expected value of an intact, unimpaired deal with no covert TED-AIs five years after the Consortium reaches TED-AI.
13.
I’m crudely bucketing covert project outcomes into takeover and no takeover, and assuming the takeover ones don’t destroy value. In practice, there will be cases in which value is destroyed but there isn’t takeover; I’m assuming otherwise for simplicity.
14.
Orr simply being more pessimistic than me about covert projects relative to deal decline.
15.
This could also be the case if they agree with our current best guess about detection but have a much higher estimate of the badness of a covert project reaching TED-AI.
16.
And there might be some cases where it’s difficult to apply safety techniques without increasing the capabilities of legal AIs.
17.
Technically, there could be more than one crossover point, but we expect that in practice there will likely be a single crossover point at which costs of delaying begin to outweigh benefits. That said, this isn't guaranteed, for example there may be predictably transient increases in deal dissolution risks at various points.
sources/ai2040-pages/comparing-possible-plans.md
+219
View File
@@ -0,0 +1,219 @@
---
url: https://ai-2040.com/supplements/comparing-possible-plans
---
## [AI 2040](/)
≡[PDF](https://ghjyjqzwz4.ufs.sh/f/9qHa0cBclQ7sf6FVzqbp2gHFXC1IbGS5ZA7DmNBusfM4YJRP)
[Supplements](/supplements)
Plan A Basics
[FAQFAQ](/supplements/faq)[How Plan A solves our 5 biggest problemsHow Plan A solves our 5 biggest problems](/supplements/how-plan-a-solves-our-5-biggest-problems)
Plan A Policies
[Covert AI ProjectsCovert AI Projects](/supplements/covert-ai-projects)[Verification PlanVerification Plan](/supplements/verification-plan)[Transparency PlanTransparency Plan](/supplements/transparency-plan)[Capability Scaling StrategyCapability Scaling Strategy](/supplements/capability-scaling-strategy)[Security in Plan ASecurity in Plan A](/supplements/security-in-plan-a)
Analysis of Plan A Policies
[Plan A AssumptionsPlan A Assumptions](/supplements/plan-a-assumptions)[Comparing Possible PlansComparing Possible Plans](/supplements/comparing-possible-plans)[Deal DeclineDeal Decline](/supplements/deal-decline)[Takeoff SupplementTakeoff Supplement](/supplements/takeoff-supplement)
Economics
[Economics of Plan AEconomics of Plan A](/supplements/economics-of-plan-a)[Compute in Plan ACompute in Plan A](/supplements/compute-supplement)[Economic Growth ExplorerEconomic Growth Explorer](/supplements/econ-explorer)
Speculative Analysis
[Space Governance PlanSpace Governance Plan](/supplements/space-governance-plan)[AI for EpistemicsAI for Epistemics](/supplements/ai-for-epistemics)[Alignment RoadmapAlignment Roadmap](/supplements/alignment-roadmap)
[About](/about)
[![AI Futures Project](/ai-futures-logo.svg)](https://ai-futures.org/)
# Comparing Possible Plans
### Eli Lifland
# Introduction
In this supplement I’ll estimate how well each plan does on various intermediate metrics, and its chance of leading to a great future. I discuss quantitative metrics for the sake of concreteness, however I’m very much not confident in my precise estimates. I am confident in the conclusion that Plan A is substantially better than Plans B-D. For plans that I think could be competitive with Plan A, see [this table](https://www.ai-2040.com/supplements/comparing-possible-plans#plans-that-might-be-competitive-with-plan-a).
I estimate how good each plan would be conditional on the government and/or leading companies attempting to execute it. The below aren’t meant to be an exhaustive taxonomy; I am considering what would happen if roughly the described plan is attempted, rather than bucketing every possible future into the plan it is closest to. For more detail on how I classify trajectories into plans, see [below](/supplements/comparing-possible-plans#classifying-trajectories-into-plans).
The plans I consider are:
1. **Plan A (as implemented specifically in the scenario): Verified slowdown + total research transparency.** A verified international slowdown deal coupled with complete transparency of AI research. AI capabilities are scaled up while maintaining high confidence in safety but balancing risks from [deal decline](/supplements/deal-decline) and [covert projects](/supplements/covert-ai-projects).
2. **Plan A: Verified slowdown + substantial transparency.** To be more generally classified as Plan A, the implementation must involve a substantial verified slowdown which involves continued AI scaling, as opposed to an indefinite halt. It must involve transparency at least as strong as embedded auditors from foreign governments which publish reports to governments (and for which redacted versions are made public), as described in our [filtered transparency proposal](/supplements/transparency-plan). Other aspects of the plan, such as [mutually assured compute destruction](/supplements/deal-decline#mutually-assured-compute-destruction-destroying-compute-fabs-and-robots), are not required.
3. **Plan B-kinetic: Sabotage China kinetically and burn some lead.** Sabotaging China in order to get more of a lead to burn on safety, with the willingness to escalate to large-scale kinetic attacks such as by drones or conventional missile strikes. To be classified as Plan B the US must have the intention to use their lead over China for safety purposes, and must in practice slow takeoff by at least 3 months.
4. **Plan B-cyber: Sabotage China non-kinetically and burn some lead.** The same as Plan B-kinetic, but without willingness to escalate to large-scale kinetic attacks. This could include cyber and supply chain attacks.
5. **Plan C+: Domestic regulation and slow China without sabotage.** The US buys substantial time via domestic regulation, without sabotaging China. They instead aim to slow down China via non-sabotage measures such as export controls and giving green cards to Chinese talent. Requires at least a 2 month slowdown.
6. **Plan C: Leading project burns some lead.** The leading AI project burns some of its lead on safety, and potentially coordinates with other frontier projects for a further slowdown without domestic regulation. Requires at least a 1 month slowdown.
7. **Plan D: Race.** Frontier AI projects race through the intelligence explosion at nearly max speed, devoting a nonzero but small percentage of resources to safety (at least 1%; lower would count as [Plan E](/supplements/comparing-possible-plans#plans-that-we-think-are-clearly-worse-than-plan-a)).
There are also other plans that the government could attempt to implement. Some of these are discussed [below](/supplements/comparing-possible-plans#other-plans).
**For simplicity,** **I assume that:**
1. **The minimum-length takeoff from Automated Coder (AC) to Top-Expert-Dominating-AI (TED-AI) is slightly less than a year, from the start of 2030 to the end of 2030.** This matches the [default Plan A AI Futures M](https://www.ai-2040.com/supplements/takeoff-supplement)[odel trajectory](https://www.ai-2040.com/supplements/takeoff-supplement). The minimum-length takeoff is what happens if the leading AI projects go at maximum speed.
2. **All plans can begin implementation in early 2029** as in the Plan A scenario, though for Plans C and D this doesn’t matter as the plan involves doing nothing until takeoff is underway. This still might involve some safety work, if doing so is useful for advancing capabilities.
I remove these conditionals and present other authors’ views [below](/supplements/comparing-possible-plans#comparing-authors-views-on-plan-likelihood-and-outcomes).
# Plan assessment table
These estimates are all rough best guesses, and the specifics of them aren’t load-bearing to our recommendation of Plan A (though the high-level takeaways are).
How the plans compare according to Eli
MedianCompetent
Expand all
Toggle **Median** vs **Competent** to switch between the median across implementations and each plan executed competently. Hover any plan, metric, or value for its reasoning, or **click within a row to see every plan’s reasoning at once**.
Metric ↓ / Plan →| Plan A| Plan B| Plan C+| Plan C| Plan D
---|---|---|---|---|---
Kinetic| Cyber
Objective metrics
▶Takeoff length (AC to takeover-capable AI)| 6 yrs| 3 yrs| 2 yrs| 1.5 yrs| 1.13 yrs| 1.02 yrs
▶Training compute safety tax (OOMs)| 2.8| 1| 0.75| 0.46| 0.13| 0.02
▶Safety compute (H100e-yrs)| 100B| 500M| 100M| 20M| 4M| 500K
▶Safety-researcher-years| 1M| 2,000| 1,000| 1,500| 500| 200
Subjective metrics (1–5)
▶Epistemic culture| 3| 1| 2| 2| 1
▶Public transparency| 4| 1| 2| 2| 1
▶Distribution of power| 5| 1| 2| 2| 1
Outcomes (medians across implementations)
▶p(alignment); i.e., 1 - p(misaligned takeover)| 72%| 50%| 45%| 40%| 25%
▶p(great future | alignment)| 58%| 50%| 55%| 50%| 40%
▶p(great future)| 42%| 25%| 25%| 20%| 10%
Plans are alphabetically ranked in terms of their objective metrics, with a big jump between Plan A and Plan B. However, plans C+ and C beat out Plan B on some subjective metrics. Plan B’s focus on security and its wartime vibe could degrade epistemics, and its likely implementation of nationalization or pseudo-nationalization would decrease public transparency and concentrate power.
# Comparing authors’ views on plan likelihood and outcomes
These estimates aggregate over uncertainties about the default capability trajectory and when the plans are implemented, as well as how competently the plans are executed. The numbers are in [this sheet](https://docs.google.com/spreadsheets/d/1FDyYPky6VeIs81ujssgQKyv-yIZc8iOZQrZv6KlJCNk/edit?gid=1821022003#gid=1821022003), and the plan classification is as described [below](/supplements/comparing-possible-plans#classifying-trajectories-into-plans).
## Likelihood
### Probability each plan happens across authors
Probability plan happens0%25%50%75%100%Plan APlan BPlan C+Plan CPlan DOtherEli — Plan A: 5%Eli — Plan B: 15%Eli — Plan C+: 10%Eli — Plan C: 20%Eli — Plan D: 25%Eli — Other: 25%Daniel — Plan A: 15%Daniel — Plan B: 10%Daniel — Plan C+: 20%Daniel — Plan C: 10%Daniel — Plan D: 30%Daniel — Other: 15%Thomas — Plan A: 3%Thomas — Plan B: 8%Thomas — Plan C+: 8%Thomas — Plan C: 12%Thomas — Plan D: 50%Thomas — Other: 19%Romeo — Plan A: 8%Romeo — Plan B: 12%Romeo — Plan C+: 18%Romeo — Plan C: 12%Romeo — Plan D: 30%Romeo — Other: 20%Ryan — Plan A: 4%Ryan — Plan B: 12%Ryan — Plan C+: 25%Ryan — Plan C: 14%Ryan — Plan D: 28%Ryan — Other: 18%Daniel · 15%Romeo · 8%Eli · 5%Ryan · 4%Thomas · 3%Eli · 25%Romeo · 20%Thomas · 19%Ryan · 18%Daniel · 15%
Reasoning (hover an author — here or on the chart — to spotlight their line and read their full reasoning).
Eli| p(govt does something at least as intense as Plan B): 35%; seems like govt wakeup has been increasing and also the importance of ASI will get more and more obvious. Though note that Plan B requires actually slowing down by at least 3 months.
p(deal | at least as intense on Plan B): 50%; deal is clearly rational but there are various blockers in practice. Again note that something at least as intense as Plan B requires at least a 3 month slowdown.
p(Plan A | deal): 30%; Plan S, a verified slowdown with less transparency, or just a somewhat weaker deal are all plausible.
This gives Plan A is ~5%, other deals are ~10%.
Let's say Plan B is 15% and other intense non-deal things are 5%, given that Plan B is quite broad.
Domestic regulation as in Plan C+ seems plausible as well, though the requirement of actually having a 2 month slowdown lowers the probability. Also, it's a bit of a narrow target to have the will to do this but not Plan B or higher.
Plan C seems more likely than C+. Plan D seems a bit more likely than Plan C.
Other being at 25% feels maybe a bit low but not too bad as I think B, C, C+, and D together are broad enough to describe a bunch of the worlds that could happen at similar levels of intensity.
---|---
Ryan| I'm very uncertain about what should count. I think Plan C+ is mostly worlds with dumb regulation that slows down AI. I think Plan B would be higher if not for the intent to spend the lead on safety.
## Outcomes
We focus on p(alignment) here, i.e. 1 - p(misaligned takeover). We ignore p(great future | alignment) in this section for simplicity.
### Probability of avoiding misaligned takeover by plan
Probability of avoiding misaligned takeover0%25%50%75%100%Plan APlan BPlan C+Plan CPlan DOtherEli — Plan A: 70%Eli — Plan B: 55%Eli — Plan C+: 50%Eli — Plan C: 45%Eli — Plan D: 32%Eli — Other: 55%Daniel — Plan A: 80%Daniel — Plan B: 20%Daniel — Plan C+: 40%Daniel — Plan C: 20%Daniel — Plan D: 10%Daniel — Other: 35%Thomas — Plan A: 80%Thomas — Plan B: 53%Thomas — Plan C+: 45%Thomas — Plan C: 35%Thomas — Plan D: 20%Thomas — Other: 30%Romeo — Plan A: 80%Romeo — Plan B: 35%Romeo — Plan C+: 40%Romeo — Plan C: 35%Romeo — Plan D: 20%Romeo — Other: 50%Ryan — Plan A: 81%Ryan — Plan B: 68%Ryan — Plan C+: 60%Ryan — Plan C: 63%Ryan — Plan D: 55%Ryan — Other: 50%Ryan · 81%Daniel · 80%Thomas · 80%Romeo · 80%Eli · 70%Eli · 55%Romeo · 50%Ryan · 50%Daniel · 35%Thomas · 30%
Reasoning (hover an author — here or on the chart — to spotlight their line and read their full reasoning).
Eli| My probabilities in the plan assessment above were conditional on:
(a) the default trajectory being the [default Plan A one](https://www.ai-2040.com/supplements/takeoff-supplement), which has a bit under a year between AC and TED-AI and a year between AC and ASI.
(b) The plan implementation beginning in early 2029, ~0.75 years before AC.My median takeoff is ~2x longer than the one described in (a), but I have significantly probability on faster takeoffs. This means that my p(alignment) estimates should do a combination of increase and regress a bit toward 50%. Also, Plan A isn't as sensitive to default takeoff length as other plans, and from a covert project perspective it's actually unclear in which way the effect size points because larger effective compute gaps between the start and end of takeoff require more software progress. The trajectory timeline to AC / start of takeoff is pretty similar to my median so for that I regress abit toward 50%.For (b), my median is that the plan gets implemented later than this, but I think it doesn't matter much for all the plans besides A because in the other plans it's best to slow down later anyway. I do lower the value of Plan A due to this consideration, though not hugely because I think covert projects would likely still be difficult to do without detection.Also, the plan assessment numbers were based on the median implementation of the plan, rather than aggregating the expected outcome over all implementations. My unconfident quick guess is that switching to expected outcome should also regress my numbers toward 50%, but not sure so this adjustmnet will be small.With the above in mind, I make the following adjustments:
A: 72->70
B: 50->55
C+: 45->50
C: 40->45
D: 25->32For Other, I think a bunch of this is random pretty good plans that aren't covered, including Plan S. E.g. there's ~40% that this is a non-Plan-A deal. Though it also includes Plan E. Overall I rate it the same as B, 55.
---|---
Ryan| These numbers are very sensitive to how you classify plans. E.g., I think Plan A as described in the scenario has maybe 3/5 as much risk as Plan A as classifed by the supplement. An actually good version of Plan C done by the leading AI company is pretty close to the numbers I give for Plan B (but worse than a great Plan B). A large fraction of "other" is worlds with not enough effort on safety to classify for Plan D.
# Other plans
There are many other possible plans that the government could follow. Below we catalog some of the ones that we find to be most promising and/or likely.
## Plans that might be competitive with Plan A
**Summary**| **Advantages (relative to A)**| **Disadvantages (relative to A)**
---|---|---
**Plan S:** **Indefinite halt.** A halt on all frontier AI capabilities progress, intended to last at least a few years. Different variants of Plan S have different conditions for resumption of AI progress; for example it could be alignment progress, lie detectors, human uploads, or intelligence enhancement.| A longer expected slowdown and more margin for error regarding scaling too fast. Potentially simpler.| Scaling to controllable AIs within the human range is helpful for accelerating alignment/control, epistemics, verification, and general understanding of AI.
**Domestic-first Plan A.** Regulate AI domestically enough to reduce AI takeover risk to acceptable levels, which will require a long slowdown. It’s possible other countries will also regulate domestically and something like Plan A won’t be needed; otherwise, transition to Plan A later.| Initial steps are achievable by the US and very helpful even if the international stage isn’t viable because it massively extends the timeline. Proactively builds trust with China.| Worse for covert projects and setting up verification than negotiating Plan A at the same time. May not be politically feasible due to race dynamics.
**GPU arms control.** International agreements for countries to limit their GPU stock or flow, analogous to historical arms reduction agreements.| Much simpler to enforce, and more historical precedent. Can achieve substantial slowdown.| Less slowdown, less ability to pay safety taxes, ongoing race dynamics mean no ability to coordinate towards safer paths through the tech tree.
**CERN for AI. An international project to develop frontier AI,** with all other projects regulated to be substantially behind in capabilities.| Easier to defend against algorithmic leakage and distillation to covert projects. Less actors at the frontier might make it easier to enforce.| Worse decisions (including worse decisions on technical safety) due to less transparency and less broad deployment. More concentration of power risk.
## Plans that we think are clearly worse than Plan A
1. **Repurpose the GPUs:** Like _GPU arms control_ , except they are now required to be used for existentially useful applications rather than being destroyed. See [this comment](https://www.lesswrong.com/posts/HE3Styo9vpk7m8zi4/evhub-s-shortform?commentId=FqDBLsLPNEqF2ahBL) for a more specific proposal in which GPUs are required to be used for giving inference capacity to external safety organizations or to do publicly transparent safety research. In theory this plan should be better than _GPU arms reduction_ , but it’s less reversible and harder to enforce.
2. **Plan B-but-don’t-slowdown, aka D-war:** Like Plan B except that you don’t slow down, i.e. you don’t burn ~any of the lead that you increased by sabotaging China. (Currently classified as a variant of Plan D when making the above estimates.)
3. **Weaker deals with China.** For example, the US could offer: "You get access to more chips for AI inference, and a non-binding or non-majority say in our safety requirements, if you give us full transparency into what you're up to.”
4. **Plan E:** Like Plan D but even less safety investment, i.e. less than 1% of resources.
# Classifying trajectories into plans
Define the _implementation window_ as the period between:
1. **Start: 1 year before[Automated Coder](https://docs.google.com/document/d/1ru6Okbxb6XuH18Cz8439sdQJazMV39hNxsWDokh97r0/edit?tab=t.0#heading=h.h1t0i8bvad56)** **(AC) would be reached if scaling at maximum speed**. An AC is a collective of AIs that fully automates an AGI project’s coding work, autonomously replacing the project’s entire coding staff. That is, you’d rather fire all of the humans than stop using AIs.
2. **End: Takeover-capable AI.** Takeover-capable AI is reached when it is practically very unlikely that humans could stop AI takeover, conditional on the AIs being adversarially misaligned. It could still be in theory possible for them to shut down the AIs by taking extremely drastic actions like shutting down the internet, but it’s unlikely that they would do so, e.g. because AIs might be heavily relied upon and/or highly persuasive.
If the plan being attempted is consistent throughout this window, then the trajectory is classified into that plan. Even if multiple plans are attempted, each trajectory can count as at most one plan; this is a non-obvious choice and it seems also reasonable to count trajectories as multiple plans.
If multiple plans are attempted during the implementation window, the trajectory is classified as the most intense plan that was attempted. This is a bit vague to use as a general classification, but for concreteness among Plans S/A/B/C+/C/D trajectories would be classified as the attempted plan that is earliest in that list. The reasoning behind this is that it seems intuitively like a trajectory should count primarily as Plan A or S even if the [deal dissolves](/supplements/deal-decline) quickly, and also should count as Plan A or S if it starts as a less intense plan but then transitions to A/S within the implementation window.
The following selection effect is a consequence of this classification: In more intense plans, people are selected for being more competent. Furthermore, less intense plans are selected for people not later transitioning to a more intense plan even if concerning evidence arises.
One selection effect that I explicitly remove is the effect of background variables such as alignment difficulty on what plan a trajectory counts as. I.e., I assume when rating plans that the distribution of alignment difficulty is the same no matter what plan is attempted; in particular my overall guess as to the distribution of alignment difficulty. As mentioned above, I don’t do this for things that humans control such as their competence.
# Appendix
## Training compute safety tax definition
The training compute safety tax is the difference between the amount of compute actually used on a training run to reach a given capability level, and the minimum amount of compute required to reach that capability. Examples of ways to productively use that unnecessary compute might be to train in a more safe but less compute-efficient paradigm, or to spend lots of compute on AI-driven scalable oversight to improve incentives during the training process.
Measuring the safety tax by this method is an imperfect proxy for what we actually care about. There is likely large variance among improvements which provide the same amount of capability in how useful it is for safety to forego that improvement.
1.
Other potential differences from Plan A, though it depends on which variant of Plan S is implemented: slower compute buildout, less transparency.
2.
A special case of this is to **Pause the GPUs:** Require that a fixed fraction of the GPUs be either (i) turned off or (ii) used to mine cryptocurrency (or be used for some verifiable, non-AI R&D purpose). This proposal has the upside and downside of making it easier to unpause the GPUs.
3.
But maybe you can mitigate this via small public deployments and selective transparency.
4.
Potentially I should allow for plan implementation to begin earlier, especially if the plan is very intense. For example, probably Plan S should count if it begins 1.5 years before the default AC time.
sources/ai2040-pages/compute-supplement.md
+1529
View File
File diff suppressed because it is too large Load Diff
sources/ai2040-pages/covert-ai-projects.md
+1286
View File
File diff suppressed because it is too large Load Diff
sources/ai2040-pages/deal-decline.md
+1129
View File
File diff suppressed because it is too large Load Diff
sources/ai2040-pages/econ-explorer.md
+33
View File
@@ -0,0 +1,33 @@
---
url: https://ai-2040.com/supplements/econ-explorer
---
## [AI 2040](/)
≡[PDF](https://ghjyjqzwz4.ufs.sh/f/9qHa0cBclQ7sf6FVzqbp2gHFXC1IbGS5ZA7DmNBusfM4YJRP)
[Supplements](/supplements)
Plan A Basics
[FAQFAQ](/supplements/faq)[How Plan A solves our 5 biggest problemsHow Plan A solves our 5 biggest problems](/supplements/how-plan-a-solves-our-5-biggest-problems)
Plan A Policies
[Covert AI ProjectsCovert AI Projects](/supplements/covert-ai-projects)[Verification PlanVerification Plan](/supplements/verification-plan)[Transparency PlanTransparency Plan](/supplements/transparency-plan)[Capability Scaling StrategyCapability Scaling Strategy](/supplements/capability-scaling-strategy)[Security in Plan ASecurity in Plan A](/supplements/security-in-plan-a)
Analysis of Plan A Policies
[Plan A AssumptionsPlan A Assumptions](/supplements/plan-a-assumptions)[Comparing Possible PlansComparing Possible Plans](/supplements/comparing-possible-plans)[Deal DeclineDeal Decline](/supplements/deal-decline)[Takeoff SupplementTakeoff Supplement](/supplements/takeoff-supplement)
Economics
[Economics of Plan AEconomics of Plan A](/supplements/economics-of-plan-a)[Compute in Plan ACompute in Plan A](/supplements/compute-supplement)[Economic Growth ExplorerEconomic Growth Explorer](/supplements/econ-explorer)
Speculative Analysis
[Space Governance PlanSpace Governance Plan](/supplements/space-governance-plan)[AI for EpistemicsAI for Epistemics](/supplements/ai-for-epistemics)[Alignment RoadmapAlignment Roadmap](/supplements/alignment-roadmap)
[About](/about)
[![AI Futures Project](/ai-futures-logo.svg)](https://ai-futures.org/)
sources/ai2040-pages/economics-of-plan-a.md
+633
View File
@@ -0,0 +1,633 @@
---
url: https://ai-2040.com/supplements/economics-of-plan-a
---
## [AI 2040](/)
≡[PDF](https://ghjyjqzwz4.ufs.sh/f/9qHa0cBclQ7sf6FVzqbp2gHFXC1IbGS5ZA7DmNBusfM4YJRP)
[Supplements](/supplements)
Plan A Basics
[FAQFAQ](/supplements/faq)[How Plan A solves our 5 biggest problemsHow Plan A solves our 5 biggest problems](/supplements/how-plan-a-solves-our-5-biggest-problems)
Plan A Policies
[Covert AI ProjectsCovert AI Projects](/supplements/covert-ai-projects)[Verification PlanVerification Plan](/supplements/verification-plan)[Transparency PlanTransparency Plan](/supplements/transparency-plan)[Capability Scaling StrategyCapability Scaling Strategy](/supplements/capability-scaling-strategy)[Security in Plan ASecurity in Plan A](/supplements/security-in-plan-a)
Analysis of Plan A Policies
[Plan A AssumptionsPlan A Assumptions](/supplements/plan-a-assumptions)[Comparing Possible PlansComparing Possible Plans](/supplements/comparing-possible-plans)[Deal DeclineDeal Decline](/supplements/deal-decline)[Takeoff SupplementTakeoff Supplement](/supplements/takeoff-supplement)
Economics
[Economics of Plan AEconomics of Plan A](/supplements/economics-of-plan-a)[Compute in Plan ACompute in Plan A](/supplements/compute-supplement)[Economic Growth ExplorerEconomic Growth Explorer](/supplements/econ-explorer)
Speculative Analysis
[Space Governance PlanSpace Governance Plan](/supplements/space-governance-plan)[AI for EpistemicsAI for Epistemics](/supplements/ai-for-epistemics)[Alignment RoadmapAlignment Roadmap](/supplements/alignment-roadmap)
[About](/about)
[![AI Futures Project](/ai-futures-logo.svg)](https://ai-futures.org/)
# Economics of Plan A
### Romeo Dean, Thomas Larsen
We project that AIs will, at some point in the next 15 years, be capable of automating all human labor: both physical and cognitive. We believe that this will have extraordinary economic impacts, causing output to grow explosively at a historically unprecedented rate (barring an AI-caused catastrophe). Most of our view comes from simple arguments about the future of AI. However, over the course of our thinking, we also found it useful to make a detailed economic model. This document is organized into the following sections:
1. Core arguments behind our views on explosive growth from AI ([link](/supplements/economics-of-plan-a#core-arguments-for-explosive-growth))
2. Why we disagree with mainstream economists ([link](/supplements/economics-of-plan-a#why-we-disagree-with-mainstream-economists))
3. A simple two-factor explorer that shows the core dynamics ([link](/supplements/economics-of-plan-a#the-simple-explorer))
4. A summary of the outputs from our full economic model in the context of Plan A ([link](/supplements/economics-of-plan-a#the-full-economic-model))
The core disagreement between us and most economists is our different expectations for AI capabilities. We believe that AIs will be able to fully automate all economically relevant work (in a median of ~6 years), while most economists believe that AIs will behave more like normal technology. We’ve already made [many arguments about AGI timelines elsewhere](https://www.aifuturesmodel.com/), so this post centrally argues for why, _conditional on AI that can automate all economically relevant tasks_ , we expect unprecedented explosive economic growth.
## Core arguments for explosive growth
Right now, the size of the economy is closely tied to human labor, so if the population were to grow massively, this would cause nearly proportional growth in total output. Once AIs can perfectly substitute for human labor, increasing the number of AIs would have the same effect as increasing human population. At least if you exclude tasks that are intrinsically human, but we don’t expect these to be a large share of the economically relevant tasks. For example, human-preference driven tasks the AI can’t substitute for like ‘human babysitting’ should not preclude an AI and robot-only economy from growing in a closed loop (bounded only by [bottlenecks that we expect to not bite strongly](/supplements/economics-of-plan-a#why-we-disagree-with-mainstream-economists)).
In fact, this undersells the likely effects because in reality, AIs won’t just be increasing in population, but also in qualitative ability leading to outputs that wouldn’t have been feasible to produce with humans. This is because, shortly after being able to substitute for human labor, AIs will become qualitatively superhuman. However, improvements to intelligence are harder to model quantitatively than increases in population, **so we’ll settle for establishing a population-based lower bound by ignoring qualitative improvements.**
**So how fast will the “population” of AIs and robots be growing at the point of full automation?** We need to do separate analyses here for physical and cognitive labor, because the underlying dynamics driving the growth rates are quite different.
* **Cognitive Labor: a doubling time of roughly 50 days.** The total cognitive labor from AI depends on total inference compute and the compute efficiency of the AIs. The simplest way of making a baseline guess here is to look at historical rates for each of these.
* **Total AI inference compute: 3.3x/year, 7 month doubling time.** [Epoch estimates](https://epoch.ai/data-insights/ai-chip-production) that the total compute in the world has been doubling every 7 months. In the future, we think there are reasons for this to slow down (it’s already requiring [2-3% of](https://finance.yahoo.com/news/big-techs-ai-spending-spree-142111465.html?guccounter=1) [GDP investment in the US](https://finance.yahoo.com/news/big-techs-ai-spending-spree-142111465.html?guccounter=1), so doubling a few more times seems hard), but also reasons for it to speed up at the point of full automation (e.g., future AIs automating chip R&D, and robots automating the AI chip supply chain).
* **Compute efficiency on achieved capabilities: 30x/year, 2.5 month doubling time.** [Epoch estimates](https://epoch.ai/data-insights/llm-inference-price-trends) that the price of a given level of AI capability decreases by an average of 40x/yr (which is equivalent to halving every 2.2 months). Assuming [hardware price performance](https://epoch.ai/trends#hardware) has been improving 1.4x/yr and margins have been flat, this gives a trend of 30x/year improvement in compute efficiency. When the cost of AI inference halves, this is at least as economically beneficial as the population of AIs doubling, because we can run twice as many AIs.
* **Putting these together, this means that the population of AIs today at a fixed capability level could increase at a rate of 100x/year, which is a 55 day doubling time.**
* **Physical Labor: a doubling time of a few days to a year.** For physical labor, we can reason directly about the number of robots in the world. At the moment, [Epoch finds](https://epoch.ai/blog/how-fast-could-robot-production-scale-up) that the number of robots is dominated by wheeled robots, drones and robotic arms, and these are not scaling very quickly. ![](/epoch-robots.png)
Right now, these robots are not capable of performing all tasks in the robot manufacturing process (including mining raw materials, transportation, and every step of the supply chain). However, once robots reach this capability threshold then there’s a clear reason to expect exponential growth in the number of robots: the rate of new robot construction will be approximately proportional to the number of robots in existence (barring raw material bottlenecks that we address below). But how fast of an exponential? We walk through our reasoning below, and also have a review of the doubling-times literature in [Appendix A](/supplements/economics-of-plan-a#appendix-a-robot-doubling-times).
**Why wouldn’t robots plateau like other products?** Right now, robots and AI compute are both doubling faster than once per year. On its own this is not strong evidence of anything: many new products grow this fast at first, and then plateau once they saturate their addressable market. We expect robots to be different, because once AI and robots can perform every task in their own supply chain, the two things that normally end fast growth stop applying:
1. **No demand ceiling.** Robots and AIs are productive assets, not consumer goods. As long as an extra robot produces more value than it costs to build, there is an incentive to keep building them, past the point where the robot population exceeds the human population and far beyond it.
2. **No human bottleneck on supply.** Today’s production growth is partly borrowed from human labor (factory workers, miners, truck drivers, engineers). At full automation the robot economy doesn’t need humans to buy anything, build anything, or deliver ingredients to its factories, so the size of the human population stops being a constraint on how big it can get.
The ceiling that does eventually bind is physical resources: surface area for solar panels, minerals in the earth’s crust to mine. But by our estimates those bottlenecks bite very late, after many orders of magnitude of growth (see the [bottlenecks section below](/supplements/economics-of-plan-a#bottlenecks-and-other-arguments)).
That leaves the question of the growth rate in between. A reasonable worry is that today’s doubling rates are inflated by human help, and that growth slows once the robot economy has to do essentially everything itself. We don’t think this changes the bottom line: analyses of what a fully automated economy could achieve with current technology and no further improvement, e.g. [Binder’s von Neumann growth rate calculation](https://defensesindepth.bio/ai-industrial-takeoff-part-1-maximum-growth-rates-with-current-technology/) from US input-output tables, capital stocks, and depreciation data, land at around one doubling per year, conservatively. An intuition pump: a modern car factory produces roughly [its own weight in cars every year](/supplements/economics-of-plan-a#appendix-a-robot-doubling-times). And a year is likely an underestimate of the long-run pace, because superhuman AI R&D should keep improving robot designs, and biology shows much faster replication is physically possible (e.g., animals like rabbits can double their population in 6 weeks, bacteria in hours). This is why we quote doubling times between a few days and a year, with the fast end becoming more plausible over time.
Overall, our argument can be summarized as:
1. AI and robots will be capable of fully automating all economically relevant tasks.
2. Therefore, the AI-robot economy will be capable of autonomously self-replicating (building more efficient AIs, building more robots, building more GPUs, etc.).
3. The replication rate will be fast: around one doubling per year even with current technology and no human help, and rising over time as AI R&D improves robot designs.
4. Therefore, the AI-robot economy will undergo rapid exponential growth, with a doubling time of a year at the slow end and plausibly months or days, until physical resource limits bite (very late, on our estimates).
Even if the true growth rate is much slower than this baseline predicts in the future, the effective AI population seems very likely to rapidly outgrow the total number of humans. Once AIs dominate the number of humans, if their population is (conservatively) increasing at 2x per year, this would involve ~100%/yr growth, whereas [current growth](https://data.worldbank.org/indicator/NY.GDP.MKTP.KD.ZG) is ~3%/yr.
## Why we disagree with mainstream economists
Most mainstream economists are skeptical of explosive economic growth from AI. Even conditional on a “rapid AI progress scenario”, [economists predicted](https://static1.squarespace.com/static/635693acf15a3e2a14a56a4a/t/69cbb9d509ada447b6d9013f/1774959061185/forecasting-the-economic-effects-of-ai.pdf) an average of 3.5%/yr GDP growth between 2030 and 2050.
One core disagreement between us and mainstream economists tends to be about how capable AIs will be. We believe that AI's ability to do tasks in the economy will look something like the red curve below absent strong regulations. This reflects AI dominating humans at all tasks and achieving 100% automation around 2031 (top expert dominating milestone), which is close to the authors' median expectations. We find that economists tend to have much lower expectations for automation in the coming decades.
Share automatable0%25%50%75%100%2025202620272028202920302031203220332034203520362037203820392040
AI 2040 Default World
AI 2040 Plan A
Acemoglu: ~5% “cost-effectively automatable” [(2024)](https://shapingwork.mit.edu/wp-content/uploads/2024/05/Acemoglu_Macroeconomics-of-AI_May-2024.pdf)
Goldman Sachs: ~25% “substitutable” [(2023)](https://www.gspublishing.com/content/research/en/reports/2023/03/27/d64e052b-0f6e-45d7-967b-d7be35fabd16.pdf)
_Note: Acemoglu and Goldman Sachs projections are not necessarily directly comparable, as their definitions for "cost-effectively automatable" and "substitutable" may diverge somewhat from our automatable definition._
Automatable tasks: AI 2040 scenarios vs prior forecasts \- ai-2040.com
In Plan A, strong regulations (coordinated internationally) are sufficient to slow this down, but still lead to ~99% automation in the late 2030s, as regulation only preserves critical AI oversight jobs (where humans are still significantly uplifted by AI). We believe that AI's capabilities will be so high that lower automation levels (e.g., due to wanting to preserve more human jobs) will be extremely uncompetitive (and thus be quickly competed away), with preferences for human-to-human interactions being (1) supplemented by more leisure, and (2) insufficient to push human wages high enough relative to the citizen's dividend to attract much labor.
### Acemoglu (2024): too few tasks automated by AI
[Acemoglu (2024)](https://economics.mit.edu/sites/default/files/2024-04/The%20Simple%20Macroeconomics%20of%20AI.pdf) (also [critiqued here](https://x.com/AndreyFradkin/status/1840837475062988911) by Leopold Aschenbrenner) assumes only ~5% of tasks in the next 10 years will be cost-effectively automatable by AI. Acemoglu’s overall estimate was that AI would increase GDP by 0.9% over the next 10 years (which is approximately a 0.08% increase in GDP annually). We think Acemoglu’s view from 2024 is already close to falsified based on current AI investment and AI revenue numbers.
Early evidence against Acemoglu (2024)
* AI investment contributed around 1% to GDP growth in the first half of 2025 alone ([JP Morgan](https://am.jpmorgan.com/us/en/asset-management/adv/insights/market-insights/market-updates/on-the-minds-of-investors/is-ai-already-driving-us-growth/), [EY](https://www.ey.com/en_us/insights/ai/ai-powered-growth)), whereas Acemoglu predicted .9% growth over the course of 10 years.
* Even at the scale of AI revenues, over the course of the last six months, Anthropic’s revenue run rate alone has [increased by $38B](https://epoch.ai/data/ai-companies?view=graph&tab=revenue) from $9B to $47B. We think this will plausibly beat the rate of increased growth per year (0.08% of US GDP) of ~$30B that Acemoglu predicted for the entire AI industry. Note: Anthropic's contribution to GDP is difficult to disentangle, but should count employee compensation + operating surplus + depreciation costs, which we think is on track to be above $30B this year.
The only ways Acemoglu’s predictions will hold up are if (1) AI revenue and investment growth essentially stops, or (2) AI investment and revenues are not counterfactual (the spending and revenue would’ve happened elsewhere anyway). Both seem unlikely.
### Jones & Tonetti (2026): slow AI automation of R&D
[Jones & Tonetti (2026)](https://web.stanford.edu/~chadj/JonesTonetti_Automation.pdf), _Past Automation and Future A.I.: How Weak Links Tame the Growth Explosion_ features an elegant growth model combining capital and labor. They grant that capital may eventually automate every task, but their argument is that because tasks are strong complements (they assume an elasticity of substitution σ=0.2\sigma = 0.2σ=0.2), the shrinking set of tasks machines cannot yet do becomes a 'weak link' whose cost share rises ([Baumol's cost disease](https://en.wikipedia.org/wiki/Baumol_effect)) and strongly bottlenecks growth. Our biggest disagreement with this model is that we think the R&D automation loop is very conservative, with the model assuming:
1. **Fixed R &D investment and research productivity.** In the Jones model R&D works through "ideas accumulation" with capital productivity ψki\psi_{ki}ψki​ changing with a fixed "idea elasticity of capital" θk\theta_kθk​, a fixed "research productivity" qˉ\bar qqˉ​, and a fixed share ιˉR\bar\iota_RιˉR​ of output YYY being invested in R&D (with diminishing returns per [ideas getting harder to find](https://web.stanford.edu/~chadj/IdeaPF.pdf)).
ψ˙kiψki=θkqˉ ιˉR YQ 1−ϕ,\begin{aligned} \frac{\dot\psi_{ki}}{\psi_{ki}} &= \theta_k \frac{\bar q\,\bar\iota_R\,Y}{Q^{\,1-\phi}},\qquad \\\ \end{aligned}ψki​ψ˙​ki​​​=θk​Q1−ϕqˉ​ιˉR​Y​,​
We think this formulation systematically underestimates the effect that AI will have on R&D, because AI will be able to automate R&D labor (particularly AI R&D itself), so ideas and productivity growth should track some notion of total R&D effort accounting for this AI uplift. We think the AI field will have its own feedback loop of R&D automation, leading to higher research productivity and higher reinvestment into R&D, ultimately leading to much higher capital productivity for AI capital.
2. **No unique treatment of AI capital.** In the Jones model, there is no unique treatment of AI. AI enters as normal capital and accumulates through the classic investment less depreciation rule K˙=ιˉK Y−δK\dot K = \bar\iota_K\,Y-\delta KK˙=ιˉK​Y−δK with fixed investment share ιˉK\bar\iota_KιˉK​ and depreciation rate δ\deltaδ. We think AI would be worth modelling on its own, because we think its task productivity and investment attractiveness will grow over time, and also have a different responsiveness to more investment and different responsiveness to R&D (e.g., algorithmic progress on the orders of 10x/year instead of Moore's law 1.35x/yr). In other words, we think the "idea elasticity of AI", "research productivity of AI" and "share of output being invested in AI R&D" will all behave very differently to how they behave for other capital, leading to AI's effects being drastically underestimated by being bundled with the capital factor of production.
Jones & Tonetti Model Explanation
Output is a CES aggregate over a continuum of tasks i∈[0,1]i \in [0,1]i∈[0,1] with elasticity of substitution σ=0.2\sigma = 0.2σ=0.2:
Y=(∫01Yiσ−1σ di)σσ−1,Yi=ψkiKi+ψℓiLi,Y=\left(\int_0^1 Y_i^{\frac{\sigma-1}{\sigma}}\,di\right)^{\frac{\sigma}{\sigma-1}},\qquad Y_i=\psi_{ki}K_i+\psi_{\ell i}L_i,Y=(∫01​Yiσσ−1​​di)σ−1σ​,Yi​=ψki​Ki​+ψℓi​Li​,
where KiK_iKi​ and LiL_iLi​ are the capital and labor allocated to task iii. Within a task the two are perfect substitutes (whichever is cheaper does it, more on this below). The task productivities of capital ψki\psi_{ki}ψki​ and labor ψℓi\psi_{\ell i}ψℓi​ are given by:
ψki=Qθkf(i),ψℓi=Qθℓ,f(i)=(1−i)μ1+μ0(1−i)μ+fˉ.\psi_{ki}=Q^{\theta_k}f(i),\qquad \psi_{\ell i}=Q^{\theta_\ell},\qquad f(i)=\frac{(1-i)^{\mu}}{1+\mu_0(1-i)^{\mu}}+\bar f.ψki​=Qθk​f(i),ψℓi​=Qθℓ​,f(i)=1+μ0​(1−i)μ(1−i)μ​+fˉ​.
QQQ is the economy's stock of ideas (the technology level); and θk\theta_kθk​, θℓ\theta_\ellθℓ​ are the _idea elasticities_ , governing how fast each factor's productivity rises as ideas accumulate, with θk>θℓ\theta_k> \theta_\ellθk​>θℓ​ as capital improves faster from ideas than labor. In their 'Moore's Law Everywhere' calibration θk≈11.2\theta_k \approx 11.2θk​≈11.2 and θℓ≈0.5\theta_\ell \approx 0.5θℓ​≈0.5. The function f(i)f(i)f(i) is capital's comparative advantage at task iii, declining in iii (tasks are ordered so capital is relatively best at low iii), μ\muμ and μ0\mu_0μ0​ are shape parameters fit to historical automation, and the floor fˉ=f(1)\bar f = f(1)fˉ​=f(1) controls the long-run end state: with fˉ≥0\bar f \ge 0fˉ​≥0 automation runs essentially to completion, while fˉ<0\bar f < 0fˉ​<0 leaves a positive set of tasks permanently un-automated (worked out below).
**Automation.** Let rrr be the rental rate of capital and www the labor wage. A task is automated when capital is cheaper than labor per productivity-adjusted-unit, i.e., when rψki<wψℓi\frac{r}{\psi_{ki}} < \frac{w}{\psi_{\ell i}}ψki​r​<ψℓi​w​. Because f(i)f(i)f(i) is decreasing there is a single crossing, so the automation fraction β\betaβ can be solved by finding this crossover point rw=ψkβψℓβ\frac{r}{w} = \frac{\psi_{k\beta}}{\psi_{\ell\beta}}wr​=ψℓβ​ψkβ​​:
(r/w) today(r/w) laterβ todayβ latermore automationmore R&Dautomated taskstasks
ψ_k/ψ_ℓ: capital vs. labor productivity ratio on task i, todayψ_k/ψ_ℓ after R&Dr/w: capital vs. labor cost ratio
Automation in Jones-Tonetti \- ai-2040.com
There are two ways to increase automation, either (1) R&D, which increases the ideas stock QQQ and grows ψk\psi_kψk​ faster than ψℓ\psi_\ellψℓ​, since θk>θℓ\theta_k> \theta_\ellθk​>θℓ​, or (2) capital becoming cheaper over time (by becoming more abundant through capital accumulation) relative to wages.
Cost-minimization assigns each task to the cheaper factor, so capital does tasks [0,β][0,\beta][0,β] and labor does [β,1][\beta,1][β,1]. We can collapse output to an ordinary two-factor CES (because each task is done by just one factor, so we only need the totals KKK and LLL each weighted by their total productivity across the tasks they do, BBB and AAA):
Y=[(BK)σ−1σ+(AL)σ−1σ]σσ−1B=Qθk[∫0βf(i)σ−1 di]1σ−1A=Qθℓ (1−β)1σ−1\begin{aligned} Y &= \Big[(BK)^{\frac{\sigma-1}{\sigma}}+(AL)^{\frac{\sigma-1}{\sigma}}\Big]^{\frac{\sigma}{\sigma-1}} \\\ B &= Q^{\theta_k}\left[\int_0^{\beta} f(i)^{\sigma-1}\,di\right]^{\frac{1}{\sigma-1}} \\\ A &= Q^{\theta_\ell}\,(1-\beta)^{\frac{1}{\sigma-1}} \end{aligned}YBA​=[(BK)σσ−1​+(AL)σσ−1​]σ−1σ​=Qθk​[∫0β​f(i)σ−1di]σ−11​=Qθℓ​(1−β)σ−11​​
Capital accumulates and labor grows as
K˙=ιˉK Y−δK,Lt=L0 ent\begin{aligned} \dot K &=\bar\iota_K\,Y-\delta K,\\\ L_t &=L_0\,e^{nt} \end{aligned}K˙Lt​​=ιˉK​Y−δK,=L0​ent​
with investment share ιˉK\bar\iota_KιˉK​, depreciation rate δ\deltaδ, and population growth rate nnn.
**R &D in the model.** New ideas are produced from a fixed share RRR of output being invested, given by:
Q˙=qˉ RλQϕ,R=ιˉR Y.\begin{aligned} \dot Q &=\bar q\,R^{\lambda}Q^{\phi},\qquad \\\ R &=\bar\iota_R\,Y. \end{aligned}Q˙​R​=qˉ​RλQϕ,=ιˉR​Y.​
Where Q˙\dot QQ˙​ is the change in the idea stock, qˉ\bar qqˉ​ is a research-productivity constant, ιˉR\bar\iota_RιˉR​ is the share of output spent on R&D, and λ=1\lambda = 1λ=1 is the elasticity of new ideas to research input, and ϕ=−2\phi = -2ϕ=−2 (<0< 0<0) the returns to the existing stock, i.e., [ideas getting harder to find](https://web.stanford.edu/~chadj/IdeaPF.pdf).
With defaults λ=1\lambda = 1λ=1, ϕ=−2\phi = -2ϕ=−2 the growth rate is:
Q˙Q=θkqˉ ιˉR YQ 1−ϕ,\begin{aligned} \frac{\dot Q}{Q} &=\theta_k \frac{\bar q\,\bar\iota_R\,Y}{Q^{\,1-\phi}},\qquad \\\ \end{aligned}QQ˙​​​=θk​Q1−ϕqˉ​ιˉR​Y​,​
And the growth rate of capital productivity ψki\psi_{ki}ψki​ is given by:
ψ˙kiψki=θk Q˙Q=θkqˉ ιˉR YQ 1−ϕ,\begin{aligned} \frac{\dot\psi_{ki}}{\psi_{ki}} &=\theta_k\,\frac{\dot Q}{Q} = \theta_k \frac{\bar q\,\bar\iota_R\,Y}{Q^{\,1-\phi}},\qquad \\\ \end{aligned}ψki​ψ˙​ki​​​=θk​QQ˙​​=θk​Q1−ϕqˉ​ιˉR​Y​,​
So R&D is modelled at a fixed productivity qˉ\bar qqˉ​ from a fixed share ιˉR\bar\iota_RιˉR​ of output. The paper sets the combined term qˉ ιˉRλ=0.0001\bar q\,\bar\iota_R^{\lambda}=0.0001qˉ​ιˉRλ​=0.0001 to match idea growth Q˙/Q=1%\dot Q/Q=1\%Q˙​/Q=1% in 2020.
**100% automation.** The model leads to 100% automation in finite time when fˉ>0\bar f > 0fˉ​>0, with labor fully displaced and its income share falling to zero. Output collapses to a single-factor form Y=BKY = B KY=BK (linear in capital) whose productivity B∝QθkB \propto Q^{\theta_k}B∝Qθk​ keeps climbing as ideas accumulate.
**When does growth explode?** To pin down the long run behaviour one can look for a steady state where output and ideas are both growing at a constant rate. This either has a finite solution (the loop R&D feedback settles to steady growth) or no solution (where it explodes). There are two sides we can find:
_Idea side._ Constant idea growth Q˙/Q=qˉ ιˉRλ YλQϕ−1\dot Q/Q = \bar q\,\bar\iota_R^{\lambda}\,Y^{\lambda}Q^{\phi-1}Q˙​/Q=qˉ​ιˉRλ​YλQϕ−1 needs the YYY and QQQ powers to offset, λgY=(ϕ−1)gQ\lambda g_Y = (\phi-1)g_QλgY​=(ϕ−1)gQ​, so ideas growing at a fraction of output (a fraction because ideas get harder to find with ϕ<0\phi < 0ϕ<0):
gQ=λ1−ϕ gY.g_Q = \frac{\lambda}{1-\phi}\,g_Y.gQ​=1−ϕλ​gY​.
_Output side._ Output grows at the income-share-weighted growth of its two effective inputs, each input's growth being its productivity (rising with ideas, gB=θkgQg_B = \theta_k g_QgB​=θk​gQ​ and gA=θℓgQg_A = \theta_\ell g_QgA​=θℓ​gQ​) plus its quantity (for capital gK=gYg_K = g_YgK​=gY​ because of fixed saving & depreciation rate, and for labor its just nnn):
gY=sK∗ (θkgQ+gY)+(1−sK∗) (θℓgQ+n).g_Y = s_K^{*}\,(\theta_k g_Q + g_Y) + (1-s_K^{*})\,(\theta_\ell g_Q + n).gY​=sK∗​(θk​gQ​+gY​)+(1−sK∗​)(θℓ​gQ​+n).
Substituting the idea side for gQg_QgQ​ and collecting the gYg_YgY​ terms gives:
gY=Φ gY+n,withΦ=λ1−ϕ(θℓ+sK∗1−sK∗ θk).g_Y = \Phi\,g_Y + n, \qquad \text{with} \qquad \Phi = \frac{\lambda}{1-\phi}\left(\theta_\ell + \frac{s_K^{*}}{1-s_K^{*}}\,\theta_k\right).gY​=ΦgY​+n,withΦ=1−ϕλ​(θℓ​+1−sK∗​sK∗​​θk​).
So apart from population growth, the model's output growth happens through Φ\PhiΦ which represents the R&D loop's contribution. Each unit of output growth feeds back scaled by Φ\PhiΦ. Summing over timesteps we get a geometric series: gY=n(1+Φ+Φ2+⋯ )=n1−Φg_Y = n(1 + \Phi + \Phi^2 + \cdots) = \frac{n}{1-\Phi}gY​=n(1+Φ+Φ2+⋯)=1−Φn​.
* If Φ<1\Phi < 1Φ<1 each round shrinks and growth settles at a finite rate proportional to population growth.
* If Φ≥1\Phi \ge 1Φ≥1 it does not and no steady rate exists, so output runs to infinity in finite time. This is [hyperbolic growth](https://en.wikipedia.org/wiki/Hyperbolic_growth) with output XXX (normalized to 1 today) obeying X˙=gˉ XΦ\dot X = \bar g\,X^{\Phi}X˙=gˉ​XΦ, with gˉ\bar ggˉ​ today's growth rate. So the growth rate X˙/X=gˉ XΦ−1\dot X/X = \bar g\,X^{\Phi-1}X˙/X=gˉ​XΦ−1 itself rises with the level. Separating and integrating from today, ∫1XX−Φ dX=gˉ t\int_1^X X^{-\Phi}\,dX = \bar g\,t∫1X​X−ΦdX=gˉ​t, and solving for output gives X=( 1−(Φ−1) gˉ t )−1/(Φ−1)X = \big(\,1 - (\Phi-1)\,\bar g\,t\,\big)^{-1/(\Phi-1)}X=(1−(Φ−1)gˉ​t)−1/(Φ−1). With negative exponent, as the inside goes →0 \to 0→0 (which happens at 1=(Φ−1) gˉ t1 = (\Phi-1)\,\bar g\,t1=(Φ−1)gˉ​t) output XXX goes →∞ \to \infty→∞. So output going to infinity happens at t∞=1gˉ (Φ−1)t_\infty = \frac{1}{\bar g\,(\Phi-1)}t∞​=gˉ​(Φ−1)1​. For Φ=2\Phi = 2Φ=2 this is simply t∞=1gˉt_\infty = \frac{1}{\bar g}t∞​=gˉ​1​ which for ~3% growth today is ~33 years from now.
Jones & Tonetti's "Moore's Law Everywhere" calibration has Φ=2.27\Phi = 2.27Φ=2.27 (see box above for what this means) leading output to go infinite around 2060. Our main disagreement is not with this calibration, but with the lack of R&D automation and lack of unique treatment of AI capital (e.g., different in returns to R&D in AI vs. normal capital).
AI Futures Model
The **[AI Futures Model](https://www.aifuturesmodel.com)** features a tighter R&D automation loop, with combined AI-human research effort feeding into AI R&D (still with diminishing returns). We think some kind of similar treatment of AI, with modelling that is calibrated to how quickly AI has been improving and responding to increased inputs (human labor, compute, etc.) is needed for a growth model to model future growth in a way that more reasonably accounts for AI.
[aifuturesmodel.comOpen ↗![AI Futures Model — timelines & takeoff](/aifm-ss.png)Open the AI Futures Model ↗](https://www.aifuturesmodel.com)
AI Futures Model \- ai-2040.com
We think it would be exciting future work to modify the Jones & Tonetti model to (A) feature a unique AI factor of production, and/or (B) make the R&D loop more dynamic (and allowing more direct AI uplift on R&D). For the modelling we have done in this scenario, we instead rely on the [AI Futures Model](https://www.aifuturesmodel.com), and what we have added is more of an economics explorer, since it takes the AI automation trajectories as exogenous inputs on paths consistent with our AI Futures Model runs for [**AI 2040 default**](https://ai-rates-calculator-git-fixed-training-compute-lesswrong.vercel.app/p?base=daniel-04-02-26&acth=6.089920916003435&arts=3.4833731503601166&bgm=true&gy=1.103409&mttm=5.281602094603481&tam=1.54170045294956) and **[AI 2040 under Plan A](https://plan-a.aifuturesmodel.com/)**. See the [simple explorer](/supplements/economics-of-plan-a#the-simple-explorer) and [full model](/supplements/economics-of-plan-a#the-full-economic-model) below.
### Bottlenecks and other arguments
Beyond the two papers above, there is a more general class of arguments about economic growth not being explosive due to bottlenecks, or historical precedents. Overall we believe that many such bottlenecks will be either weak, routed around due to massive incentives, and/or will bite very late, and that AI will be disanalogous to historical precedents.
Objection: **Previous technologies haven’t led to explosive growth, and AI won’t be any different.**
We agree that previous technologies haven’t led to explosive growth (well at least the fast explosive growth we expect from AI, long run economic growth has actually been [better fit by a power law than an exponential](https://coefficientgiving.org/research/modeling-the-human-trajectory/)).
![](/economics-of-plan-a/longrungdp-nowatermark.png)
[Long run economic growth on a log plot shows that it has been faster than an exponential trend](https://ourworldindata.org/grapher/global-gdp-over-the-long-run?yScale=log)
The core difference is that past technologies automated a narrow domain (e.g. washing machines automated the task of physically cleaning clothes, the internet automated many communication channels), while AI could dominate humans at all (non-human-specific) tasks. So AI could also create new tasks that don’t exist today (e.g., datacenter maintenance, robot manufacturing, AI research) like past technologies did, but the disanalogy is that AI will also be able to automate those new tasks because it will dominate humans both cognitively and then physically (through robotics R&D and production). The humans will only be maintaining the datacenters, making the robots, doing the AI research, up until the point where AI and robots can do it better than them, then they no longer will be (at least once the AIs and robots are numerous enough, and again, absent strong, successful regulation, more on that below). The only tasks left for humans will be driven by human-preference which we think will make for a drastically different economy than today.
Objection: **There will be regulatory bottlenecks that prevent growth.**
We think that it is conceivable for regulation to massively slow AI explosive growth; and in fact, the economy in the Plan A scenario grows much more slowly than it would have absent regulation. However, the type of regulation that we think is required to create this type of slowdown involves explicit international coordination to all agree to hold off on explosive AI driven growth.
This is for two reasons.
1. **International competition:** if it’s the case that any country in the world could decide to allow their robots to grow exponentially and soon eclipse the industrial capacity of the rest of the world combined, there would be a massive incentive to do so, and it is hard to believe that all countries would accept this massive handicapping absent coordination.
2. **Internal reinvestment spirals:** regulation on AI companies seems unlikely to bite on what they do on internal R&D in a pre-emptive way. So if a company reaches a base of AIs and robots that are capable of self-replicating, AI companies might be able to vertically integrate a supply chain and rapidly outcompete the government (e.g., quickly building billions of robots in a desert offshore). Additionally, even if some sectors are highly regulated, currently highly regulated sectors aren’t the ones that drive explosive growth, e.g., R&D is generally not highly regulated and there will be a massive incentive to cut red tape wherever these are blocking explosive growth. (Note that in AI, we’ve already seen examples of skirted regulations with [copyright law violations](https://www.bakerlaw.com/bartz-v-anthropic/) and [xAI datacenter construction](https://newsletter.semianalysis.com/p/xais-colossus-2-first-gigawatt-datacenter)).
The regulation that we think is sufficient involves (a) explicit international cooperation to limit the kinds of R&D leading to AI and robots that can replace the work of, or don’t answer to a wide range of humans, and (b) limit explosive growth of robots and the amount of compute in the world through, for example, a cap-and-trade regime.
Objection: **There won’t be explosive growth because there will be long lags between AI being available and AI being actually adopted.**
So far, AI has diffused unprecedentedly quickly, even at its current capability level. When AIs are superhuman and there are billions or trillions of dollars on the line, we expect firms to quickly adopt AI. Even if many firms lag dramatically, only some firms need to adopt AI. The firms that do will grow explosively and will quickly become the majority of the sector. There will be strong market pressure to adopt superhuman AI quickly, and we believe this will be enough to cause unprecedentedly fast growth.
Objection: **There won’t be explosive growth due to the time it takes to build new physical infrastructure.**
Infrastructure bottlenecks can and will be solved by building more infrastructure. There will be massive economic pressure at the point where it is constraining AI or robot populations (at and approaching the point of full automation) to make this happen quickly (and maybe with partial help from AI and robots). Once robots can perform the relevant physical tasks, the rate of infrastructure construction again becomes roughly proportional to the robot population.
There will of course be serial time bottlenecks (9 women can’t make a baby in one month), and so too, an arbitrary number of robots will not be able to speed up the time it takes to build a robot factory to be infinitesimal. But for technology very similar to what already exists (production lines in factories, with mining), we don’t think serial construction times prevent short doubling times; see [Appendix A](/supplements/economics-of-plan-a#appendix-a-robot-doubling-times) for our estimates.
Objection: **Robot and semiconductor explosions will be bottlenecked by rare materials.**
We appear to be very far from exhausting physical resources on earth. This will slow down growth (on earth) eventually, but only after robots have mined substantial fractions of the earth’s crust. There is room for 30-100s of years worth of current extraction rates depending on the mineral before we even exhaust its proven reserves, and it’s plausible that with robot-powered mining, something on the order of 1% of earth’s crust would be straightforwardly reachable. Once we become bottlenecked by materials on earth, the robots could expand to space. Of course, as the 'materials get harder to find' the growth rate will slow, but it might still be very fast.
55y5.4My107y221My308y4.9My43y217ky35y3.0My125y15My38y17My231y103My5.9By10y100y1ky10ky100ky1My10My100My1By10ByIronAluminumPhosphateCopperNickelLithiumCobaltRare earthsSilicon
Years of minerals left at 2024 extraction rates \- ai-2040.com
Years until proven reserves run out
Years until 1% of continental crust runs out
Production & reserves: [USGS MCS 2025](https://pubs.usgs.gov/publication/mcs2025). Crustal abundance: [Wedepohl (1995)](http://apostilas.cena.usp.br/moodle/pessenda/projes/simposio/artigo7.pdf); continental-crust mass 2×1022 kg (CRUST2.0, [Pasyanos et al. 2007](https://ui.adsabs.harvard.edu/abs/2007AGUFM.V33A1161P/abstract)).
## The simple explorer
We propose a simple toy model with two factors of production: **humans** and **AI**. A continuum of tasks combines in a constant-elasticity-of-substitution (CES) aggregate with elasticity σ. A growing fraction of tasks is **automatable** : AI and robots can only do those, while humans can do any task, and within a task humans and machines are perfect substitutes.
The full model below adds capital, multiple sectors, prices, land, energy, and endogenous investment, but the qualitative story is the same one this toy makes visible.
## The full economic model
[Open the Full Explorer ↗Our full economic growth explorer, that models the Plan A scenario in a more hand crafted way, and features e.g., capital, robots, prices, land, and energy.](/supplements/econ-explorer)
The full version of the explorer is a detailed [economic growth model](/supplements/econ-explorer) that we built to model the economy in AI scenarios like Plan A. We have two main disclaimers about this model:
1. The model makes some assumptions about the world that we don’t endorse, plausibly contains bugs, and has very high uncertainty (i.e., we don’t trust the outputs to be accurate). That being said, we think it behaves relatively reasonably in these scenarios and we think it is the best existing model for the economic effects of the intelligence and industrial explosions.
2. The model’s preset is conditional on Plan A, our scenario in which the government takes substantial and effective steps to regulate AI and drastically slow things down. The second model preset shows something closer to the growth we expect in reality absent the regulatory interventions. Under the default world inputs, the model predicts real output growing ~5000x during 2033, which corresponds to a sustained doubling time of around 1 month.
3.0% / yrfrom 202410,0001,000100101Years till 20411B10B100B1T10T100T1,000T10,000T100,000T1,000,000TGWP ($)HistoricalPlan AAI 2040Plan D
Gross World Product \- ai-2040.com
### AI 2040 Plan A Scenario
Under the Plan A scenario, effective regulation limits growth in three ways.
1. It slows down R&D (particularly algorithmic research), leading to AIs that can fully substitute for all economically relevant work arrive around 2035 instead of 2031.
2. It stops AI and robots from becoming entirely self-sufficient (automating 100% of tasks) because of regulatory interventions to require human oversight and auditing.
3. The cap-and-trade regime limits the total number of robots and AI hardware to a controlled doubling time of around 6 months for most of the period.
#### Plan A assumptions
_Task automation fraction in Plan A. We split labor into cognitive labor and physical labor to show that cognitive labor is automated first because it can happen entirely with AIs running on the cloud, while the automation of physical labor requires a massive buildup of robots. In the Plan A scenario, there is 99% automation reached around 2035, with humans maintaining oversight roles thanks to successful regulation, and retaining a small fraction of economic output (still large in absolute terms)._
HumansAIRobotsdotted = Plan D (no deal)
10M100M1B10B100B1T10T100THumansAIRobotsAI (Plan D)Robots (Plan D)2025203020352040count (log scale)
_AI and robot populations in Plan A. Before 2029, these reflect unconstrained projections. After that, they reflect the Plan A regulations, where both R &D is restricted, and production is capped from 2032 onwards to a roughly 6 month doubling time (with AI hardware production slowed after 2035 to a ~3y doubling time)._
Human, AI & Robot Population \- ai-2040.com
#### Plan A economic outcomes
Overall, partially informed by our economic model, we expect the main economic effects of Plan A to include:
* **An average of roughly 90% yearly growth in real output from 2032 to 2037.** Broadly speaking, throughout this period AI and robots are capable of doing the vast majority of economic tasks, and are allowed to roughly double every six months. Rather than doubling the economy every six months, output increases around 90% per year on average (because of the modelling choice to have capital and human labor partially bottleneck growth).
* **Most of the economic output becomes driven by AI and robots.** Income shares in the economy shift greatly towards AI and robots as they become capable of doing more and more tasks, and also become more numerous.
Human cognitiveHuman physicalAIRobots0%25%50%75%100%2025202620272028202920302031203220332034203520362037203820392040
Share of US Labor Output \- ai-2040.com
* **Enormous AI and robot permit revenues redistributed through a “Citizen 's Dividend”.** To a first approximation, the combined AI hardware and robot cap-and-trade permits are the central bottleneck to the economy doubling in size (on average) every year. Naively, that implies the total value of the permits (if well-priced by a competitive market) should be around the size of the current economy, because that is how much additional GDP they will add to the following year. This is reasonable if the AI and robot companies are relatively myopic about the value of the AI and robots (i.e., they mostly base their decision based on 1 expected year of output), which we think may be true under these levels of growth, given potentially fast progress driving obsolescence of tech that is even a few years old. Our [cost modelling](/econ-explorer/cost-model) suggests that the actual manufacturing costs will be a small share of GDP which suggests the permit costs will dominate. So we think the permit revenues will roughly equal the annual GDP when the economy is close to doubling every year.
2030Income tax87%Corporate tax6%Sales tax7%$10Ttotal revenue2032Compute permits57%Robot permits23%$65Ttotal revenue2034Compute permits26%Robot permits74%$180Ttotal revenue
US Tax Revenue Sources \- ai-2040.com
In Plan A, most of the US permit revenue share (75% in 2035) is redistributed to the US population as a Citizen’s Dividend, resulting in roughly $1M/yr per person in 2035 and around $10M/yr in 2040.
Wages and capital income from stocks would still be unequal, but with the equally distributed Citizen’s Dividend overall income equality would improve drastically, and the poorest Americans would see a drastic elevation in income.
203020352040$1K$10K$100K$1M$10M$100M$1B
_Per-person income (2025 dollars)._
US Income Distribution in Plan A \- ai-2040.com
* **Relative prices in the economy shift, with goods and services generally falling in price, while land, positional goods and human-bottlenecked outputs rising.** In Plan A, both cognitive and physical labor from AIs and robots becomes vastly more abundant. Specifically, cognitive labor is 100x more abundant by 2033, and the same is true for physical labor by 2039. So anything the AI and robots can do (which is essentially to make every single good and provide every single service that exists today) will, generally speaking, become a lot cheaper in real terms. Throughout the 2030s in Plan A, AIs are relatively more abundant compared to robots, which means that the price of renting a human-equivalent robot is much higher than a human-equivalent AI worker. In some years, the gap we model is around 100x.
_Rental costs for a full-time human-equivalent in any automated cognitive or physical domain over the course of Plan A._
In 2035, the cost of 1 FTE of labor in the average cognitive domain is $3,000 and in the average physical domain around $200K. These prices would be lower if not for the regulations limiting the amount of AI chips and robots being produced and deployed. In many domains such as law and medicine, since demand is relatively inelastic, we expect the GDP contribution to go down dramatically. In 2035, about 10x as much health care is consumed relative to 2025, but since it's ~100-1000x cheaper, the dollar-wise GDP contribution is 1-10% of what it was.
For fixed things like land, positional goods (like New York apartments) and human-bottlenecked services (e.g., human day care), we expect the relative pricing to increase, because their supply hasn’t changed. A (very rough model) for the price of land is that there are three main factors: (i) a larger share of spending gets devoted to land (10% -> 50%), and (ii) average spending increases due to the AI boom (100x), (iii) through repurposing of land (e.g., agriculture to housing) and deregulation, available land increases 10x. So with 500x increase in spending and only 10x increase in supply the price should increase 50x. The economic model has a more complicated model for land with more assumptions (which we don’t necessarily endorse), and produces concrete rental prices for urban, rural, and agricultural land separately. On average, land prices increase around 10-100x between the types.
The model predicts land costs increasing around 100x, while building costs only increase 2x. This implies that the price of housing will become dominated by the price of the land (as opposed to the cost of building housing), so dense housing (e.g. skyscrapers) will become relatively much cheaper than sparse housing. For single family homes, (where the current land vs. build cost share is usually around 40/60), this means that their price is up around 40x on average, but 97% of the home’s value on paper is the land. For 100 floor skyscrapers, land is currently typically 5% of the cost, so the price only goes up 7x. So it’s now commonplace for 100-floor skyscrapers to be built rapidly near major cities almost entirely by robots to provide cheaper housing. In 2035, the Citizen’s Dividend is roughly $1M per person, and it should only take around 15% of it to afford a typical high rise apartment near the city. Also, in deregulated places robots should be able to build far improved transportation that allows for rural homes to also become attractive, where we think the land will be around 10x cheaper.
We have an explorer in the [scenario exploration tab](/econ-explorer/scenario-exploration) for what the median consumer might be able to afford. The default assumes 15% of their spending is on urban land, 15% on rural land, 35% is on AI labor and 35% is on robot labor. Since AI labor is relatively cheap, by 2034 it becomes the case that each American can afford the equivalent of a mid-sized corporation working for them personally. Robots are more expensive and slower to scale, so in 2034 the median American has an average of a single robot working for them. By 2039, everyone can afford a construction team of robots, four hectares of rural land and a small urban block.
* **There are high real interest rates around 100-200% and depending on monetary policy choices, either massive deflation or large money creation (to keep inflation around 2%).** We believe interest rates will [increase drastically](https://www.basilhalperin.com/papers/agi_emh.pdf) in Plan A as companies are willing to bid up the robot and AI production permits up to levels close to all of the current year’s GDP and pay interest up to 100-200%: since the robot and AI permits can be converted (barring the comparatively negligible costs of actually producing the AI and robots) into quantities of AI and robots that increase economic output 100-200%. In our modelling, this happens because of explosive growth in cognitive and physical labor, combined with our modelling decision to have production combine labor and capital with limited substitutability (a CES aggregate with elasticity 1.1, close to cobb-douglas). Because capital cannot fully stand in for labor at that elasticity, the exploding effective labor supply makes each unit of capital much more productive, so the marginal product of capital (MPK; how much extra output you get from one extra unit of capital per year) increases greatly throughout Plan A. Since in equilibrium the real interest rate equals MPK minus depreciation, this implies real rates rise similarly throughout the takeoff. We model the savings rate being responsive (but inelastic) with respect to these rising interest rates, but are not confident in the particular parameter we chose.
We imagine two mainline monetary policy responses being possible in this situation:
(1) If the government chooses to keep the money supply relatively fixed, then there would be a fixed amount of dollars chasing an increasingly abundant number of goods and services, so each dollar would be spread over a growing number of goods and services, meaning the average price is lower (each dollar buys more goods and services), i.e., leading to massive deflation. The problem with this could be a [deflationary debt spiral](https://en.wikipedia.org/wiki/Debt_deflation), where the AI and robot companies can’t pay back loans in dollars because the robots and AIs are worth nominally less than the loans written the year before. One way to solve this could be for the loans to be denominated in AI and robots, so the companies pay back the loans with some percentage of the AI and robots instead of dollars. (2) Another possibility could be that they attempt to maintain an ongoing annual 2% inflation rate. This would involve a lot of money creation to keep up with the explosive real growth, and there might be high measurement difficulty and time lags making it hard to create the money at the correct level.
Note this wouldn’t change the fact that the consumer perceived inflation rates would depend dramatically on the basket of goods chosen, as the relative price swings systematically bring down the typical 2025-basket-of-goods dominant goods and services, with the AI and robot automation disproportionately affecting these areas. People will be able to afford many more cognitive-labor-bottlenecked services (e.g. financial advisors, lawyers, travel agents), and manufacturing bottlenecked goods (e.g. computers, cars, clothes, nice food), but less land, and less positional goods or non-automatable things (e.g., human day care). Because of the relative price swings, even if average inflation is 2%, some sectors will have large price increases (inflation) and others will have large price decreases (deflation).
### AI 2040 Default World
In the AI 2040 default world, R&D and production quantities are mostly unconstrained by regulation, so there is 100% automation reached by early 2031. With our model’s naive endogenous investment and cost modelling explained below, we get 5000x growth in 1 year during 2033.
We model the investment required for robot and AI production (cost per quality-adjusted unit) by stacking pure “learning by doing” or economies of scale modelling (using [Wright's law](https://en.wikipedia.org/wiki/Learning_curve), i.e., cost / unit) combining multiplicatively with design improvements (using [Jones' model](https://en.wikipedia.org/wiki/Jones_model), i.e., quality / unit). The idea is to reflect the separate gains from chip production efficiency (more chips of a fixed design being produced per dollar by fabs, i.e., progress that TSMC makes) with chip design efficiency (producing more final hardware performance per dollar from a fixed fab, i.e., progress that Nvidia makes). More on this in the cost model section of the [model explanation](/econ-explorer/model-explanation). The endogenous investment mode then allocates investment between AI hardware, robot hardware, and capital based on a naive myopic return (i.e., so that each earns the same real return at the margin once you net out depreciation). We don’t necessarily endorse this as a way to model endogenous investment, but we’ve chosen it for simplicity for this illustrative model, and (as previously mentioned) we think the outputs are plausible because of the core arguments made in sections 1 and 2, rather than because of this modelling.
A feedback loop happens where AI and robot investment leads to more cognitive and physical labor, which can then be reinvested in R&D and more production efficiency. AI and robot investment gives higher returns as AI and robots become more capable (according to the exogenous trajectory that assumes full automation in 2031). The explosion in R&D labor flows through to plummeting production costs for AIs and robots.
## Appendix A: Robot doubling times
There is a nascent but growing literature on this topic. In particular [this Forethought report](https://www.forethought.org/research/the-industrial-explosion), which splits its doubling time estimates into three phases:
1. _Initial_ (One-time gain from AI-directed human labor): on the order of **~1 year**.
2. _Acceleration_ : historical experience curves suggest **the number of robots will double 1-5 times before the robot growth rate doubles.**
3. _Peak / maximum_ (biological anchors): **days to weeks** as an upper bound (fruit flies double in days, rats in ~6 weeks; bacteria in hours but probably too cognitively basic to bootstrap).
Out of these we broadly agree, except we think the acceleration phase is unlikely to match historical experience curves: those curves are evidence from human industries, and our best guess is that at full-automation, AI-driven R&D will be able to drive a faster experience curve for robotics. Our best guess is that it could look like Moore’s law (0.2 doublings per halving of cost) or faster, meaning that within a few months, the doubling time might halve multiple times, quickly approaching the ‘peak’ phase. Especially in the regime where AI continues to improve qualitatively, we also think that for the ‘peak / maximum’ phase the actual ceiling might be minutes or hours, because of [bacteria with ~10 minute doubling times](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3751608/) providing an existence proof of something a superintelligent AI may be able to match with a nanotechnology design.
[This blog post](https://defensesindepth.bio/ai-industrial-takeoff-part-1-maximum-growth-rates-with-current-technology/) by Damon Binder assumes _full automation but no other technological improvement_ , framing this result as a conservative assumption, and finds **headline doubling time of ~1 year ignoring consumption** (with a plausible range of roughly 8 months to 2 years depending on assumptions). He computes the Von Neumann growth rate (the maximum rate at which the economy's physical capital can reproduce itself if all output is reinvested) directly from US BEA input-output tables, capital stocks, and depreciation data. He first assumes labor is free (abundant AI and robots), and then adjusts for actual AI and robot costs, resource depletion, construction time lags, and consumption to get estimates ranging more in the 1-2 year range.
Another recent piece is [this Epoch report](https://epoch.ai/blog/how-fast-could-robot-production-scale-up), which summarizes current 2025 robotics trends (6 months for humanoids, albeit at very low volumes), acceleration under historical demand shocks (e.g., WW2 and Ukraine), and current factory-build latencies (6-9 months in China). This piece is explicitly about near-term scaling (i.e., excluding AI-directed scale-up), which we think establishes a strong baseline for an initial doubling time of around 1 year at full automation.
See also older work in [the “Industrial Explosion” section of another Forethought report](https://www.forethought.org/research/preparing-for-the-intelligence-explosion#the-industrial-explosion), [this Open Philanthropy report](https://www.openphilanthropy.org/research/could-advanced-ai-drive-explosive-economic-growth/) and [this Epoch blog post](https://epoch.ai/blog/explosive-growth-from-ai-a-review-of-the-arguments), as well as our own arguments in the [AI 2027 robot economy doubling times box](https://ai-2027.com/slowdown#slowdown-2028-03-31):
[ai-2027.com/slowdownopen ↗](https://ai-2027.com/slowdown#slowdown-2028-03-31)
Robot economy doubling timesSlowdown ending · March 2028
During World War II the United States and many other countries converted their civilian economies to total war economies. This meant converting factories that produced cars into factories that produced planes and tanks, redirecting raw materials from consumer products to military products, and rerouting transportation networks accordingly.
We are imagining something similar, except faster because superintelligences are directing and managing the whole process. Roughly speaking, the plan is to convert existing factories to mass-produce a variety of robots (designed by superintelligences to be both better than existing robots and cheaper to produce) which then assist in the construction of newer, more efficient factories and laboratories, which produce larger quantities of more sophisticated robots, which produce even more advanced factories and laboratories, etc. until the combined robot economy spread across all the SEZs is as large as the human economy (and therefore needs to procure its own raw materials, energy, etc.) By that point, the new factories will have produced huge quantities of robotic mining equipment, solar panels, etc. in anticipation of needing to meet demand far greater than the legacy human economy can provide.
How fast would this new robot economy grow? Some reference points:
* The modern human economy doubles every twenty years or so. Countries that have developed especially rapidly (e.g. [China)](https://www.macrotrends.net/global-metrics/countries/CHN/china/gdp-gross-domestic-product) sometimes manage to double their economies in less than a decade.
* A modern car factory produces roughly its own weight in cars in less than a year. Perhaps a fully robotic economy run by superintelligences would be able to reproduce itself in less than a year, so long as it didn’t start to run out of raw materials.
* Yet that seems like it could be a dramatic underestimate. Plants and insects often have “doubling times” of far less than a year—sometimes just weeks! Perhaps eventually the robots would be so sophisticated, so intricately manufactured and well-designed, that the robot economy could double in a few weeks (again assuming available raw materials).
* Yet even that could be an underestimate. Plants and insects are operating under many constraints that superintelligent designers don’t have. For example, they need to take the form of self-contained organisms that self-replicate, instead of an economy of diverse and more specialized vehicles and factories shipping materials and equipment back and forth. Besides, bacteria and other tiny organisms reproduce in hours. It’s possible that, eventually, the autonomous robot economy would look more like e.g. a new kind of indigestible algae that spreads across the Earth’s oceans, doubling [twice a day](https://enviroliteracy.org/how-long-does-it-take-for-algae-to-multiply/) so that it covers the entire ocean surface in two months, along with an accompanying ecosystem of predator-species that convert algae into more useful products, themselves fed into floating factories that produce macro-structures like rockets and more floating factories.
Obviously, all of this is hard to predict. It’s like asking the inventors of the steam engine to guess how long it takes for a modern car factory to produce its own weight in cars, and also to guess how long it would take until such a factory first exists. But economic growth rates have accelerated by [multiple orders of magnitude](https://wiki.aiimpacts.org/featured_articles/precedents_for_economic_n-year_doubling_before_4n-year_doubling) over the [course of human history](https://www.openphilanthropy.org/research/modeling-the-human-trajectory/), and it seems plausible to us that after superintelligence they will accelerate by orders of magnitude more. Our story depicts economic growth accelerating by about 1.5 orders of magnitude over the course of a few years.
quoted from [AI 2027](https://ai-2027.com/slowdown#slowdown-2028-03-31), the box as published in the Slowdown ending (March 2028)
## Appendix B: Energy model
Global energy consumption in 2025 was around 19.0 terawatt-years (TW-yrs). In the Plan A scenario, there is roughly 200x cumulative GWP growth by 2040 (on our [default model assumptions](/econ-explorer/world)). If the energy intensity of the economy (ratio of GDP to energy consumption) stayed constant, then energy consumption would also increase by that same factor, but historically we have observed a decrease in energy intensity since 1920. The headline numbers below are computed directly from the model's Plan A output and our elasticity assumption, so they stay in sync with the [economic model explorer](/supplements/econ-explorer) defaults.
Energy ×44GDP ×109×1×2×5×10×20×50×100×200182018501880190019201950198020002024
Global Energy vs. GDP \- ai-2040.com
Global primary energy and GDP, both indexed 1820 = 1. 1820–2016: [Bercegol & Benisty (2020)](https://arxiv.org/abs/2008.10967). 2020/2024: [IEA Global Energy Review 2025](https://www.iea.org/reports/global-energy-review-2025).
We expect this energy decoupling to continue, because the drivers of growth (AI, robots and accompanying capital) should all become more energy efficient per unit of economic output (AI hardware in particular has a [strong power efficiency trend](https://epoch.ai/blog/trends-in-machine-learning-hardware#energy-efficiency)). In our [energy model](/econ-explorer/energy-model), we assume the energy decoupling trend of the last few decades will continue to become more aggressive, and energy consumption will grow at roughly 40% the pace of economic growth (we call this _energy growth / GDP growth_ the energy elasticity of GDP; ε = 0.4). We are highly uncertain about this value, and find values between 20% and 70% to be plausible (80% CI; conditional on this economic growth trajectory).
GDP ×203Energy ×8×1×10×100×10002025203020352040
Energy vs. GDP in Plan A \- ai-2040.com
Global energy and GDP under Plan A, indexed to 2025 = 1. GDP is the scenario-growth-model Plan A Output Y; energy via E ∝ Yε, ε = 0.40 (energy grows at 40% of the GDP rate).
**Energy model at a glance**
computed from the Plan A GWP trajectory and the econ-explorer energy defaults (E ∝ Yε).
2025 primary energy| 19 TW-yrs
---|---
Plan A GWP growth (2025 to 2040)| x203
Energy elasticity of GDP (ε)| 0.40
Energy growth at ε = 0.40| x8.4
2040 primary energy (ε = 0.40)| 159 TW-yrs
2040 range (ε = 0.20 to 0.70)| 55 TW-yrs to 784 TW-yrs
Primary energy scales as GWPε off the 2025 baseline of 19 TW-yrs. The GWP multiple is the scenario-growth-model Plan A Output Y; ε and the baseline mirror the econ-explorer defaults, so this stays in sync.
An energy elasticity of GDP of 40% leads to energy consumption on earth growing only around 8x during Plan A even as GWP grows by roughly 200x. The resulting 2040 primary-energy total (central case, and the 20% to 70% elasticity range) is computed in the summary above, and updates automatically with the economic model's defaults.
Depending on the source used for this energy, there may be significant implications for the earth’s temperature. There are two separate things to consider (1) a net increase in waste heat on earth, and (2) emissions, particularly CO2 emissions, leading to temperature increases.
* **Waste heat.** Solar panels convert energy that was arriving on earth anyway from the sun, so they only change earth’s net energy if they change the reflectiveness (albedo) of the earth’s surface in a major way. Using another energy source like nuclear or fossil fuels, on the other hand, creates a direct increase in waste heat by releasing new energy to the atmosphere that was locked up in the earth’s crust. That said, even at the higher end of the levels we may reach by 2040 in the Plan A scenario, waste heat on its own does not become a major consideration.
ε=0.20 (55 TW)ε=0.40 (159 TW)ε=0.70 (784 TW)Plan A 2040 (80% CI)today (~19 TW)10× today100×1000×1%10%solar surface:Nuclear / FusionSolar on landSolar on ocean (cooling)+0°C+2°C+4°C+6°C+8°C+10°C+12°C10 TW100 TW1 PW10 PWSurface temperature increase ΔT (°C)Sustained energy production (TW, log scale)
Warming from Waste Heat \- ai-2040.com
Stefan-Boltzmann no-feedback equilibrium. Heat factors (W added per W generated): nuclear/fusion 1.0; solar on land +0.56; solar on ocean −0.089. Solar assumes 50% panel efficiency, panel albedo 0.10, surface albedos 0.35 (land) / 0.06 (ocean).
* **Carbon emissions.** If fossil fuels were used as the sole source for the energy scale up, carbon emissions would become a significant problem by 2040, unless there were significant mitigations taken (e.g., removing the CO2 through direct air capture), with the equilibrium surface temperature increasing to +3.0oC+3.0^oC+3.0oC over pre-industrial levels, up from +1.8oC+1.8^oC+1.8oC today.
# Warming vs. Cumulative CO₂
pre-industrial (280 ppm, 0°C)today (427 ppm, +1.8°C)+2.4°C55 TW10th %ile+2.8°C159 TW50th %ile+4.3°C784 TW90th %ilePlan A, 2040assuming 100% fossil fuelEquilibrium ΔT+0°C+1°C+2°C+3°C+4°C+5°C+6°C01k2k3k4k5k6kEquilibrium ΔT (°C)Atmospheric CO₂ above pre-industrial (GtCO₂)
Equilibrium warming equations [Myhre et al. (1998)](https://agupubs.onlinelibrary.wiley.com/doi/10.1029/98GL01908) find the climate converging to a given temperature for a given atmospheric CO₂ stock. Today's observed +1.3–1.4°C is below the derived +1.8°C equilibrium value, in line with ([Geoffroy et al. 2013](https://journals.ametsoc.org/view/journals/clim/26/6/jcli-d-12-00195.1.xml)), where ~55% of equilibrium warming materialises within 10 yrs of CO₂ stabilisation, ~70% in 100 yrs, ~99% by 1000 yrs. Plan A markers assume 100% fossil-fuel generation at 2.5 GtCO₂/TW-yr, with the ~45% airborne fraction staying in the air ( [Friedlingstein et al. 2025 / Global Carbon Budget](https://essd.copernicus.org/articles/17/965/2025/)).
Therefore, we think there should be CO2 emission mitigation policies agreed to globally. Thankfully we think proven [direct air capture (DAC)](https://www.iea.org/energy-system/carbon-capture-utilisation-and-storage/direct-air-capture) would provide an affordable path to mitigating CO2 increase, especially with help from AI and robot labor during the 2030s. Two specific ideas we have for carbon capture policy include:
1. **Private cap-and-trade market** (no government subsidy necessary), where any fossil fuel emitters globally are required to offset 100% of their emissions with equivalent carbon capture credits (with DAC companies making revenue by capturing carbon and selling the credits to the carbon emitters).
2. **Auctioned carbon removal subsidy** (government subsidy for net carbon removal), where the government(s) set the goal of not just capping emissions, but actually net removing carbon from the atmosphere, with the goal of returning to pre-industrial levels. They can set a “willingness to pay” curve, based on the social cost of carbon. We imagine something like the following curve being sufficient for affordably returning the earth to pre-industrial CO2 levels by 2040 with around $85T global government spending in total from 2033-2040, or around 0.1% of annual GWP on average during that period.
pre-industrialtoday2025 expected DAC cost ($600B/Gt)2030 expected DAC cost ($400B/Gt)2035 expected DAC cost ($100B/Gt)2040 expected DAC cost ($30B/Gt)SCC$-200B$-100B$0B$100B$200B$300B$400B$500B$600B$700B01k2k3k4k5k6k7kCarbon Capture Cost ($B per Gt removed)Total atmospheric CO₂ (GtCO₂)
Carbon Capture Subsidy \- ai-2040.com
Subsidy curve (WTP per GtCO₂ removed)Expected Direct Air Capture (DAC) cost by yearPlan A path
**Total Plan A subsidy: ~$ 85T** (price × Δstock, summed). Prices: DAC × (1+10%), path sits on SCC curve. Sources: [EPA 2023](https://www.epa.gov/environmental-economics/scghg), [IEA DAC 2022](https://www.iea.org/reports/direct-air-capture-2022/executive-summary), [CDR.fyi 2025](https://www.cdr.fyi/blog/direct-air-capture-market-snapshot-2025).
Under some naive energy cost modelling that AI and robot labor accelerate progress in energy costs, and assuming a CO2 cap-and-trade forcing net neutral carbon emissions is enacted in 2035, we find the following energy cost, associated investment, and resulting energy mix.
1.
We’ve written about our reasoning for this assumption elsewhere, especially in our [timelines and takeoff model](https://www.aifuturesmodel.com/), and so will take it for granted here. Our team’s estimates for the probability of this outcome range from roughly 70% to 90%.
2.
For example, one common model of growth is the Cobb-Douglas production function, which models growth as Y=A∗Kα∗L1−αY = A* K^\alpha * L^{1-\alpha}Y=A∗Kα∗L1−α, where A and α\alphaα are constants, L is labor, and K is capital. A typical α≈13\alpha \approx \frac{1}{3}α≈31​ so overall growth scales with L23L^{\frac{2}{3}}L32​. Plus more people would build more capital, so in steady state output should double barring e.g., natural resource constraints, which are probably mild for many orders of magnitude of growth.
3.
This is somewhat of an oversimplification because raw compute isn’t the only bottleneck on number of AI copies; in reality it is a complex combination of actual ‘compute’ (FLOP/s), but also memory bandwidth, memory capacity, and interconnect. However these factors don’t substantially change the bottom line.
4.
Of course, we expect these trends to change, and more detailed and mechanistic models such as the AI Futures Model predict that the compute efficiency growth rate will vary over time.
5.
Inference compute may change as a fraction of total compute, but for our simple model we’re just assuming that the share is flat at the point of full automation.
6.
AIs today are mostly being scaled to be smarter, not just more numerous, so this isn’t happening in practice. After the point where AIs can perfectly substitute for human labor, increasing intelligence will also be happening, so this argument is only a lower bound for the economic impacts.
7.
This argument holds unless regulation massively slows down the possible growth. However, unilateral regulation is insufficient because any country that allowed explosive AI growth will gain an insurmountable economic edge. Therefore, regulation to prevent explosive economic growth must involve international coordination.
8.
The full “rapid progress” scenario from the FRI report is: By the end of 2030, in the rapid-progress world, AI systems are capable of competing with the best human minds and workers, and can surpass them. Human creativity and leadership remain valued, but mostly for setting high-level vision— day-to-day execution can be left to silicon-based systems. Autonomous researchers can collapse years-long research timelines into days, weeks, or months, creating game-changing technologies, such as materials that revolutionize energy storage, or bespoke cancer cures. No human freelance software engineer can outperform AI. The same goes for customer service (e.g., call center and support chat), paralegal, and administrative workers (e.g., bookkeepers or scheduling assistants). Indeed, models have become so capable that AI can create an album of the same caliber as the Grammy Album of the Year. Additionally, a single AI agent can generate a Pulitzer- (or Booker Prize-) caliber novel according to current (2025) standards, adapt the book into an engaging two-hour movie, negotiate the resulting book and movie contracts, and launch the marketing campaigns for both while its sibling agents manage the book publishing company and movie studio at the level of highly competent CEOs. Not only do Level-5 robo-taxis exist, but they are, on average, 99.9% safer than human piloted cars and can venture anywhere off-road that a competent human driver can. Meanwhile, robots can navigate an arbitrary home anywhere in the world, make a cup of the most popular local hot beverage, clean and put away the dishes according to the local custom, fix any plumbing issues that arise while they’re doing the dishes—and they can do it all faster and more reliably than most humans and without human guidance. Robots in advanced factories can autonomously perform the full range of tasks requiring the highest levels of dexterity, coordination, and adaptive decision-making.
9.
For example, there could be a R&D function combining R&D labor from a human population with labor from an AI population with forecasts of how useful/capable that AI labor will be (e.g., directly modelling the productivity of AI across R&D labor tasks).
10.
For example, see AI coding time-horizons [doubling every ~four months](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) recently, with similar results (though more uncertain) [in other domains](https://metr.org/blog/2025-07-14-how-does-time-horizon-vary-across-domains/), along with the population/efficiency doubling times explained in [section 1](/supplements/economics-of-plan-a#core-arguments-for-explosive-growth), we think Q˙/Q\dot Q/QQ˙​/Q should be more responsive to increased R&D automation.
11.
The continental crust makes up the continents _plus_ the continental shelves (shallow seas out to roughly the 200 m depth break). It has an average thickness of 35-40 km and covers about 40-45% of earth’s surface area. This suggests that by mining around 4% of earth’s surface area (total area around the size of Australia), to a depth of 4km, would consist of about 1% of the earth’s continental crust.
12.
The default world inputs assume a capability trajectory like [this one](https://www.aifuturesmodel.com/p?base=eli-12-29-25&acth=5.386766760341415&arts=3.372873086588689) from the AI Futures Model, where AI and robots can fully automate (100%) of cognitive and physical tasks around the beginning of 2031.
13.
Definitions for the task universe and automatable vs. automated are [here in the model explanation](/econ-explorer/model-explanation/automation-definitions).
14.
We model the relationship between labor and capital (things like factories, tractors, land, raw materials, etc) as a CES aggregate with elasticity σ = 1.1, slightly on the substitutes side of [Cobb-Douglas](https://en.wikipedia.org/wiki/Cobb%E2%80%93Douglas_production_function) (σ = 1, where the labor and capital income share ratio would stay flat). This is a judgment call that could very well be wrong! After full automation, what counts as “capital” is a blurry category, especially since we count robots separately. In reality, we expect AI and robots to substitute with many kinds of capital (e.g., robots replacing equipment, or AI labor leading to R&D that obviates the need for a lot of existing capital), while other kinds of capital bottleneck growth (e.g., land, especially with land regulation; raw materials; etc.) Our best guess is that these forces net out mildly on the side of substitutability, which is why we set σ slightly above 1. Note this runs contrary to historical measurements, which generally find complementarity (meaning elasticity parameter σ < 1; [Chirinko, 2008](https://ideas.repec.org/a/eee/jmacro/v30y2008i2p671-686.html); [Knoblach et al., 2020](https://onlinelibrary.wiley.com/doi/10.1111/obes.12312); [Gechert et al., 2022](https://doi.org/10.1016/j.red.2021.05.003)), but we think the capital stock of an AI-driven economy is different enough in kind from the capital behind those estimates that mild substitutability is the better guess.
15.
Land was 40%, increases 100x (40%*100), housing was 60%, increases 2x (60%*2), which leads to a total land (40) + housing (1.2) cost 41.2x higher than today. Of which 97% (40/41.2) is the land’s cost.
16.
Land’s 5% increases 100x (5%*100), and the building’s 95% increases 2x (95%*2), which leads to a total land (5) and housing (1.9) cost 6.9x higher than today. Still with the land being 72% of the cost.
17.
Our story depicts the conversion process going about 5x faster. We think this is a reasonable guess, taking into account bottlenecks etc., for how fast this conversion could go if a million superintelligences were orchestrating it. Of course we are very uncertain.
18.
Possibly also more advanced sources of energy, such as fusion power.
19.
Quick napkin math: The [Empire State Building](https://en.wikipedia.org/wiki/Empire_State_Building) has an area of 2.77m sq ft, and weighs 365k tons. Gigafactory Shanghai has an [area of 4.5m sq ft](https://en.wikipedia.org/wiki/List_of_Tesla_factories) and [produces 750k](https://en.wikipedia.org/wiki/Gigafactory_Shanghai) vehicles per year, mostly Model 3’s and Model Y’s, which weigh about two tons each. Presumably the Empire State Building has a higher mass-to-square-foot ratio than the Shanghai Gigafactory (since it is vertical rather than horizontal and needs stronger supports) so if anything this underestimates. Thus it seems that a factory which probably weighs well less than a million tons is producing 1.5 million tons of cars each year.
20.
We don’t think it would run out. Initially the robot economy would be dependent on human mines for materials. But by the time it outgrows these sources, the millions of superintelligences will have prospected new mines and developed new technologies to exploit them. Imagine e.g. undersea mining robots that strip-mine the seabed for rare minerals, new chemical processing pathways that more efficiently convert raw ore from above-ground stripmines into useful raw materials…
21.
If the current economy doubles every twenty years, one order of magnitude faster would be a doubling in two years, two orders of magnitude faster would be a doubling in 0.2 years, and so forth. The hypothetical superintelligent-algae-economy described above would be about four orders of magnitude faster growth than the current human economy.
22.
Using the [Energy Transition Institute 2024](https://www.energy-transition-institute.com/article/statistical-review-of-world-energy-2025) direct-method total of 592 EJ, and adding the [IEA’s 1.3% 2025 growth estimate](https://www.iea.org/reports/global-energy-review-2025) (+8 EJ) gives roughly 600 EJ in 2025, or ~19.0 terawatt-years. Where a terawatt-year is the amount of energy equivalent to 1 terawatt of power running for one year. Note: 1 TW-yr = 31.5 EJ = 8,760 TWh.
23.
On average, energy growth has moved at 70% the pace of GDP growth since 1920. In recent decades (since 1980), it has moved at 60% the pace of GDP growth, and in the last 10 years, it moved at 52% the pace.
24.
We use data from [Bercegol & Benisty (2020)](https://arxiv.org/abs/2008.10967) and [IEA Global Energy Review 2025](https://www.iea.org/reports/global-energy-review-2025) collected to view [here](https://docs.google.com/spreadsheets/d/1yxCYv4lsoVGJqdwh2Oe28ngiAlO-ZyW4Ng5HmraaX3s/edit?gid=67630334#gid=67630334).
25.
Energy scales as GWP^ε; at ε = 0.40 and the Plan A GWP multiple of ~200x by 2040, that is ~200^0.4 ≈ 8x. The exact value tracks the model output shown in the summary table above.
sources/ai2040-pages/faq.md
+101
View File
@@ -0,0 +1,101 @@
---
url: https://ai-2040.com/supplements/faq
---
## [AI 2040](/)
≡[PDF](https://ghjyjqzwz4.ufs.sh/f/9qHa0cBclQ7sf6FVzqbp2gHFXC1IbGS5ZA7DmNBusfM4YJRP)
[Supplements](/supplements)
Plan A Basics
[FAQFAQ](/supplements/faq)[How Plan A solves our 5 biggest problemsHow Plan A solves our 5 biggest problems](/supplements/how-plan-a-solves-our-5-biggest-problems)
Plan A Policies
[Covert AI ProjectsCovert AI Projects](/supplements/covert-ai-projects)[Verification PlanVerification Plan](/supplements/verification-plan)[Transparency PlanTransparency Plan](/supplements/transparency-plan)[Capability Scaling StrategyCapability Scaling Strategy](/supplements/capability-scaling-strategy)[Security in Plan ASecurity in Plan A](/supplements/security-in-plan-a)
Analysis of Plan A Policies
[Plan A AssumptionsPlan A Assumptions](/supplements/plan-a-assumptions)[Comparing Possible PlansComparing Possible Plans](/supplements/comparing-possible-plans)[Deal DeclineDeal Decline](/supplements/deal-decline)[Takeoff SupplementTakeoff Supplement](/supplements/takeoff-supplement)
Economics
[Economics of Plan AEconomics of Plan A](/supplements/economics-of-plan-a)[Compute in Plan ACompute in Plan A](/supplements/compute-supplement)[Economic Growth ExplorerEconomic Growth Explorer](/supplements/econ-explorer)
Speculative Analysis
[Space Governance PlanSpace Governance Plan](/supplements/space-governance-plan)[AI for EpistemicsAI for Epistemics](/supplements/ai-for-epistemics)[Alignment RoadmapAlignment Roadmap](/supplements/alignment-roadmap)
[About](/about)
[![AI Futures Project](/ai-futures-logo.svg)](https://ai-futures.org/)
# FAQ
**Q: Isn’t Plan A bad for concentration of power risks?**
A: No, Plan A reduces COP risks by a lot. The status quo involves an insane amount of power concentrating into the frontier AI companies (which are all based in the US). If they solve the alignment problem, the result will be a handful of AI CEOs or USG officials controlling an army of superintelligences, and hence having the ability to take over the world. (This is what happened in the slowdown ending of AI 2027).
Plan A dramatically improves on the status quo by:
1. Increasing the number and diversity of frontier AI companies. Because of regulations slowing the frontier, trailing actors can catch up, and in the Plan A scenario, by 2040, there are dozens of companies across different countries near the frontier of AI development.
2. Increasing the transparency and visibility into AI development. A core dynamic driving concentration of power risks is a dynamic where most of the power centers in the world do not realize they are going to be disempowered by AI until it is too late. In Plan A, the takeoff period is much longer and much more transparent, so the public, the US government, and countries besides the US all have much more time to see what is going on inside the labs.
3. Spreading out the AI-driven transformation of the planet over a few more years, giving more time for everyone to prepare and react, making it more likely that something resembling our current system of checks and balances persists instead of crumbling into chaotic conflict.
4. Decreasing the feasibility of inserting secret loyalties into AI systems. There is high quality compute accounting,so it is no longer feasible for anyone to obtain hidden compute, which could be used to train a backdoor into the AIs.
**Q: Wouldn’t China not agree to the deal? Even if they did agree on paper, wouldn’t they defect and keep developing AI anyways?**
A: China is likely to want something along the lines of Plan A because the status quo outcome for China is that they will be outcompeted by US AI companies. This will result in either misaligned AIs taking over the world, or the US obtaining a decisive lead in AI capabilities sufficient to disempower China. (These two options correspond to the race and slowdown endings of AI 2027 respectively). The CCP today does not seem very aware of the importance of AGI or ASI, but this will change as AI has bigger and bigger impacts on the world.
The question isn’t “Will China defect from the deal?” The question is, “What’s the biggest defection they could get away with, and how bad would that be?”
There are two ways of defecting from the deal: (1) building a massive covert project that you attempt to keep hidden from the rest of the world, and (2) using the existing legal compute.
![](/supplements/faq/image1.png)
A baseline estimate for the maximum size of covert project that they could have is ~1.2M H100e. We do a more thorough analysis of this question in [our covert project supplement](/supplements/covert-ai-projects), and we have a branch of our scenario gaming out how China could defect.
We prevent the use of existing legal compute through training verification mechanisms. We assume US and Chinese auditors are given full transparency into the other’s R&D datacenters, allowing them to check if the compute is being used how they claim it is being used. We do a more thorough analysis of this question in our [verification supplement](https://www.ai-2040.com/supplements/verification-plan).
**Q: Isn’t Total Research Transparency bad because it leaks algorithmic secrets to China and other AI companies?**
No current AI companies claim to be secure against nation state level adversaries. They aren’t secure even for model weights, which are much easier to secure than algorithmic secrets because model weights are large, often on the order of 1-10TB, while algorithmic secrets are often memorizable by individual humans. Because AI algorithms will have already leaked to competing nations, the costs of full transparency here are relatively low compared to the enormous benefits.
**Q: Plan A is complicated. Shouldn’t we do a simpler and more straightforward plan, like shutting it all down (Plan S)?**
Our proposed implementation of Plan A is very complicated, and even a good implementation will incur significant existential risk. However, all simpler plans that we are aware of would incur even more risk than Plan A. In particular the main issue with Plan S is that it does worse than Plan A at making forward progress towards solving the safety/alignment problems that prevent further capability scaling, because we are shut down at an earlier capability level. This is bad because at some point, both Plan A and Plan S will break down and return to racing, and in Plan S, much less alignment progress will have been made by this point. That said, we are very sympathetic to Plan S and could imagine being convinced that some version of it is better after all.
**Q: The US government is incompetent. Why are we trusting the US government to do a good job regulating AI?**
For any plan, incompetent execution of that plan is a serious concern. We are in fact very worried about incompetent execution of Plan A, and so we wrote a [branch of our scenario explaining how this could lead to existential catastrophe](https://ai-2040.com/?choices=plan-a-root#branchpoint-flawed-safety-case-is-approved).
Who else is going to regulate AI, if the US government doesn’t? We don’t trust the industry to self-regulate effectively. Moreover, if the US government is kept in the dark while AI companies race through an intelligence explosion, that seems like an unstable situation that could lead to sudden wakeup and thrashing about, or worse, a US company effectively puppetting the government (as hinted at in the [AI 2027 slowdown ending](https://ai-2027.com/slowdown)). If the US government is not in the dark, they will want to regulate AI, and we are trying to figure out and describe what good regulation would look like.
Large parts of Plan A are designed to improve the AI-related competence of the government—and the wider world—as fast as possible. For example, this is part of why we recommend [Total Research Transparency](https://www.ai-2040.com/supplements/transparency-plan) instead of some more moderate kind of transparency. This is also part of why we recommend limiting the rate of AI progress; it gives the world more time to react and prepare for each new level of capability. We also have things to say about [using AI to improve epistemics](https://ai-2040.com/supplements/ai-for-epistemics), and about immediate actions the government could take to improve its understanding of AI and capacity to regulate effectively. Finally, a big motivation for Plan A is to prevent extreme concentrations of power, which is good in its own right but also helps mitigate government incompetence.
**Q: Plan A just isn’t going to happen. Don’t you know that?**
The Plan A scenario is a recommendation, not a forecast; it is _not_ what we expect to happen. We think many of the ideas in AI 2040 are good and will be helpful inspiration for people trying to steer things in a better direction, even if Plan A doesn’t happen. That said, we also think that Plan A is plausible enough that it is worthwhile to push for it.
Over just the last year, there has been a massive shift in the awareness and understanding of AI among policymakers. As the effects of AI get larger, there will be vastly more wakeup. Before the point of no return—the point at which AIs or AI-powered humans can disempower the rest of the world—AI will probably have transformative effects on the world. It’s hard to be confident in any specifics, but some illustrative possibilities include:
* AIs can automate the majority of all jobs: physical jobs with robots, cognitive jobs with AI agents running in the cloud. This leads to massive widespread unemployment.
* Extreme economic impacts other than unemployment. For example, perhaps AI companies will continue increasing their revenue at ~3x/year until 2030, which would mean that the frontier AI companies would have a $10T/year revenue. This is an unprecedented amount of wealth and power in a company, and will trigger alarm bells.
* AIs completely change the nature of warfare. For example, AI piloted drone swarms may render existing mechanized militaries (e.g. infantry, tanks, planes, aircraft carriers) completely obsolete. This might lead to significant changes in the geopolitical balance of power as countries who aren’t at the AI frontier will fall behind militarily.
* Extreme misuse capabilities, e.g. suppose it’s the case that AIs are able to cheaply design and spread bioweapons with commonly accessible materials. Hardening the world against these capabilities will be extremely difficult.
It’s hard to know what the political environment will look like after AI has massive effects on the world. The main question is how early transformative effects on the world happen relative to AI-caused permanent lock in. Plan A is much easier and more effective if implemented early—before the AIs have fully automated the AI R&D process.
sources/ai2040-pages/how-plan-a-solves-our-5-biggest-problems.md
+73
View File
@@ -0,0 +1,73 @@
---
url: https://ai-2040.com/supplements/how-plan-a-solves-our-5-biggest-problems
---
## [AI 2040](/)
≡[PDF](https://ghjyjqzwz4.ufs.sh/f/9qHa0cBclQ7sf6FVzqbp2gHFXC1IbGS5ZA7DmNBusfM4YJRP)
[Supplements](/supplements)
Plan A Basics
[FAQFAQ](/supplements/faq)[How Plan A solves our 5 biggest problemsHow Plan A solves our 5 biggest problems](/supplements/how-plan-a-solves-our-5-biggest-problems)
Plan A Policies
[Covert AI ProjectsCovert AI Projects](/supplements/covert-ai-projects)[Verification PlanVerification Plan](/supplements/verification-plan)[Transparency PlanTransparency Plan](/supplements/transparency-plan)[Capability Scaling StrategyCapability Scaling Strategy](/supplements/capability-scaling-strategy)[Security in Plan ASecurity in Plan A](/supplements/security-in-plan-a)
Analysis of Plan A Policies
[Plan A AssumptionsPlan A Assumptions](/supplements/plan-a-assumptions)[Comparing Possible PlansComparing Possible Plans](/supplements/comparing-possible-plans)[Deal DeclineDeal Decline](/supplements/deal-decline)[Takeoff SupplementTakeoff Supplement](/supplements/takeoff-supplement)
Economics
[Economics of Plan AEconomics of Plan A](/supplements/economics-of-plan-a)[Compute in Plan ACompute in Plan A](/supplements/compute-supplement)[Economic Growth ExplorerEconomic Growth Explorer](/supplements/econ-explorer)
Speculative Analysis
[Space Governance PlanSpace Governance Plan](/supplements/space-governance-plan)[AI for EpistemicsAI for Epistemics](/supplements/ai-for-epistemics)[Alignment RoadmapAlignment Roadmap](/supplements/alignment-roadmap)
[About](/about)
[![AI Futures Project](/ai-futures-logo.svg)](https://ai-futures.org/)
# How Plan A solves our 5 biggest problems
### Thomas Larsen
**Problem #1: Loss of Control to superintelligent AIs.**
Due to the slowdown agreement, the intelligence explosion happens over the course of at least 10 years. Instead of being forced by competitive pressures to automate work and trust AIs, companies now are forced to slow down until they have a rock solid safety case. In the Plan A scenario, we have ~5 years with top expert level AI before handoff, and we could realistically get more time. If alignment ends up being hard enough that a 10 year slowdown isn’t sufficient, the infrastructure for a longer pause is set up.
Due to the transparency and access provisions in the deal, there is way more oversight of AI developers, and many more total people have the relevant access to do AI alignment work with frontier models.
(See more [here](https://ai-2040.com/supplements/comparing-possible-plans#plan-assessment-table)).
**Problem #2: AI will lead to massive concentration of power into whatever group controls the first superintelligences (if anyone).**
The transparency act already helps somewhat to prevent CEOs backdooring their models. The Total Research Transparency deal then “seals the deal” to a much greater extent; any attempt at inserting a hidden agenda into your biggest and most-used models–whether initiated by a CEO or a President–would be visible to multiple world governments. The model spec transparency + algorithmic progress slowdown + Total Research Transparency deal also makes there be many companies and governments with similar levels of AI, which is great for spreading out the power more generally, not just narrowly preventing secret loyalties/agendas.
Because the intelligence explosion is now happening in a slow and drawn out way, and we’ve prevented disparities in model access, so everyone has access to the same level of AI capabilities, anyone who is getting disempowered will be able to realize that and then negotiate for a better deal. Normal people have access to smart enough AIs that they can understand the situation well, and they can vote for ethical and thoughtful politicians, who will maintain and improve the power the public has, leading to a virtuous cycle. This leads to the Citizens Dividend and space governance proposal being implemented and spreading out the power forever.
**Problem #3: World War 3 risk due to superintelligences disempowering many countries, including nuclear powers.**
Plan A prevents WW3 by ending the “winner take all” effects of a race to superintelligence. No longer is there a race to ASI with each side fearing what the other might do if they get there first, and with each side contemplating taking aggressive action to slow down the other side, and with one or more sides starting to freak out as they realize they are on a path towards losing. No longer are there a bunch of middle countries gradually realizing that no matter who wins, they lose, and starting to freak out also.
**Problem #4: AI taking everyone’s jobs.**
The reasons that unemployment is bad are that people lose their (a) income, (b) power, and (c) source of meaning. Plan A slows down AI induced job loss to a rate where the negative effects of unemployment can be mitigated.
The main reason that people need jobs is to have a source of income. Plan A addresses the loss of income via the Citizen’s Dividend. Each person gets a share of income large enough to support an extremely high quality of life.
A second reason why jobs are important is that they empower their holders. CEOs and governments are incentivized to care for people who provide economic value. However, once AIs can do all the economically relevant tasks, governments and companies lose a strong incentive to provide for their citizens/employees. See the “concentration of power section” for more.
A final reason jobs are important is that they provide meaning for many people. Here we just say that it won’t be that bad (people can get meaning in a bunch of ways, not just economically useful work! Family, friends, romance, hobbies, sports!) and that the benefits outweigh the costs. If you disagree, well, we also describe Plan S… there’s basically no way to avoid mass unemployment if you allow companies to build and deploy AIs that obsolete humans. And in particular, banning the use of AI in various professions won’t work for long.
**Problem #5: AI misuse by small actors.**
Widespread access to smart AI will make it trivially easy to build bioweapons (or other technology that would cause huge damage at little cost to the attacker). Terrorists, states (e.g. North Korea), or misanthropists who are motivated to cause immense harm could leverage AI to do a much larger attack than they would have otherwise.
Plan A, as part of mitigating covert projects, limits the capability of open weight models to pre-deal levels. Closed models are trained to refuse dangerous requests. In case access limitations fail, Plan A involves massive investments into biosecurity and other measures to improve the resilience of the world.
There’s an alternate approach to misuse taken in Plan B, C, and D, which is to speedrun the intelligence explosion and hope to build superintelligence that can directly take over the world, hence preventing misuse. It’s plausible that these plans look better for misuse risk than Plan A because there’s so little time for the misuse to occur before superintelligence. But this strategy imposes unacceptable levels of loss of control and concentration of power risk, and so doesn’t pass overall cost benefit.
sources/ai2040-pages/main.md
+1320
View File
File diff suppressed because it is too large Load Diff
sources/ai2040-pages/plan-a-assumptions.md
+544
View File
@@ -0,0 +1,544 @@
---
url: https://ai-2040.com/supplements/plan-a-assumptions
---
## [AI 2040](/)
≡[PDF](https://ghjyjqzwz4.ufs.sh/f/9qHa0cBclQ7sf6FVzqbp2gHFXC1IbGS5ZA7DmNBusfM4YJRP)
[Supplements](/supplements)
Plan A Basics
[FAQFAQ](/supplements/faq)[How Plan A solves our 5 biggest problemsHow Plan A solves our 5 biggest problems](/supplements/how-plan-a-solves-our-5-biggest-problems)
Plan A Policies
[Covert AI ProjectsCovert AI Projects](/supplements/covert-ai-projects)[Verification PlanVerification Plan](/supplements/verification-plan)[Transparency PlanTransparency Plan](/supplements/transparency-plan)[Capability Scaling StrategyCapability Scaling Strategy](/supplements/capability-scaling-strategy)[Security in Plan ASecurity in Plan A](/supplements/security-in-plan-a)
Analysis of Plan A Policies
[Plan A AssumptionsPlan A Assumptions](/supplements/plan-a-assumptions)[Comparing Possible PlansComparing Possible Plans](/supplements/comparing-possible-plans)[Deal DeclineDeal Decline](/supplements/deal-decline)[Takeoff SupplementTakeoff Supplement](/supplements/takeoff-supplement)
Economics
[Economics of Plan AEconomics of Plan A](/supplements/economics-of-plan-a)[Compute in Plan ACompute in Plan A](/supplements/compute-supplement)[Economic Growth ExplorerEconomic Growth Explorer](/supplements/econ-explorer)
Speculative Analysis
[Space Governance PlanSpace Governance Plan](/supplements/space-governance-plan)[AI for EpistemicsAI for Epistemics](/supplements/ai-for-epistemics)[Alignment RoadmapAlignment Roadmap](/supplements/alignment-roadmap)
[About](/about)
[![AI Futures Project](/ai-futures-logo.svg)](https://ai-futures.org/)
# Plan A Assumptions
### Thomas Larsen
# Summary
Plan A is a scenario: it combines forecasts with policy recommendations. Some of the forecasts and recommendations we are confident in; others are just guesses we made because we had to, in order to achieve the desired level of concreteness.
| **Confident**| **Unconfident**
---|---|---
**Prediction**| E.g.: “AGI is coming soon and will be a bigger deal than the industrial revolution”| E.g.: “AGI will arrive in 2030”
**Recommendation**| E.g. “There should be an international deal to limit the speed of AI takeoff”| E.g. “Many new datacenters should be constructed in Mongolia and Canada.”
This supplement goes through the major predictions and recommendations of the scenario and explains which are which. The supplement tries to capture all our main recommendations, so if something isn’t mentioned here, it is probably a prediction, and insofar as it’s a recommendation, it is not an important one.
Since this is a document about specific beliefs, it’s written from the perspective of one of the scenario authors (Thomas). Other authors by and large agree with the central claims but have somewhat different opinions on the specifics.
## Forecasts Summary
For each of the following, I am >80% confident:
* **AI R &D will be fully automated within the next ~15 years**, absent substantial regulation, war, or societal collapse. (See the [AI Futures Model](https://www.aifuturesmodel.com/) for more detail; AIFP employees have somewhat different views, but all agree that 2027 and 2030 are plausible and reasonably central years.)
* **Within a few years of AI R &D automation we will build AIs that are better than humans at all economically relevant tasks** (absent substantial policy intervention). These AIs would quickly become vastly cheaper than humans, and so soon humans will become economically irrelevant.
* **If we build vastly superhuman AIs, the AIs could take over, if they collectively wanted to**. Therefore, it is critically important that most of them do not want to.
I’m not confident about:
* **The exact timeline.** In AI 2027, the automation of coding happened in 2027; in Plan A, this would have happened in 2030 in the absence of governance, but is slowed by a few years. My 90% confidence interval is mid-2027 to ~2050. We chose 2030 because it's near enough that our specific planning is useful, but far enough that the government has time for the preparation phase we recommend. If timelines are shorter, there isn't time to prepare and so Plan A becomes more rushed; if longer, more of the details are wrong but the high-level plan stays the same. ([more](/supplements/plan-a-assumptions#timelines))
* **Takeoff speeds.** How long between automating coding and AIs that dominate the top human at everything? We assume a default of ~1 year (my median), but my 80% CI ranges from 2 months to 5 years. Under my views about takeoff, the plan is basically similar under different takeoff speeds, though there are some small prioritization updates depending on the direction. ([more](/supplements/plan-a-assumptions#takeoff-speeds))
* **Alignment difficulty**. We don’t know how hard it will be to solve the AI alignment problem. I think it’s possible that even without a major slowdown, we are able to solve the alignment problem. Similarly, I think it’s possible that even 10+ years of dedicated effort, with lots of AI assistance, won’t be enough. Plan A is designed to buy time and attempt to gain evidence about how difficult the problem is. If evidence accumulates that AI takeover risk is small, then we can proceed faster, otherwise, the pause extends. ([more](/supplements/plan-a-assumptions#alignment-difficulty))
* **How difficult it will be to verify compliance with a slowdown deal.** It could be that it’s basically intractable to do major AI development without access to giant datacenters which are easily monitored. On the other hand, it could be that algorithmic progress is doable even with a very small amount of compute, which means that very quickly even a very small cluster could do an intelligence explosion. In even the most pessimistic case, we think it’s very likely that a deal could be robustly enforced for a year or two. ([more](/supplements/plan-a-assumptions#verification-difficulty))
* **Many other strategic variables.** ([more](/supplements/plan-a-assumptions#less-central-updates))
For each of these key strategic variables, the Plan A scenario chooses a reasonably central value with respect to my views. For many of these my views are very uncertain, and many policy decisions should be made with more information about these variables than we currently have.
## Policies Summary
In the next section, I discuss which of our policy recommendations in the scenario are robust across a wide range of empirical updates.
These are robustly good policy recommendations:
* **Improve AI Preparedness in government** _(importance: medium)._ Transparency, improving technical expertise within government, and measures to accelerate verification research. These policies are good because they are low-regret, and in some scenarios (like Plan A), they are very important.
* **Model spec transparency and verification against power grabs** _(importance: medium)._ Model specifications (of both internal and external models) should explicitly prohibit assisting with power consolidation (by CEOs, presidents, or anyone else), and AI projects should be required to implement verification measures to prevent backdoors.
* **US/China deal to slow the capabilities explosion, based on compute governance** _(importance: high)._ This is the core of Plan A. I’m confident that a cooperative international approach (Plan A) is better than the alternatives because the alternatives involve being in a race during the intelligence explosion, which incurs huge risks. I’m not confident about the exact variant of Plan A, or whether Plan A as we’re imagining it (a many year mutual slowdown) is better than Plan S (a full shutdown). Conditional on attempting to “slow down” AI progress, doing this via compute governance looks like by far the most effective way of accomplishing this goal (even though I’m not sure exactly how well it will work).
* **Redistribution of AI-generated wealth** _(importance: medium)._ As AIs automate the entire economy, the wages that humans will be able to earn will plummet to near-zero. By default (and assuming we avoid AI takeover), AI will cause historically unprecedented wealth inequality. I feel confident that there needs to be some measures for wealth redistribution.
* **Defensive Acceleration** _(importance: medium)._ Governments, AI companies, and the public should invest in technologies that make society more robust. Particularly important domains include AI for epistemics, bio defense, and cyber security. In the absence of a concerted effort to pursue defensive acceleration, increased offensive capabilities coming from high AI capabilities will lead to significant risk. Defensive acceleration also has positive externalities because it makes it harder for misaligned AIs to take over the world, and thus gives us more time to do alignment research
* **Thoughtful post-ASI governance** _(importance: high)._ We feel good about some general principles for post ASI governance: (1) have lots of smart people use (aligned) AI assistance to think carefully about effective governance, (2) be careful making irreversible commitments, (3) make it hard for any individual actor to consolidate power.
These recommendations depend on particular assumptions that we’re not confident in, or have alternatives that might be better:
* **How fast to scale AI capabilities after a deal** _(importance: high)._ Scaling faster incurs more immediate safety risk, while scaling slower means that you delay safety progress that’s downstream of having smarter AIs and incurs risk of deal dissolution and covert projects. These decisions will and should be made largely based on the evidence at the time about how risky particular AI systems and deployments seem to be, and how good our mitigations seem to be. ([more](/supplements/plan-a-assumptions#how-fast-to-scale-ai-progress-after-a-slowdown-deal))
* **Massive datacenter buildout** _(importance: medium)._ In our Plan A scenario, we recommend that governments encourage a massive datacenter buildout. This recommendation is contingent on having implemented an effective international deal to slow down AI, and even conditional on that, it’s not clearly good. The case in favor is that hardware scaleups are more governable than software scale ups because we can prevent the hardware from proliferating. But there are major costs: (i) it increases the required accuracy of verification within the legal projects, and (ii) if the deal breaks down (and failsafe mechanisms fail), it could result in the intelligence explosion afterwards being much faster. ([more](/supplements/plan-a-assumptions#massive-datacenter-buildout))
* **Our verification plan** _(importance: high)._ The verification plan described in Plan A is important for making sure that both the US and China are following the agreement. But I’m quite unconfident about the exact best way to implement it. We have a specific proposal which involves first implementing inference-only (and stopping the construction of new AI models) while sprinting to figure out how to verify training and experiments. Many alternative approaches to verification exist. ([more](/supplements/plan-a-assumptions#our-specific-verification-plan))
* **What transparency/security tradeoffs to make** _(importance: high)._ Security and transparency both have obvious upsides, but are in tension. We’d like as much transparency as possible to get as many eyes as possible on the safety cases being presented, but we’d also like to secure model weights, algorithms, and inference tokens. Overall, we think that we should be willing to pay substantial security costs in return for improved transparency, because we are more worried about major AI projects abusing their power and/or making mistakes, than we are about defecting projects with a tiny amount of compute (who are the main beneficiaries of poor security). The tiny projects just aren’t going to be as powerful, and bad or reckless behavior by them correspondingly doesn’t matter as much. ([more](/supplements/plan-a-assumptions#transparency-and-security-tradeoffs))
* **Our post-ASI governance plan** _(importance: high)._ [Our space governance plan](https://www.ai-2040.com/?choices=plan-a-root#playbook-epilogue) is, in my opinion, a reasonable first pass about what should be aimed for, and is generally applicable to any scenario where humanity retains control. However, I am not at all confident in the details, and in these scenarios, we will have aligned, superhuman AIs to help us puzzle this out, so my main recommendation is to use the huge amounts of AI labor to come up with a better plan. ([more](/supplements/plan-a-assumptions#our-post-asi-governance-plan))
* **Broad deployment of AI** _(importance: medium)._ Broad deployment spreads power and could improve societal epistemics. Some central ways it could be bad are (i) the net effect on societal epistemics might be bad, (ii) broad deployment could make a persuasion/political power takeover easier for the AIs (or for a human attempting an AI takeover) or (iii) they could lead to the proliferation of offense dominant capabilities. On net, I think broad deployment is probably good, but I feel uncertain, and it depends on how good mitigations are in place for each of the above failure modes. ([more](/supplements/plan-a-assumptions#broad-deployment-of-ai-during-the-takeoff-period))
* **Various other details of Plan A implemented in our particular scenario.** This includes things like datacenters on the oceans, incentive shaping with respect to alignment, incentive shaping with respect to covert projects (e.g. hardened compute stockpiles), how to trade off deal dissolution risk, covert project takeover risk, and risk of incompetent governance.
The above were largely details about Plan A that might change. There are also structurally different plans that may be competitive with Plan A, which we discuss [here](https://www.ai-2040.com/?choices=plan-a-root#other-plans-that-are-competitive-with-plan-a).
Still, I think we should be aiming for something like Plan A across a wide range of scenarios. The rest of this document has two sections, which both discuss how our plan would change in response to empirical changes.
1. **Empirical Updates** answers the question “if an empirical fact changed (e.g., suppose timelines were shorter or longer), how would our policy change”.
2. **Policy Updates** answers the question: “for each of our major policy recommendations, what would make us change that recommendation”.
# Empirical Updates
In this section, I’ll go over how Plan A would change in response to some important empirical assumptions that we make in the scenario, which might turn out to be different in practice.
## Timelines
We don’t know when AGI will be built; it’s possible that it could be built this year or next year, it’s also possible that it will take decades.
In the Plan A scenario, the full automation of AI R&D was on track to happen in 2030 by default, but then was pushed back by government involvement. We’ve written much more on AI timelines; you can find our most updated work [here](https://www.aifuturesmodel.com/). AIFP employees continue to have (slightly) different views about AI timelines, but we all think that 2030 is a plausible and reasonably central year for AGI to arrive.
**Why did we choose to write this scenario assuming 2030 timelines?**
It’s good to showcase a variety of different plausible scenarios. We already wrote a very short timeline scenario (AI 2027), so we figured it would be good to choose a different timeline for our next one. 2030 is my (Thomas’) median currently, i.e. I think there is a 50% chance things will proceed slower, and 50% chance things will proceed faster. In the future, if time remains, we’d like to do scenarios for other authors like Eli.
**Overall, Plan A remains my central plan ~regardless of timeline** , because the underlying problems (AI takeover, AI enabled concentration of power, and rapid AGI induced societal upheaval) are the same regardless of timelines, and Plan A is our best guess for how to deal with these problems.
Regardless of the timeline, Plan A is most effective the earlier an international deal that involves mutual compute declarations and transparency begins. When we get to the beginning of the intelligence explosion (e.g. the automated coder milestone), verification becomes substantially more difficult, because defecting projects can involve many fewer humans, and therefore be much harder to detect.
If the timeline is much longer (e.g. 2035+), we begin to have much more uncertainty about the situation. The world will also be more transformed pre-AGI relative to our scenario, there will be larger economic effects of AI, AI company revenues (and share of the overall market) will (probably) be higher, there will be more robots around, and there will be more wakeup to the importance of AI. Particular effects include:
* We become somewhat less worried about covert projects because compute stockpiles are inherently insanely large, and so the relative amount of compute that covert projects have compared to the verified projects will be smaller. (The absolute amount of compute, on the other hand, is larger). However, there are also reasons to be more concerned about covert projects—a counteracting effect is that there will be more compute distributed throughout the world which will be harder for intelligence agencies to locate.
* It becomes more important to do the economic parts of the Plan A policy package (e.g. the Citizen’s Dividend) earlier, because more people are affected earlier.
* It becomes more important to do D/acc earlier because there is a longer period of time with reasonably intelligent AIs around, and current AIs already provide substantial biouplift.
* Since the overall amount of compute in the world is higher, we become more worried about a new paradigm being discovered that’s much more compute efficient, which could lead to a fast intelligence explosion.
If timelines are much shorter (e.g. 2026 or 2027), there isn’t time to do the preparation phase, and so we have to do a version of Plan A with no preparation.
* It’s more likely that we should pivot to one of the contingency plans that involves less government capacity. Unfortunately, these contingency plans have other downsides. For example, shutting down large fractions of AI relevant compute would also turn off inference of existing models, which would be economically costly.
* It’s more urgent that an agreement to slow down AI takeoff happens early, because we’ll soon be so deep into the intelligence explosion that it’s impossible to slow down.
## Takeoff speeds
In this scenario, similar to AI 2027, we chose to assume that the default length of time between AC (Automated coder) and TED-AI (Top-Expert-Dominating AI) was 1 year. This is roughly my median, but we again have large uncertainty; my 80% CI at time of writing for the takeoff duration is something like [2 months; 5 years].
**Why did we choose 1 year takeoff speeds by default?** This is my median at the time of writing Plan A, and seemed like a representative scenario to plan for.
**How does the plan change under different takeoff speeds?**
**If takeoff is faster than 1 year** , things become generally more difficult, and not just for Plan A. The main specific changes are:
* Doing an international deal to slow down takeoff becomes more important, because the status quo is more risky, because AI driven change without deal happens much faster and there is less time to react to each change and prepare for the next changes.
* Enforcement of an international deal becomes more difficult. Faster takeoff typically involves much less hardware increase, because if the takeoff occurs over months, there isn’t time for a massive compute scale up. The possibility of an intelligence explosion without a massive compute scaleup is the key assumption behind being worried about covert projects; if such an intelligence explosion wasn’t possible, then covert projects with relatively small amounts of compute would be much less concerning. Since the intelligence explosion happens faster, it also makes covert projects faster, reducing the amount of slack that the international Consortium has to work with.
* The slowdown agreement needs to be started earlier, which would make the enforcement easier (and therefore the whole plan more viable).
* Since enforcement of the deal is more difficult, this is an update towards making tradeoffs that help with enforcement but with various other costs, such as (i) leaning more towards security on the security/transparency tradeoff, (ii) making less algorithmic progress, and therefore scaling more slowly, even if scaling further would still be safe, and (iii) doing more early hardware scaling, which makes verification and self-destruct plans more load bearing, but helps reduce misalignment risk.
**If takeoff is slower than 1 year,** e.g. even without regulation it takes several years to go from the full automation of coding to the first artificial superintelligence, then we basically update in the opposite direction on all of the above tradeoffs.
* Overall risk becomes lower, because there is more time by default to deal with risks as they arise. Therefore, the overall plan will attempt to incur a smaller amount of risk, and will be willing to make more aggressive tradeoffs to reduce risk.
* Covert project takeover in particular becomes less likely, because it is extremely unlikely that they could bootstrap into existentially dangerous AI systems with only a tiny amount of compute. This updates us even further in favor of doing Total Research Transparency (because we are relatively more worried about the large projects than the small projects), and away from doing the large hardware scaleup. The plan can involve just doing a slowed down version of the intelligence explosion on the existing compute stockpile, leaking algorithms to the trailing project, but relying on the fact that they are so compute limited that they cannot make significant further AI progress.
* AI takeover is less likely, and so we can devote relatively more effort on mitigating other risks. It’s still very important to slow down AI progress, because there’s still a significant chance that AI alignment won’t be solved during the default takeoff, and because slowing down helps to mitigate other risks like concentration of power. However, it’s more likely that we wouldn’t actually slow down much in this world. Therefore, doing the slowdown part of the treaty is less load bearing.
Overall, we recommend pursuing Plan A regardless of takeoff speeds. However, changes to our takeoff speeds do substantially change the specific implementation of Plan A, in the ways noted above.
## Alignment difficulty
In the Plan A scenario, humanity rallies and creates a high assurance safety case for handing off to ASI. However, **we are much more confident in the general picture that more time and effort lead to a greater chance of solving the alignment problem than the specific story told here.** We don’t know how hard it’ll be to align ASI. Opinions vary substantially within our team (and even more outside of our team). Given the uncertainty, our view is that we should take a portfolio approach, and invest substantially in both prosaic AI safety agendas (e.g. variants of RLHF) and moonshots (e.g. uploading human brains).
The plan we are advocating for is one in which the world coordinates to give the scientific community breathing room and resources to figure out how difficult the alignment problem is. If Yudkowsky is right that AI alignment is difficult, our view is that evidence would increase during this time period and then the pause would be extended. Likewise, if the optimists are right and alignment is easy, we would get evidence of this during that period, and we could proceed at a faster pace (though it’s not clear that we should, given the other risks of doing so, most notably concentration of power).
I think that alignment is probably difficult enough that without a well managed multi-year slowdown, we will incur substantial AI takeover risk. Even with a ten year slowdown, as depicted in the scenario, there is a significant chance that alignment was too hard, or that it’s feasible but human error causes us to fail anyway. You can see more about the team’s views about alignment success likelihood conditional on following different plans [here](/supplements/comparing-possible-plans#outcomes).
**What do we do if alignment is much harder?**
Gather as much evidence about alignment difficulty as possible by:
* Deploying and observing AI systems to get empirical feedback about how aligned they seem to be.
* Building model organisms—AIs trained to deliberately be misaligned—and see if alignment techniques are good enough to catch and train away the misalignment. Do this for deceptive alignment, sycophancy, reward hacking, etc.
* Giving a large, heterogenous set of people access to the models (ideally including model internals) to avoid groupthink.
* Have explicit safety cases that lay out the evidence and arguments for why the models can and should be trusted, and publish them so they can be critiqued and debated by the scientific community.
During Plan A, maintain the optionality of extending the pause by:
* Monitoring for covert projects and shutting them down if detected
* Ensuring that the flow of new chips is highly secure and not going to any rogue projects
* Improve coordination and verification technology, both with large investment and using AIs to improve this technology. In particular, using AIs to build privacy preserving auditing and reliable lie detection seems very promising.
* Minimizing software progress that leaks to covert projects. If alignment is harder, we should update towards making tradeoffs that enable us to stay slowed for longer.
* Making sure decisionmakers have access to the best information on alignment difficulty, including by making sure that people who are more pessimistic about alignment have access to the models and the current safety cases, so that they can find problems in them before it becomes catastrophic.
If alignment is extremely hard, e.g., humanity isn’t making substantial progress on alignment whatsoever, then the correct call may be to pivot to Plan S, and instead of attempting a managed takeoff, shut it all down. Then, we could pursue alternate strategies to build superhuman AI that are inherently safer than ML, such as uploads. Unfortunately, these approaches have substantial downsides as well, notably the risk that the moratorium breaks down and we go back to racing before enough progress is made on uploads or other strategies in that vein.
**What if alignment is much easier?**
Slowing down AI takeoff is still probably good for many other reasons, including avoiding AI-driven concentration of power, figuring out appropriate post-ASI governance, transitioning to a regime where human labor is no longer economically relevant in a healthy manner, hardening the world to AI enabled superweapons, etc—basically, we still need to solve all of the problems that we dealt with in Plan A other than misaligned AI takeover.
The main difference is that there is a higher chance of humanity being able to successfully muddle through these other risks without permanent lock in. This means that the necessity of a slowdown does go down substantially if alignment is much easier than I expect.
## Verification difficulty
_Related reading:[Covert Project Branch](https://www.ai-2040.com/?choices=plan-a-root#branchpoint-china-attempts-a-covert-agi-project), [Covert AI Projects Supplement](https://www.ai-2040.com/supplements/covert-ai-projects), [Verification Plan](https://www.ai-2040.com/supplements/verification-plan)._
I’m not sure how difficult verification will be. Our main strategy for verifying compliance with an AI agreement is (1) limit the amount of compute that covert projects have access to, and (2) ensure that legal projects are doing what they claim. Both of these will be somewhat lossy, and we’re unsure about what the mapping is between “amount of unmonitored compute” and “how fast could an illicit intelligence explosion reach an existentially dangerous capability threshold”.
Now we’ll go through each of these in a bit more detail:
* **Covert project verification difficulty.** In the Plan A scenario, we assume that the maximum plausible covert project size without detection is about 1% of pre-deal compute. Our assumption here in the scenario is close to the worst case scenario. My best guess is that neither side would build a covert project in a Plan A scenario because (i) there’s no incentive to because it’ll be out paced by the legal projects, and (ii) both sides are scared of AI takeover risk; and building a covert project would undermine the deal. Sub-questions I’m most uncertain about:
* **What fraction of pre-deal compute can be tracked down?** Our uncertainty over pre-deal compute is very relevant, because it will determine the extent to which China’s initial compute declaration can be falsified. Existing experts we’ve talked to maintain huge amounts of uncertainty about how large China’s existing compute buildout is. As a percentage, this might become easier to track as AI datacenters get larger, but as an absolute number more and more compute will be harder to track down over time. However, the intelligence community may have access to private information that’s much more accurate today, and they might begin to gather this information in the near future. Overall, it’s possible that denying covert projects access to huge numbers of GPUs will be one of the most load bearing components in the non-proliferation plan; but it’s also possible that other factors such as HUMINT will be far more important.
* **How effective will HUMINT be for detecting covert projects?** It seems likely to me that a covert project of the size we assume in the [covert project branch](https://www.ai-2040.com/?choices=plan-a-root#branchpoint-china-attempts-a-covert-agi-project) of Plan A (500k H100e, $7.9B) would be quickly detected by intelligence gathering. Historical mega projects that were attempted in secret (e.g. the Manhattan Project) seem to have been riddled with spies; our baseline assumption should be that anything on the order of this size will be quickly detected and shut down. However, later into the intelligence explosion, the number of people involved in a covert project could be much smaller, as AIs could automate the AI research and robots could automate the construction and maintenance of the datacenter, dramatically reducing the probability of leaks.
* **What is the necessary reduction in access to compute resources?** In the Plan A scenario, we assume that an AI project with 10x less compute can do an intelligence explosion 5.5x slower; this is discussed more in [the appendix of our Covert Projects Supplement](/supplements/covert-ai-projects).
* **Legal project verification difficulty.** In the Plan A scenario, we assume that we can verify inference-only very quickly, and that we can verify training runs after a six month sprint to develop that technology. It could be easier (e.g. it’s possible to do at the software level), or it could be harder (particularly because governments are slow and ineffective at this). Our assumption here is on the optimistic end because we assume high competence in execution.
* If legal project verification is more difficult, then increasing hardware buildout is more scary. Quantitatively, with no hardware buildout, then we’d need something like 99% assurance on legal project compute; with a +4 OOM increase in stock, we’d need 99.9999% assurance on legal project compute.
There are various other particular verification dynamics which could affect the rest of the policy strategy:
* **Favorability of SW scaling?** A key dynamic for covert projects is stealing algorithms from legal projects. Therefore, legal projects should consider pursuing algorithms that are less favorable for covert projects. In particular: (1) HW/SW co-design, i.e. research algorithms that are only effective on specifically designed chips that covert projects don’t have access to, and (2) heavily scale-dependent algorithms. We’re unsure how favorable both of these paths will be; the more favorable they are, the more capabilities progress can be done without speeding up covert projects.
* **How much edge-hardware is required?** Some applications strongly prefer edge hardware (hardware that operates on the device as opposed to the cloud): notably military applications (because of jamming) and self-driving cars (because you don’t want your car to crash if there’s a network issue). This poses a challenge for verification because edge compute could be removed from the device and funneled to a covert project, or potentially backdoored and used as part of a large distributed training run. We’re uncertain to what extent large quantities of this will be required, if there are other solutions, and how good the mitigations look for preventing usage of edge compute in training runs.
* **How difficult will security be?** Nation-state proof security is basically required to make the legal project verification work. Nation state proof algorithmic security is helpful for preventing covert projects. Our baseline assumption on security is that it’s possible to make verification mechanisms sufficiently secure, most algorithms will not be possible to secure (because they are so small), and that model weights will be possible to secure (because they are so large). I’ll now go through the updates if our current assumption is wrong for each of the desired security properties:
* **If verification security isn’t viable** then our ideal strategy would either involve (1) doing a moratorium on further AI development (and potentially inference) until security is viable or (2) relying heavily on trust.
* **If algorithm security is viable** then we’d have more buffer against covert projects than we are currently modeling in Plan A. We’d also have less of a need to do a compute buildout, because one of the core motivations for building compute was to mitigate algorithms leaking to covert projects.
* **If model weights security isn’t viable** then we’d either need to (1) not scale AI capabilities until security becomes viable or (2) need to invest more heavily in defending against exfiltrated AIs. At very high levels of capability, this might involve extremely drastic measures, because we’ll need to defend against extremely capable AIs.
## AI Paradigm Shifts
Plan A currently assumes gradient descent on giant neural networks remains the dominant AI paradigm until the late 2030s. We assume significant advances are made within that paradigm, including (i) the addition of neuralese (both as long term memory, and replacing chain-of-thought) and (ii) dramatic scaling in reinforcement learning (many more diverse environments that are much larger than existing environments).
Substantial algorithmic changes, even within the broad “gradient descent on giant neural networks” paradigm, might change the implementation details of parts of Plan A. For example:
1. We assume that online learning via live weight updates (instead of in context learning or delayed weight updates) is not necessary for generally top expert level AI systems of the type developed in Plan A. If live weight updates were necessary, then we would need to have some training infrastructure happening on the inference datacenters, which would complicate the inference verification because we’d need to live update the reference models in the verification infrastructure, which would increase the security risk surface area. It would also prevent us from using chips that are hardcoded to do inference on a given model (e.g. [Etched](https://www.etched.com/careers)), which could be part of the inference-only verification scheme.
2. We assume that restricting access to compute is an effective measure to prevent covert projects. If there are, for example, new architectural discoveries with dramatically (e.g. 1000x) more favorable compute scaling than current architectures, this would make covert projects much more dangerous. That said, if such architectural discoveries are in the pipeline so to speak, all the other plans are much less likely to work too—a discovery which allows a covert project to overtake the legal projects in Plan A, despite a huge compute disadvantage, would in Plan D cause an extremely fast, discontinuous “FOOM” to ASI within whichever frontier AI project first found it. These worlds are extremely scary and probably the best way to handle them is something like Plan S.
There are also obviously many “unknown unknowns” in AI paradigm shifts which could change Plan A in unforeseen ways, in practice we expect that many updates to the Plan A baseline will need to be made in response to further paradigm shifts.
## Less central updates
* **State capacity.** Plan A relies on high state capacity on AI, in particular, the capacity to distinguish existentially unsafe training runs and deployments. However, in worlds where the state capacity is low enough that they are not able to distinguish safe and unsafe AI, we recommend pivoting to another plan. Two salient alternatives are: (1) having an extremely conservative bar for safety, (e.g. shut it all down), or (2) limit the speed of the intelligence explosion through more blunt techniques (e.g. mutual GPU destruction).
* **Capability level of the initial deal.** The later the deal begins, the more difficult enforcement becomes. In particular, once we get substantial automation of AI research, it becomes easier to run a covert project on AI with only a small number of people involved, which reduces the surface area for leaks. This means that the overall regime has less time, and so risk will be higher.
* **Distillation mitigation.** We discuss [our strategy for mitigating distillation in the verification supplement](https://docs.google.com/document/d/1K4vxl_IMf58OgNXbWntQl6BQ06yAWZ62WmW5Ypp4KYA/edit?tab=t.86wiss1dhlds#bookmark=id.9ax99eo6ehyb). It’s unclear how well this will work. If it doesn’t work well, it’s an update towards being more concerned with covert projects. It is also an update towards slower capability scaling being optimal. Finally, it’s also an update towards strongly limiting broad deployment, although I still think that even if mitigating distillation is very difficult, it would still be worthwhile to do broad deployment.
# Policy Updates
## How fast to scale AI progress after a slowdown deal?
It’s unclear, and will be dependent on observations at the time. Here are some key considerations:
1. **Is the deal likely to break down soon?** If the deal is more likely to break down soon (for example, due to political reasons, or due to covert project takeover), then you should be willing to take on more risk, and so you should be more willing to scale faster.
2. **How good do current safety cases look (and how misaligned do the models seem)?** The more AI takeover risk there is from marginal scaling, the slower we should scale capabilities.
3. **How large are the returns to better AIs?** If better AIs won’t contribute useful labor (for example because we can’t trust their outputs, and control isn’t very scaleable), then we shouldn’t scale. On the other hand, if scaling AI capabilities would dramatically increase the amount of high-quality alignment research we can do, it may be worth scaling, even if doing so is risky.
4. **Will the weights and algorithms leak to covert projects, and how bad are the leaks?** If the model weights and/or algorithms of a scaled up AI model were to leak this would accelerate defecting projects by some amount, decreasing the amount of slack the regime has. If it’s possible to secure the model weights and algorithms, or to scale capabilities without doing substantial algorithmic progress, then we should update towards scaling more. In the Plan A scenario, we mostly scale with hardware buildouts, to avoid the need to make substantial algorithmic progress, because we are not optimistic about securing the algorithmic secrets.
5. **How much risk comes from current AIs vs future AIs?** The earlier in the intelligence explosion we are, the more we should be sympathetic to scaling earlier, because the marginal risk of current scaling is much lower, whereas if we’re in the capabilities regime where we are incurring most of the risk, we should be much more careful and tend to slow down more. (This is why, in the scenario, the Consortium scales up AI capabilities reasonably but not entirely cautiously in the early 2030s, and then pauses at top expert level in 2035.)
6. **Is it possible to scale up capabilities without substantial algorithmic improvements?** In plan A, we try to scale up AI capabilities via hardware scaling, while heavily limiting algorithmic progress. However, it may be the case that scaling up AI capabilities inherently leads to lots of algorithmic progress (e.g. via distillation). Moreover, the main alternative to scaling up algorithms is scaling hardware. However, it may be the case that most algorithmic progress is downstream of hardware progress. In this case, the most effective way of governing algorithmic progress might be to limit hardware itself.
Ultimately, trading off these considerations will require quantitatively modeling each of these sources of risk: covert project risk, risk of AI takeover from current AIs and future AIs, etc, understanding how the speed of deployment impacts each of these risks, and then making complicated tradeoffs about the overall best path forward.
Plan A’s trajectory is essentially based on the viewpoint that deal dissolution risk will probably be high, that risks from early AGIs (e.g. ACs and SARs) can be mitigated via control (i.e. that control-based safety cases can be made good enough very quickly, for AIs at that level), and that we can elicit a huge amount of useful labor from these early AGIs. It also assumes that we can do most of our capabilities scaling via a rapid hardware buildout, and strongly limit the total amount of algorithmic progress that occurs, including successfully mitigating distillation (largely by hiding the intermediate reasoning traces). Under these assumptions, the best trajectory seems to involve scaling quite rapidly at the beginning of the slowdown agreement, and then slowing down dramatically at the maximum controllable capability level.
Plan A’s specific trajectory is our best guess for the best way to handle these tradeoffs. However, we aren’t confident whatsoever that this is the correct path, and we hope that decisionmakers at the time will have a better understanding of these empirical factors at the time, whereas we have to make rough guesses.
Another very plausible trajectory, Plan S, is essentially the opposite choice about how fast to scale: immediately issue a moratorium on further AI capability scaling. This policy looks better if there is a small risk of deal dissolution, and/or the returns to better AIs for safety research are low, and/or the risks of any capability scaling are high.
## Massive datacenter buildout
It is desirable to do a massive datacenter buildout in Plan A if (and only if!) the upside of scaling quickly, reducing covert project takeover risk, and having additional capacity to pay alignment taxes is bigger than the risk of verification failing or the additional risk incurred by the deal failing with a gigantic compute stockpile.
In a bit more detail, the main upsides of a massive datacenter buildout are that it allows AI developers to:
* **Scale quickly without leaking algorithms to covert projects.** In the previous section, we discussed the conditions under which it is good to increase AI capabilities, and the tradeoffs associated with going faster and slower. Simplistically, to increase AI capabilities, you can either improve algorithms or scale compute. By leaning heavily on compute, we can avoid making algorithmic progress in the first place, which is the best way to ensure that said progress doesn’t leak.
* **Pay alignment taxes.** Large compute-based alignment taxes might be very helpful for safety. For example, it might be that neuralese is much more performant than current architectures, but also has dramatically worse safety properties. If we have enough compute that we can just accept the performance penalty and continue with current architectures anyways, this might dramatically reduce risk. Without a massive compute buildup, we will by definition not be able to pursue architectures that are much less compute efficient.
The main downsides are:
* **Verification and regulation becomes more load bearing.** Since the verified projects have even more compute, it is more important that this compute is used safely, and, in particular, not used to do an intelligence explosion as fast as possible. So it becomes even more important that the regulations that AI projects are subject to are sufficiently conservative that they preclude an intelligence explosion, and that the verification is sufficiently robust that companies or rogue AIs can’t undermine it, and illegally gain control of a huge amount of compute within the verified clusters.
* **If the deal breaks down with a gigantic compute cluster, and the clusters are not destroyed, the intelligence explosion might be extremely rapid.** One naive estimate is that a 10x of compute leads to a 5.5x increase in the speed of the intelligence explosion, so a 10,000x increase in compute might increase the speed of the intelligence explosion by 900x. So if it would take a year by default, it might take less than a day with the Plan A hardware explosion. This is obviously extremely dangerous, hence why we put so much emphasis on mutually assured compute destruction.
Putting all of this together, my view on how to decide to what extent we should pursue a massive hardware buildout under Plan A is as follows:
1. If covert projects aren’t a concern (because, for example, it’s impossible to do a reasonably fast intelligence explosion on a 100,000 GPU cluster), then **don’t scale hardware at all, or only go slowly.**
2. If there are reliable mechanisms for destroying compute after the deal dissolution, and reliable verification mechanisms so that we can have high confidence that there will not be major rogue internal deployments (either by countries or AIs), and there is reliable verification for within-deal compute, then the downsides of a compute buildout are small. Therefore, it is beneficial to **scale hardware ~as aggressively as possible.**
3. Otherwise, it is going to be a complicated tradeoff, which needs to be decided by weighing the magnitudes of the various costs and benefits listed above.
## Our specific verification plan
The verification plan used in the Plan A scenario can be summarized as follows:
1. Mutually declare the locations of all large datacenters. Check the declared counts against intelligence estimates.
2. Agree to pause training and experiments. Install inference-only verification devices on the declared clusters to verify this pause.
3. Race to set up verifiable (e.g. radically transparent) training and experiment clusters; when complete, resume capability scaling within the constraints of safety and transparency.
We discuss this in more depth [here](https://docs.google.com/document/d/1K4vxl_IMf58OgNXbWntQl6BQ06yAWZ62WmW5Ypp4KYA/edit?tab=t.86wiss1dhlds#heading=h.g5eefrev1iir).
This version of the verification plan assumes that at the beginning of the deal (1) we can verify inference, and (2) AI development verification is feasible within a relatively short research sprint (but is not immediately feasible). It is most likely that one or both of these assumptions will be wrong; these assumptions are my central guess of what could be done under Plan A assumptions of political will and competence, but reality will most likely involve less good preparations.
What if we don’t have verification measures ready in time?
Three high level buckets of verification approaches are:
1. **Technical verification methods**. These methods would allow ongoing AI usage or development, with both sides of a deal being confident that the other isn’t doing anything dangerous. An example of this is what we do in Plan A where we initially rely on inference-only, while sprinting to technical methods to verify training and experiments.
2. **Rely on trust,** i.e. accept that either side could defect, and hope that neither side is foolish enough to try, or that they’d be caught by HUMINT and so forth if they did.
3. **Blunt verification techniques** (e.g. compute destruction or mutual power down).
yesnoyesnoViable technicalverification methods?Rely on trust?1 · Technical verificationinference-only devices, then verifiabletraining & experiment clusters (Plan A)2 · Rely on trustaccept that either side could defect;HUMINT etc. as the backstop3 · Blunt verificationmutual GPU destruction, mutualpower-down, burn compute on mining
Three buckets of verification approaches \- ai-2040.com
It is highly desirable that we have technical verification methods ready before the beginning of the intelligence explosion. If inference verification devices are ready, then access to AIs gets to continue at the beginning of the deal. If development verification is ready, then we don’t need to stop AI capabilities scaling or AI safety research that involves frontier models at the beginning of the deal, and can instead scale until it is sufficiently unsafe not to (following the considerations above).
2 and 3 will likely only be temporary regimes, and be paired with a sprint to get technical measures online. Therefore, we hope that eventually we’ll get to sufficiently good technical methods.
The main downside of 2 is that it runs a risk of defection. Therefore, it may also run into increased political difficulties, as both parties will be worried that the other will defect, and so will be less likely to sign in the first place. However, it’s not correct to entirely write off trust, because it may be that the risk of AI takeover becomes so high that neither side is incentivized to defect, because doing so would so likely result in existential catastrophe. If high AI takeover risk becomes common knowledge (for example, after we’ve caught early misaligned AIs in the act of sabotage), relying on trust may be the best option.
The main downside of 3 is that it involves a temporary period where large scale inference goes offline. However, this would be less bad than one might naively think; even an agreement to power off (or destroy) 90% of the compute, would only force a reversion to 10x cheaper models. Since cost of a given level of capability has historically decreased very rapidly ([averaging ~40x/yr](https://epoch.ai/data-insights/llm-inference-price-trends)), having 10x less compute would roughly correspond with having to use SOTA models from 7.5 months ago. A downside of 3 is obvious; if defection is viable, it becomes more likely that both sides would choose to defect in the first place.
Note that this is one upside of Plan S over Plan A; Plan A requires (1), while Plan S only requires (3).
## Transparency and Security tradeoffs
There is an inherent tradeoff between security and transparency—to secure something, you inherently need to prevent some group from being able to see it.
Security% of progress competitors can’t stealTransparency% of relevant info that’s publicLivestream thesingularityOpen source everythingexcept the weights and datasetsAirgap the researchers & auditors;relay all safety-relevant infovia paper memosAirgap the singularity;hand off to airgapped auditorsTodayPlan A: increase both
Security vs. transparency \- ai-2040.com
Plan A biases towards transparency, in large part because I am more worried about the risk of poor governance within the legal projects than the risk of covert projects defecting and successfully undermining the deal. Increasing transparency will on average improve the quality of the governance decisions made by the Consortium, whereas increasing security makes it harder for covert projects to succeed. Furthermore, even if the governance decisions are perfect, the legal projects may not be able to solve alignment in time themselves, and may need help from the wider world to e.g. notice some subtle flawed assumption in a safety case. Also, transparency is very helpful for verification.
That being said, we are currently nowhere near the pareto frontier between these two dimensions. Plan A proposes dramatically increasing both security and transparency into frontier AI projects. We discuss transparency-related tradeoffs in much more detail in our [Transparency Supplement](/supplements/transparency-plan#thomas-larsen).
## Our Post-ASI governance plan
_Related reading:_ [Space governance supplement](https://www.ai-2040.com/supplements/space-governance-plan) _,[AI 2040 epilogue](https://www.ai-2040.com/?choices=plan-a-root#playbook-epilogue)._
We make a concrete guess as to what a positive post-ASI governance plan might look like. However, we expect that there exist better proposals, which we haven’t thought of yet, that will be discoverable at the time. Therefore, we want readers to orient to our plan as a baseline with which to compare other plans to on dimensions that they care about. Then, we hope that humanity will spend vastly more effort—both human and AI cognitive labor—on designing a system that performs better.
## Broad Deployment of AI during the takeoff period
The Plan A scenario involves broadly deploying AI into the economy during takeoff. This has massive effects on the world during takeoff: almost everyone loses their jobs and starts relying on Citizen's Dividend. Overall, I think that it is better for massive AI impacts to happen gradually, e.g., over the course of a 10 year takeoff, than to have all of the impacts happen really quickly, right at the end, once AIs are vastly smarter than humans.
CapitalHuman cognitiveHuman physicalAIRobots0%25%50%75%100%2025202620272028202920302031203220332034203520362037203820392040
US factor income shares (Plan A) \- ai-2040.com
(See more about how AI impacts the broader economy in [our economics model](https://www.ai-2040.com/supplements/econ-explorer), and discussion in our [economics supplement](https://www.ai-2040.com/supplements/economics-of-plan-a))
However, this relies on a few key questions:
1. Is the net effect on societal epistemics positive or negative? I currently feel highly unsure. On one hand, broad deployment of human level AI will increase access to information, like the internet. On the other hand, while the internet is definitely helpful for some people, it’s unclear if the average effect on societal epistemics is good or bad. More research is needed on this; what plan we actually recommend at the time would depend a lot on our best guess about the epistemic effects of broad AI deployment. With our current state of knowledge, I currently don’t feel like this is a major consideration in either direction.
2. Does broad deployment meaningfully increase the difficulty of AI control? I think probably not, because the main risks of AI takeover route through rogue internal deployments and recursive self improvement. The case for broad deployment increasing risks is that it may lead to AIs having affordances that might be helpful for takeover, especially (i) a large robot workforce and manufacturing base, (ii) control over robot armies, (iii) access to biology labs, and (iv) regular contact with a broad range of humans. I think that these risks are likely mitigatable through a combination of monitoring AI behaviour, limiting access to the most dangerous physical infrastructure (e.g. biolabs), preventing any one AI system from having control of a large fraction of the physical resources, and preventing collusion between different AI systems. However, I do think that it’s plausible that broad deployment makes control much harder. Hopefully this can be handled via limiting AI deployment in a particular domain of concern, but it may in practice limit deployment across many domains.
3. How high are misuse risks from broadly deployed closed-source models? My inclination is that these risks are reasonably low, because (1) I’m optimistic about adversarial robustness in a regime like Plan A, where you are slowing down substantially in order to improve safety, and you have the capacity to run lots of monitoring, and (2) even if it’s possible to jailbreak widely deployed AI models, I think that the defensive measures put into place will be sufficient for preventing existential catastrophes from misuse risk. However, it is plausible that both of these assumptions fail (either because of intrinsically high difficulty or incompetent execution), in which case, we would be incurring serious misuse risk from broadly deployed models.
If we’re not broadly deploying AI through society, the main alternative is to be developing AI inside closed projects. This incurs additional risks. Whatever closed off group of people that are managing the intelligence explosion without contact with the rest of the world will have a huge amount of power over the future. For an extreme case, the closed project might literally be developing AI in secret, without notifying the rest of the world, as described in [Training AGI in Secret would be Unsafe and Unethical](https://www.lesswrong.com/posts/FGqfdJmB8MSH5LKGc/training-agi-in-secret-would-be-unsafe-and-unethical-1). A secret intelligence explosion would be bad for misalignment risk, because far fewer people would know and understand what’s going on, and be able to give input on the alignment techniques, and would be bad for the distribution of power, because powerseeking individuals might be able to seize control of the AGI project, and build ASI aligned with their values (as opposed to humanity more broadly)
A better proposal for avoiding broad deployment is to do local deployment, but maximize transparency: publish evaluation results, publish demos of AIs doing very impressive things, publish the AI-generated research (on non-AI domains), publish the model spec, etc. This is far better than a secret intelligence explosion. However, it seems likely that if this is happening, humanity will be still largely in the dark about what’s going on, because to normal people, without AI impacting their day to day life, it’ll seem like some far-away science project that people like to talk about—it’ll be in far mode, not near mode. Therefore, it seems likely that broad deployment of AI will improve epistemics around AI in particular, which seems positive.
## Appendix: How plausible and how desirable are our key policies?
Plan A is neither a pure recommendation nor a pure prediction. A pure recommendation in which every actor makes maximally responsible moves starting tomorrow would have been too implausible. And we already wrote a pure prediction scenario—that's AI 2027. So we chose to make Plan A a compromise between plausibility and desirability.
The interactive graph below shows roughly where each of the scenario's policies sits on these two dimensions. Plausibility is judged at the point in the scenario when the policy is adopted, conditional on everything that has happened up to then. It does not represent how likely the policy is to happen starting from today's world. For instance, "Pause at top human level" is rated highly plausible not because we think a pause is highly likely from where we stand in 2026, but because in a world where the deal has held for years and control-based safety cases are visibly reaching their limits, we expect decisionmakers to choose something like it. Desirability measures how much better we currently expect a policy to be than the most plausible alternative at that same point in the scenario. It is not a ranking of which policies we're most keen to see adopted today. A low score can mean that our recommendation is subject to lots of unpredictable empirical updates or that alternative policies look about as good to us as our recommendation.
Click on a policy to read more.
less plausiblemore plausibleless desirablemore desirable
Click a point to read the policy’s explanation.
Policy plausibility vs. desirability \- ai-2040.com
1.
This is in scare quotes because, as our scenario illustrates, even if we do a well-enforced international deal to limit the pace of AI progress, the overall pace of AI progress will be extremely fast compared to what most people are expecting and used to. For more on why we think this, you can view our models of AI progress (the [AI Futures Model](https://www.aifuturesmodel.com)) and of AI economic impacts ([Economics of Plan A supplement](https://ai-2040.com/supplements/economics-of-plan-a)), as well as numerous blog posts and of course [AI 2027](https://ai-2027.com).
2.
AGI = Artificial General Intelligence, or, AIs that have a similar amount of general intelligence to humans. Our actual thinking uses more precise concepts which you can read about in the [AI Futures Model](https://ai-rates-calculator.vercel.app/). Concepts like AC, SAR, TED-AI. However, we find "AGI" to be a handy shorthand for some vague combo of the above milestones, because we expect those milestones to arrive within a few years of each other or less. So when we say 'AGI', understand that we are gesturing at milestones such as the above, and being deliberately nonspecific because the context doesn't require precision
3.
Eli’s timelines are similar to mine, but his takeoff would be about two times slower.
4.
Note: some people use takeoff speeds to refer to the period after TEDAI, it’s an ambiguous term, but for the purposes of this document, I’ll use the definition given here: the length of time between AC and TED AI.
5.
To illustrate, suppose that the length of time between AC and TEDAI is one month. Even if we can (optimistically) use compute governance to slow down rogue projects with ACs by 20x, you still only have 2 years until the rogue projects have TEDAI, which is a smaller window than Plan A assumes.
6.
This reasoning is given in the [datacenter buildout](/supplements/plan-a-assumptions#massive-datacenter-buildout) section below.
7.
Note that covert projects doing an intelligence explosion is only one of the reasons to be worried about them, there’s also the risk of them directly obtaining a decisive strategic advantage; so it would still be risky if they were, e.g., able to train a top expert level AI.
8.
As well as various aspects of takeoff besides speed, notably the possibility of fast progress on a small, fixed, amount of hardware.
9.
There’s a good chance that empirical feedback will be highly ambiguous, but overall I’m optimistic that if there’s a long period with broad access to experiment on roughly human level AIs, then we will surface a huge amount of evidence related to the difficulty of alignment. Given the current (poor) epistemic environment, I don’t think this evidence will be sufficient for broad scientific consensus (until we have a good fundamental understanding of AI), but I personally expect to learn a lot and change my mind in substantive ways.
10.
Note that pre-deal compute matters a lot because it seems probable that post-deal compute will be effectively monitored, because (1) there are very few extremely expensive fabs which can build the relevant chips, all of which can be directly monitored with auditors, and (2) it is intractable to build illicit fabs.
11.
Specifically, security to China, because China is probably the best placed actor to run a covert project.
12.
For example, we might need to do things like move all humans into bunkers that are robust to nanotechnology (friendly nanotech is defending), mirror life (no air is allowed to enter), and superpersuasion (all information coming in is mediated by friendly AIs, who screen out superpersuasive content).
13.
Thanks to the transparency of AI development to other companies, many governments, and the public, there are some failsafes in place for governments making poor decisions about safety techniques.
14.
Probably covert projects are somewhat of a concern, but it’s very unclear, and depends centrally on when international coordination to slowdown starts—the later a deal happens, the higher the risk of defection because (1) we’re closer to superintelligence, and (2) the smaller the personnel footprint is because the AIs can do a larger fraction of the tasks involved in running a covert project.
15.
If algorithmic efficiency improves at ~40x per year, then the number of years of progress _t_ lost by a 10x compute reduction is log_40(10) = .624 years = ~7.5 months.
16.
This appendix was written by Miles Kodama.
sources/ai2040-pages/security-in-plan-a.md
+170
View File
@@ -0,0 +1,170 @@
---
url: https://ai-2040.com/supplements/security-in-plan-a
---
## [AI 2040](/)
≡[PDF](https://ghjyjqzwz4.ufs.sh/f/9qHa0cBclQ7sf6FVzqbp2gHFXC1IbGS5ZA7DmNBusfM4YJRP)
[Supplements](/supplements)
Plan A Basics
[FAQFAQ](/supplements/faq)[How Plan A solves our 5 biggest problemsHow Plan A solves our 5 biggest problems](/supplements/how-plan-a-solves-our-5-biggest-problems)
Plan A Policies
[Covert AI ProjectsCovert AI Projects](/supplements/covert-ai-projects)[Verification PlanVerification Plan](/supplements/verification-plan)[Transparency PlanTransparency Plan](/supplements/transparency-plan)[Capability Scaling StrategyCapability Scaling Strategy](/supplements/capability-scaling-strategy)[Security in Plan ASecurity in Plan A](/supplements/security-in-plan-a)
Analysis of Plan A Policies
[Plan A AssumptionsPlan A Assumptions](/supplements/plan-a-assumptions)[Comparing Possible PlansComparing Possible Plans](/supplements/comparing-possible-plans)[Deal DeclineDeal Decline](/supplements/deal-decline)[Takeoff SupplementTakeoff Supplement](/supplements/takeoff-supplement)
Economics
[Economics of Plan AEconomics of Plan A](/supplements/economics-of-plan-a)[Compute in Plan ACompute in Plan A](/supplements/compute-supplement)[Economic Growth ExplorerEconomic Growth Explorer](/supplements/econ-explorer)
Speculative Analysis
[Space Governance PlanSpace Governance Plan](/supplements/space-governance-plan)[AI for EpistemicsAI for Epistemics](/supplements/ai-for-epistemics)[Alignment RoadmapAlignment Roadmap](/supplements/alignment-roadmap)
[About](/about)
[![AI Futures Project](/ai-futures-logo.svg)](https://ai-futures.org/)
# Security in Plan A
### Romeo Dean, Thomas Larsen
Security is an area where expertise and access to classified information is particularly important for having an informed opinion. Nonetheless, this supplement provides a high-level overview of the security needed in Plan A in our best understanding, and why we think it is tractable.
We think it useful to think about three types of security, in increasing levels of difficulty from our point of view:
1. **Model weights security:** Defending the exfiltration of model weights, and other large datasets.
* Metric we care about: Exfiltration bandwidth. I.e., how much can the attacker steal in a given amount of time?
2. **Verification integrity security:** Can we trust the results of our verification measures?
* Metric we care about: Assurance level (the confidence-coverage curve explained in the verification supplement). I.e., how confident are we that the maximum unverified compute usage was less than X% (for values of X between 0 and 100%).
3. **Algorithmic confidentiality:** Defending the exfiltration of code, algorithmic secrets, and anything else that doesn’t classify as ‘large datasets’.
* Metric we care about: We are unsure. On the one hand, we could think about algorithmic security in terms of exfiltration bandwidth as well, but algorithmic secrets might be tiny and have an unpredictable mapping from data size to usefulness. Therefore it’s probably best to think about the percentage of algorithmic efficiency that is exfiltrated and usable by the attacker.
Throughout this supplement, we assume an [OC5 attacker](https://www.rand.org/pubs/research_reports/RRA2849-1.html) per RAND’s definition, because in Plan A the US and China will be worried about each other being potential attackers.
## Plan A security summary
In our scenario, we assume that SL5 security for model weights and to detect verification tampering is **viable with extreme effort; this is our low confidence best guess. It may turn out to be impossible** , in which case Plan A would need to be modified, which we discuss [here](/supplements/plan-a-assumptions#less-central-updates).
The security level desirable in Plan A is as follows:
| **Model Weights**| **Algorithmic Secrets**| **Verification**
---|---|---|---
**Inference Datacenters**| SL5-100TB-5y(Maximum exfiltration of 100TB over 5y)| n/a| SL5(increasing assurance over time as compute scales)
**R &D Datacenters**| SL5-100TB-5y| Depends on the transparency regime. In Total Research Transparency, most are intentionally transparent, small fraction SL5.| SL5(increasing assurance over time with compute scale)
## Model weights security
Goal: We think the goal for model weights security in Plan A requires that frontier model weights (and similarly large important datasets) cannot be exfiltrated and used by potential adversaries and/or covert projects.
Concretely, we think the goal should be to defend against around 100TB of exfiltration over 5 years. Which we define as: weights security level 5, for 100TB over 5 years, or SL5-100TB-5y for short. We don’t think this will be trivial, but we do think it is highly tractable.
The following figure shows the size we expect frontier models to be in Plan A (and we also expect other datasets to be of similar magnitude, since data and parameters should continue to trade off with each other in compute optimal scaling).
**Frontier model size, 2022 to 2040**
Model sizeTraining compute
reportedours (α=0.5)α ∈ [0.2, 0.8]Nesov
1101001k10k100k1M2022202620292032203520382040Trillion parameters2026 anchor2029 dealGPT-4 ~1.8Tα=0.8α=0.2~650T (Nesov)31,000T~158T
### Plan A implementation
The **inference data centers** have SL-5 model weights security via defenses including:
* They are surrounded by an information boundary, which is the datacenter walls, including a copper box (a faraday cage) preventing signals from coming in or out, except for one cable in and out of the datacenter, and a secure exchange chamber for other things that need to cross the perimeter.
* For the output cable (which needs to exist, in order to communicate with the outside world), we make sure that the only information coming out of that wire is verified tokens produced by whitelisted models. You can read more in our [verification supplement](https://www.ai-2040.com/supplements/verification-plan). This is also similar to the proposal in [this paper](https://arxiv.org/abs/2511.02620).
* For the other things that need to cross the boundary, power in, heat out, people in and out, we have further side channel mitigations and total bandwidth capping.
The **R &D datacenters** have two deliberate channels that allow information to leave the datacenter: (1) physically sending model weights to the inference datacenters, and (2) the fully transparent output channel used by researchers and for queries.
* **The transportation of the model weights to the inference datacenters.** When a model is approved for public deployment, a package including the weights file, and any approved inference code (e.g., control scaffolds) gets downloaded on devices encrypted (sequentially) by both the US and China and physically escorted by both sides to inference clusters. At the inference clusters, they are added to the inference whitelist to allow for inference verification on public deployment.
* **The transparent output channel** caps the total bandwidth coming off the datacenter at 1 MB/s, which means that over a 5 year period, 157 TB leave the datacenter. This is enough to let 50,000 people continuously read outputs at around 10 tokens / second (which might have thousands of corresponding internal chain of thought / scaffold tokens that don’t need to be outputted). Model weights of a current frontier model are currently on the order of 1T parameters (which corresponds to 1TB), so exfiltrating model weights via this channel would require hijacking .6% of the (fully transparent) output channel continuously for 5 years (or a larger fraction for a smaller duration of time), which would very likely be noticed. In Plan A, with compute scaling favoured over algorithmic progress, we expect the model weights of frontier models to become closer to 100T parameters, so a full exfiltration would require a continuous 60% of the output bandwidth for 5 years, ignoring compressibility which shouldn’t be a huge factor. There are additional layers of defense against model weight exfiltration within the datacenter: notably, we attempt to keep model weights on the opaque internal database.
The remaining threat models involve side channel attacks (such as staff members carrying small flash drives), and we don’t have a creative solution here: access to all of the data centers is tightly controlled, and well-funded, creative security teams are constantly red teaming and hardening the facilities. It also helps that the datacenters in Plan A are built from scratch in areas with low background radiofrequency (RF) noise with security top of mind.
Finally, the model weights also need to be regularly transported between the R&D datacenters (where they are trained) and the inference datacenters. We imagine that this could happen by transporting the model weights using several physical storage devices that are locked and continuously monitored by representatives of both the US and China for the entire duration of the transportation.
## Verification integrity
Goal: The goal for verification integrity should be to achieve a desirable assurance curve: The assurance curve captures information about the level of **coverage** (the % of the AI compute that the verification solution makes claims about) and **confidence** (with what probability is the verification solution correct about the claims it makes on the covered resources). We define an assurance curve as a set of relevant (coverage, confidence) values.
For example, with the middle curve, there is 99.99% confidence that 99.9% of the compute was verified to have been used correctly. Of course, such curves are theoretical and will be hard to measure, but conceptually we think its useful to think about the verification integrity goal in terms of this curve.
We think that the verification integrity goal should compare the risk incurred from verification integrity subversions (i.e., internal rogue compute uses) to external ongoing risk from other sources of rogue compute usage, i.e., covert projects, and aim for the possible internal rogue compute uses to be negligible in terms of additional risk vs. the external uses. Since internal rogue compute might have access to the opaque internal information (frontier weights, datasets and potentially some defended algorithms/code depending on the transparency regime) it is potentially much riskier than an equivalent amount of external covert project compute usage. A good approximation of ‘how much riskier’ might be to consider the gap in algorithmic efficiency available inside the R&D datacenters vs. in the potential covert projects.
Concretely, we think we should take something like the 90th percentile estimate of covert project effective compute (in terms of the frontier project’s current software level, so it's scaled to be comparable to internal compute use), and then set the verification integrity target such that we are 90% confident that the coverage of internal compute usage is at the level X (e.g., X=99.999% of the internal compute) such that the potential rogue internal compute (1-X) would be 10x smaller than the covert project’s effective compute. This basically would be like saying “we are 90% confident that rogue internal deployments would be about equivalent to a 10x smaller covert project than what we think might exist.”
### Plan A implementation
Verification in both R&D and inference datacenters needs to be mutually trusted by the US and China, so that they are both confident only the approved training runs, experiments, and inference workloads are running. In our concrete proposal, inference and R&D verification both rely mostly on network taps for evidence collection and trusted recomputation servers for verification. In order to have durable trust in this verification solution, these devices should be mutually trusted (or alternatively, trusted [optical splitters and unilaterally trusted devices](https://nacicankaya.substack.com/p/catching-misreporting-about-ml-hardware-bd2)). There also needs to be physical security to check these devices don’t get tampered with or modified, and trust in the software stack they build on top of them (which probably means it should be extremely simple, and maybe even formally verified).
Outside of the verification devices themselves, verification will also rely on wider security properties of the cluster that overlap with model weights security. For example, there not being side channels that could let an attacker smuggle out the results of rogue internal computations.
Over time, the verification assurance will need to improve drastically with the scale of compute. We discuss this in more detail in our [verification supplement](https://www.ai-2040.com/supplements/verification-plan).
## Algorithmic security
Goal: This depends on the transparency regime that we are aiming for, which is discussed in our [transparency supplement](https://www.ai-2040.com/supplements/transparency-plan).
More concerned aboutpoor regulationMore concerned aboutcovert projectsProposal #1:Radical TransparencyProposal #2:Filtered TransparencyProposal #3:Algorithmic SecurityTransparency proposals spectrum \- ai-2040.com
### Plan A: Total Research Transparency
To a first approximation, there will no longer be any algorithmic secrets in AI under the Total Research Transparency version of Plan A. There are a few exceptions:
1. Massive (potentially synthetic) datasets have similar properties to model weights (in particular, are very large), and so it is viable to limit their output via upload limits out of the R&D datacenter. This isn’t sufficient for preventing the exfiltration of any given data; but could prevent the exfiltration of most data. Data progress is a substantial fraction of overall algorithmic progress, and so securing data might be both tractable and important.
2. Hyperparameters found via computationally intensive procedures. For some hyperparameters, efficient settings for them are found via expensive procedures like sweeps over the possible values, and then tests for the training run. These values are small, but do not have to be known by any human. The code for selecting these hyperparameters can be transparent, but we can prevent anyone from knowing the value that running the code found, because it is computationally expensive to determine the value.
1.
Initially, this simply means “when the company that owns the model decides to deploy it publicly.” Later there’ll be more regulations, which may or may not take the form of a formal pre-deployment approval process.
2.
The process for downloading the weights file is extremely controlled, with both US and Chinese auditors unlocking a secure enclosure where there is a secure device that can make download requests to the opaque internal database and encrypt the files. It is the only device in the entire cluster with a usb (or similar) port that allows any physical device insertion for downloading.
3.
1 MB/s × 60 × 60 × 24 × 365.25 × 5 = 157 TB.
4.
This assumes FP8 parameters, i.e., that each parameter is 1 byte.
5.
In Plan A, we also recommend deliberately increasing the size of the model weights in order to get the security benefit of the weights files being bigger and thus harder to exfiltrate. This would require additional training compute because you are no longer selecting model size to be on the [optimal compute-optimal curve](https://arxiv.org/abs/2203.15556).
6.
Cheap compression that requires tiny amounts of compute, and therefore might go undetected by the verification measures, should not go beyond a factor of 2-4x, e.g., through [huffman encoding](https://en.wikipedia.org/wiki/Huffman_coding) (lossless) or cheap post-training quantization techniques (e.g., [](https://arxiv.org/abs/2210.17323)[GPTQ](https://arxiv.org/abs/2210.17323), [](https://arxiv.org/abs/2306.00978)[AWQ](https://arxiv.org/abs/2306.00978), [](https://arxiv.org/abs/2211.10438)[SmoothQuant](https://arxiv.org/abs/2211.10438), [](https://arxiv.org/abs/2402.04396)[QuIP#](https://arxiv.org/abs/2402.04396), [](https://arxiv.org/abs/2401.06118)[AQLM](https://arxiv.org/abs/2401.06118)) which compress weights by 2-4× at negligible quality cost using typically 0.0001-0.01% of training compute. For compression attacks that go further, e.g., more aggressive quantization or LoRA-style decomposition (e.g., [](https://arxiv.org/abs/2106.09685)[LoRA](https://arxiv.org/abs/2106.09685), [](https://arxiv.org/abs/2305.14314)[QLoRA](https://arxiv.org/abs/2305.14314), [](https://arxiv.org/abs/2310.11454)[VeRA](https://arxiv.org/abs/2310.11454), [](https://arxiv.org/abs/2402.09353)[DoRA](https://arxiv.org/abs/2402.09353), [](https://arxiv.org/abs/2303.10512)[AdaLoRA](https://arxiv.org/abs/2303.10512)), which might have >1000x compression, detection of the unauthorized computation to do this compression becomes the more load bearing security property than the bandwidth limit itself. Even still, 100TB weights being compressed 10,000x to 10GB still means you need to use 0.1% of the output channel for 116 days (1 KB * 60 * 60 * 24 * 116 = 10GB) without detection (which might require a large hit from steganography). Regulation can always be aware of the risks of compression and do red-teaming to determine how tractable it is, and adjust training run rules and/or the output bandwidth cap accordingly.
sources/ai2040-pages/space-governance-plan.md
+863
View File
@@ -0,0 +1,863 @@
---
url: https://ai-2040.com/supplements/space-governance-plan
---
## [AI 2040](/)
≡[PDF](https://ghjyjqzwz4.ufs.sh/f/9qHa0cBclQ7sf6FVzqbp2gHFXC1IbGS5ZA7DmNBusfM4YJRP)
[Supplements](/supplements)
Plan A Basics
[FAQFAQ](/supplements/faq)[How Plan A solves our 5 biggest problemsHow Plan A solves our 5 biggest problems](/supplements/how-plan-a-solves-our-5-biggest-problems)
Plan A Policies
[Covert AI ProjectsCovert AI Projects](/supplements/covert-ai-projects)[Verification PlanVerification Plan](/supplements/verification-plan)[Transparency PlanTransparency Plan](/supplements/transparency-plan)[Capability Scaling StrategyCapability Scaling Strategy](/supplements/capability-scaling-strategy)[Security in Plan ASecurity in Plan A](/supplements/security-in-plan-a)
Analysis of Plan A Policies
[Plan A AssumptionsPlan A Assumptions](/supplements/plan-a-assumptions)[Comparing Possible PlansComparing Possible Plans](/supplements/comparing-possible-plans)[Deal DeclineDeal Decline](/supplements/deal-decline)[Takeoff SupplementTakeoff Supplement](/supplements/takeoff-supplement)
Economics
[Economics of Plan AEconomics of Plan A](/supplements/economics-of-plan-a)[Compute in Plan ACompute in Plan A](/supplements/compute-supplement)[Economic Growth ExplorerEconomic Growth Explorer](/supplements/econ-explorer)
Speculative Analysis
[Space Governance PlanSpace Governance Plan](/supplements/space-governance-plan)[AI for EpistemicsAI for Epistemics](/supplements/ai-for-epistemics)[Alignment RoadmapAlignment Roadmap](/supplements/alignment-roadmap)
[About](/about)
[![AI Futures Project](/ai-futures-logo.svg)](https://ai-futures.org/)
# Space Governance Plan
### Mia Taylor (Forethought), Fin Moorhouse (Forethought), Max Dalton (Forethought), Thomas Larsen (AI Futures Project)
# Preliminaries
It should be possible for almost all humans to get almost everything that they want from a post-ASI world. With huge numbers of robots capable of almost any task a human can do, it will be possible to provide everyone with the living standard that only the richest people today enjoy. People could spend their time doing what's most meaningful to them—raising children, creating art, or spending time with friends. With ASI-driven advances in medicine, we'll be able to treat mental illness, chronic pain, aging, cancer, and other diseases that shorten or worsen people's lives today. In short, if we achieve the future described in Plan A, there will be enough to go around to give everyone a very good life. Many of the tough tradeoffs that are the focus of modern politics will be resolved. So in some sense, these decisions need not be extremely fraught.
Yet, at the same time, the decisions we make are extremely high-stakes. If we handle things well, then we could create an enormously flourishing future for our descendants. But there are many ways we could waste this opportunity. We could fight over the resources, leaving a few people in control of the future and everyone else with nothing, or burning vast resources in internecine conflict. Even if we avoid those pitfalls, we could still misuse resources—squandering them on positional goods or failing to leverage ASI to figure out how best to achieve what we truly care about.
We want a governance system that will allow us to make wise decisions from a foundation of abundance and cooperation.
This document focuses on how to decide what to do with resources outside the solar system (henceforth referred to as “space resources”). This is an important aspect of post-ASI governance—the overwhelming majority of accessible resources are outside of the solar system and the overwhelming majority of future beings may live outside our solar system—but it is not the only important question.
## Epistemic status
We’re currently not confident in how the treaty signatories in Plan A should approach post-ASI governance: it depends on a lot of details about people’s preferences, technological capabilities, and optimal coordination mechanisms that are only partly worked out.
Luckily, by the time the treaty signatories are making these decisions, they should have superintelligent automated researchers investigating the nature of the universe, doing moral philosophy, and figuring out optimal coordination mechanisms. And we’ll have AI therapists, advisors, and negotiators to help us understand our preferences and make deals with other actors.
So we think the actual best plan is to defer many of these questions until we have ASI-driven advice and coordination. However, there is value in concreteness and in sharing best guesses, so we wanted to provide some analysis. But we wouldn’t want readers to anchor too much on the ideas we present below.
## What’s special about post-ASI governance?
Post-ASI governance will differ from historical precedent in several important ways.
Technological progress may quickly unlock new pools of resources that no one has previously exploited. The most important example is resources outside of our solar system.
* Because these resources have been unused, there are no historical claims to navigate. We should aim to govern them in a just and equitable way.
* There will likely be extreme abundance relative to the present day. With just a tiny fraction of available resources, we’ll probably be able to give everyone alive today a long, happy, healthy, and extremely prosperous life by current standards.
* We think that the most important question that governance will have to face is what to do with the huge quantities of remaining resources after we’ve ensured that everyone alive has enough to live an incredible life. Some salient options include:
* _Personal consumption_. People could develop ever-more-expensive ways to derive enjoyment from resources. But most resources are probably poorly suited to use for personal consumption. Most resources are outside our galaxy and thus [millions of light-years away](https://en.wikipedia.org/wiki/Andromeda_Galaxy). If you want to stay on Earth and consume anything—matter, energy, information—derived from another galaxy, you’d have to wait millions of years. If, on the other hand, you left Earth for another galaxy, then if you ever returned, you’d find that millions of years had passed.
* _Positional goods_. People could spend resources on a zero-sum competition to have the most impressive galactic-scale art projects, or the largest number of statues of themselves. Positional goods could soak up a lot of resources. While there are probably generally diminishing returns to consuming more resources as people get richer, this might not hold for positional goods, since status is relative. We think it would probably be a mistake to spend huge quantities of resources on this type of zero-sum competition.
* _Satisfying moral preferences_. We could create societies full of flourishing people, or enormous nature reserves. We think this is the most promising way to use the cosmic endowment. But then the question becomes: how should we decide which moral preferences to pursue?
Technological progress will also change _how_ we make governance decisions:
* We may be able to make decisions with lasting effects far into the future.
* For example, we could hand over resources or hard power to ASI systems that have been aligned to a particular set of values. This will likely give us the option of creating [extremely long-lasting institutions](https://www.forethought.org/research/agi-and-lock-in).
* Typically later generations adopt somewhat different values than their parents, and as a result society’s values change over time. But [new technologies](https://www.forethought.org/research/preparing-for-the-intelligence-explosion#value-lock-in-mechanisms) [may enable parents](https://www.lesswrong.com/posts/8aRFB2qGyjQGJkEdZ/christian-homeschoolers-in-the-year-3000) to exercise tighter control over the values that their children develop.
* People will have access to superhuman advice and coordination assistance.
* This means people will be better able to identify and advocate for policies that truly achieve what they care about.
* But we’ll probably also need to make our governance structure more robust than ever to people trying to game it, either individually or in a group. Just because a governance structure has not been severely exploited in the past does not guarantee that it won’t be severely exploited in the future.
And we will have large responsibilities to make these decisions go well for people who aren’t able to participate:
* The overwhelming majority of the affected parties probably won’t be able to participate in the decision to govern space directly, because they won’t have been born yet. This is already true to some extent—in modern democracies, citizens set policies for non-citizens, adults decide for children, humans decide for animals—but in the post-ASI setting this becomes far more extreme. Present people may make decisions that affect the lives of vast numbers of people who cannot give input. The responsibility to get these decisions right is enormous.
## Evaluating proposals
In this appendix, we’re evaluating proposals based on how well they achieve the following desiderata, which we believe will be broadly appealing across different value systems.
* **Preference satisfaction**. The governance system should try to satisfy all the participants’ preferences as much as possible. Some aspects of this include:
* _**[Paretopianism](https://aiprospects.substack.com/p/paretotopian-goal-alignment)**_. Given a massive windfall, it should be possible to make nearly everyone vastly better off than they are today, and a good proposal should do exactly that.
* _**Limit extreme downside**_. Our post-ASI governance proposal should have a very low chance of producing outcomes widely considered extremely bad, such as [huge amounts of suffering](https://centerforreducingsuffering.org/research/a-typology-of-s-risks/), or all of space being destroyed.
* _**Satisfy bounded views**_. If someone has a preference or moral view that is not very costly to satisfy, we should satisfy it. For example, on a cosmic scale, it’s fairly cheap to ensure that every human alive in 2040 has a long, flourishing life.
* **Satisfy common-sense ethics**. The process should not violate common-sense ethical views. This weighs against proposals where a small number of people predictably end up controlling the fate of the universe or where future generations end up totally disempowered.
* **Satisfy objective morality**. Some aspects of ethics may be objective and discoverable. If so, we should try to identify and conform to them.
These next criteria aren’t terminal goals, but we think that processes that satisfy them are more likely to achieve the criteria above.
* **Coordination**. The governance system should enable coordination on large-scale collective action problems. For example, if we encounter alien civilizations, we may need to coordinate to establish friendly relations with them. We may also need to coordinate to avoid actions that risk destroying all of Earth-originating civilization, or to avoid space races where factions compete to claim as many resources as possible, burning huge amounts of resources in the process. These goals probably require some minimal government functions: protecting property rights, enforcing prohibitions, and potentially collecting taxes to fund public goods.
* **[Moral trade](https://amirrorclear.net/files/moral-trade.pdf)**. It might be possible to nearly satisfy many different moral viewpoints simultaneously. For example, some views might especially care about resources close to Earth or the near future, while others might be more indifferent across time and space: we might satisfy both by letting the former choose what happens in the Milky Way while the latter choose what happens in the rest of the universe. Here are two types of trade that we want our proposal to facilitate.
* _**Bilateral trade**_. This is particularly well suited for satisfying uncommon preferences. These trades are most straightforward in market systems, where two people can literally trade deeds to resources or sign agreements about how they will use their resources. But it’s possible that some voting systems–e.g., [the](https://en.wikipedia.org/wiki/Vickrey%E2%80%93Clarke%E2%80%93Groves_mechanism) [Vickrey-Clarke-Groves mechanism](https://en.wikipedia.org/wiki/Vickrey%E2%80%93Clarke%E2%80%93Groves_mechanism)—could also achieve similar results.
* _**Coordination on[moral public goods](https://www.forethought.org/research/moral-public-goods-are-a-big-deal-for-whether-we-get-a-good-future)**_. People might share some values while also holding idiosyncratic preferences—individually preferring to pursue their own preferences, but collectively preferring that everyone work toward the shared values.
Coordination on moral public goods is most straightforward in voting systems, since people can just vote to devote resources to shared projects. It’s possible that voluntary contracts—where people pledge to spend resources on shared goods if others do the same—could achieve similar results, but this is less straightforward because of [free-riding incentives](https://www.forethought.org/research/moral-public-goods-are-a-big-deal-for-whether-we-get-a-good-future#appendix-b-causal-coordination-through-voluntary-contracts).
It might be particularly effective to coordinate to switch from funding positional goods to funding moral public goods. If done in a way that leaves people's relative ability to purchase positional goods unchanged, then this is great for everyone: relative status doesn't change, but people are able to satisfy their non-positional preferences, too.
* **Reflection**. ASI will likely uncover novel considerations and empirical facts highly relevant to how people want to use space resources (e.g., are there objective moral truths, and what are they?). We want each person to have the time and resources necessary to think carefully about what they value before making important decisions, and we want to ensure that we invest in ASI-fuelled research on important questions to inform people’s decisions.
* **Pluralism**. Humans today have many different visions of the good. It’s possible that after deep reflection with ASI we’ll converge on a single set of shared values. More likely, we’ll still have some disagreements. Where people have different value systems, we prefer proposals that will allocate resources to pursue all of them.
* **Avoid brittle decision-making structures**. There are likely many important considerations that we’re not yet aware of. Where possible, we favor flexible decision-making structures that won’t make it difficult to solve unforeseen problems.
* **Fairness**. Each living adult should have an opportunity to give equal input into the process. We don’t think that this criterion necessarily means that each person ends up controlling an equal quantity of resources. For one thing, some people might care a lot about what happens with resources closer to Earth while not caring much about resources in distant galaxies, and we’re happy for them to trade with people who care a lot about distant galaxies. Nor do we think that this criterion requires that everyone is equally happy with the outcome. For instance, if some decision is made by vote, then some people might end up voting for the losing side. But we want a neutral process that doesn’t privilege any particular people or viewpoints over others.
* **Political feasibility**. Unlike the other criteria, this is purely pragmatic: a proposal must be acceptable to the relevant decision-makers to be implemented. It’s unclear who these decision-makers would be, but they could include the citizens and/or leadership of the US, China, or other powerful countries. It might also include AI company CEOs or shareholders.
# Summary of the most promising options for allocating space resources
We think that it’s best to decide on a specific structure for allocating space resources in 2040 at the time that the treaty is decided ([more discussion here](/supplements/space-governance-plan#should-we-commit-to-a-process-right-away)).
We’re currently fairly uncertain about what the best structure is, and that decision should be deferred until decision-makers have access to aligned ASI advice. But currently, these seem like the best object-level proposals.
Basic proposal
**Direct endowment + trade**(with prohibitions). Each adult is assigned an equal share of resources. They can trade, donate, give away, or use those resources in any way they want, except for a small category of prohibited activities (more).
Key advantages
Most views held by living humans will have at least some resources devoted to them.
Bilateral trade between people with different moral views is very straightforward.
Key disadvantages
It’s harder to coordinate to fund moral public goods due to free-rider problems.
Basic proposal
**Vote on how to use resources**. Each adult gets to participate in a vote on how to use space resources (more).
Key advantages
It’s straightforward to coordinate to fund moral public goods.
Key disadvantages
All known voting systems have serious issues, such as:
* Inefficiencies, e.g., strong minority preferences are overruled by very weak majority preferences.
* Incentives to strategically misrepresent preferences, either alone or in collusion with others.
Basic proposal
**Direct preference aggregation**. We use novel technology to directly elicit people’s preferences (e.g., through an interview with an ASI using lie detection) about the use of space resources and aggregate these preferences (more).
Key advantages
Many voting systems fail to get a globally-optimal outcome because people strategically misrepresent their preferences. But direct endowment fails to fund moral public goods. By eliciting people’s honest preferences and then aggregating them, we might be able to get the best of both worlds.
Key disadvantages
People might object to the preference elicitation step, especially if the method is time-consuming or invasive.
**Basic proposal**| **Key advantages**| **Key disadvantages**
---|---|---
**Direct endowment + trade**(with prohibitions). Each adult is assigned an equal share of resources. They can trade, donate, give away, or use those resources in any way they want, except for a small category of prohibited activities (more).| Most views held by living humans will have at least some resources devoted to them.Bilateral trade between people with different moral views is very straightforward.| It’s harder to coordinate to fund moral public goods due to free-rider problems.
**Vote on how to use resources**. Each adult gets to participate in a vote on how to use space resources (more).| It’s straightforward to coordinate to fund moral public goods.| All known voting systems have serious issues, such as:
* Inefficiencies, e.g., strong minority preferences are overruled by very weak majority preferences.
* Incentives to strategically misrepresent preferences, either alone or in collusion with others.
**Direct preference aggregation**. We use novel technology to directly elicit people’s preferences (e.g., through an interview with an ASI using lie detection) about the use of space resources and aggregate these preferences (more).| Many voting systems fail to get a globally-optimal outcome because people strategically misrepresent their preferences. But direct endowment fails to fund moral public goods. By eliciting people’s honest preferences and then aggregating them, we might be able to get the best of both worlds.| People might object to the preference elicitation step, especially if the method is time-consuming or invasive.
(We discuss alternative proposals that we like less [here](/supplements/space-governance-plan#unpromising-options)).
Other important decision points include:
* At what time should votes, preference elicitation, or distribution of resources happen? (discussion [here](/supplements/space-governance-plan#early-decisions-vs-late-decisions))
* Should later generations get a say? (discussion [here](/supplements/space-governance-plan#later-generations))
* Should we prefer a hybrid approach that allocates some resources according to one proposal and allocates other resources according to other proposals? (discussion [here](/supplements/space-governance-plan#one-process-or-many-processes))
* How do we enable each decision-maker to make the best decisions by their own lights? (discussion [here](/supplements/space-governance-plan#reflection))
# Some available options
Here, we discuss options for governing future resources. We begin by describing a couple of notable options which seem relatively unpromising on the criteria above, then turn to proposals that we find more promising. In both cases, we’re trying to cover important and illustrative cases, not every possible outcome.
## Unpromising options
### Anarchy
No property rights system is established for space resources. To use a resource, an actor must either be the first to claim it or seize it from another actor. Actors can do whatever they want with resources they currently control—there’s no governance system for prohibiting particularly bad behaviors or any other forms of large-scale coordination (e.g., to fund public goods).
_Drawbacks:_ there could be significant negative-sum conflict over resources, and insecure property claims would make trade—including moral trade—harder.
Anarchy is worse than the feasible alternatives. We do expect that it is not a stable long-term equilibrium. Given much more advanced tools for coordination and communication, we expect people to create ad hoc governance structures. But a government that emerges organically from anarchy could be worse than one designed intentionally. In particular, it might favor the interests of those who were able to claim and defend lots of space resources early on.
### Finders-keepers property rights
Actors can claim unclaimed resources or they can buy resources from earlier owners, but they can’t seize claimed resources without the owner’s permission (similar to homesteading). As before, no effective system exists for large-scale coordination or public goods provision, and no governance system can enforce prohibitions.
_Strengths:_ we think this is an improvement on anarchy. Once a claim is recognised, the holder no longer needs to spend resources defending it. Moreover, secure and widely recognised property claims make trade possible.
_Drawbacks:_ value is no longer wasted contesting claimed resources but is still wasted in the race to claim them. It also seems likely that a small number of powerful actors—perhaps whoever can send out a wave of probes first—will claim most resources. And this proposal has no mechanism for public goods provision or enforcement of prohibitions, so it forgoes potentially large gains from moral public goods and from preventing egregiously unpopular uses of resources.
Next, we discuss governance proposals that we think improve on these scenarios according to the criteria above.
## More promising options
### Direct endowment and trade
Each person receives title to a portion of all future resources, which gives them the right to decide how those resources will be used in the future. For example, people can sell their shares of future resources, gift them to other people or to trusts, or [make bilateral deals with each other](https://www.forethought.org/research/should-we-lock-in-post-agi-agreements-under-uncertainty) about how they’ll use their resources. Property rights are well-enforced in practice. As before, no effective system exists for large-scale coordination or public goods provision, and no governance system can enforce prohibitions.
(A related approach: auction titles to all space resources and then redistribute the proceeds equally. In theory, if we make some standard assumptions, this scheme would result in the same distribution of ownership as direct endowment.)
_Strengths:_ when property rights are clear and well-established from the outset, bilateral trade—including moral trade—becomes easier. Compared to finders-keepers, there are no longer incentives to waste resources through racing.
Since each property-holder is free to decide what to do with their share of the future, at least some resources will be devoted to pursuing most moral views, making it likely that cheap-to-satisfy views will be satisfied.
_Drawbacks:_ moral public goods would have to be funded through voluntary contracts, and it’s not clear whether voluntary contracts will be sufficient to ensure that moral public goods are funded at the optimal level, since [individuals will face a strong incentive to free-ride](https://www.forethought.org/research/moral-public-goods-are-a-big-deal-for-whether-we-get-a-good-future#appendix-b-causal-coordination-through-voluntary-contracts).
Since direct endowment doesn’t involve a way to impose universal prohibitions on how to use future resources, the same drawbacks carry over from anarchy and finders-keepers: if there are some uses of resources which are egregiously bad by most people’s lights, but preferred by a few actors, it may be hard to prevent those uses, and bad actors could even threaten those uses to extract concessions. This is the mirror image of the moral public goods drawback: there is no easy way to coordinate to prevent moral public bads.
A fuzzier concern: we think it’s probably better for the current generation to think of themselves as stewards who seek to create the best future by their own lights in the space that they control, rather than as “space emperors” who try to squeeze the maximum personal enjoyment or aggrandizement out of their share of resources. If people get unilateral control over space resources _and_ they’re allowed to buy and sell them like normal property, we worry that this might push people to think of the resources as a personal windfall rather than a serious moral responsibility.
_Uncertainties and decision points:_ We’d also need to make a call on decisions like:
* Who gets resources? (For example, should resources be reserved for future generations?)
* Should people be allowed to sell, transfer, or wager their resources immediately? Or do we want to—somewhat paternalistically—enforce a waiting period for people to carefully consider their options?
* What is the relationship of a galaxy’s “owner” to the future inhabitants of the galaxy?
One important variant on this is to add some mechanisms for preventing people from using resources in egregiously bad ways. In the section [below](/supplements/space-governance-plan#mitigating-downside-risks), we discuss some possibilities.
### Voting on collective uses of resources
Everyone votes on proposals for what to do with space resources. These proposals could range from very concrete (e.g., “create digital minds according to this specification”), to very meta (e.g., “delay doing anything with space resources right now, and re-assess with another vote X years from now” or “allocate some future resources to private ownership”), or something in between.
_Strengths._ Compared to the other proposals so far, it’s easier to fund [moral public goods](https://www.forethought.org/research/moral-public-goods-are-a-big-deal-for-whether-we-get-a-good-future#scenario-1-causal-coordination)—if people prefer resources to go to a shared project rather than divided among all voters, they can vote for that proposal.
It’s also a more flexible plan: once future resources are transferred to private ownership, recovering the ability to make collective decisions might be impossible even if almost everyone wanted it. With some kind of regular vote, the electorate can switch to a different plan at the time of the vote, including agreeing to delay doing anything with space resources.
_Drawbacks:_ All known voting systems have significant problems. Some are simply inefficient: for instance, majority rule faces the familiar problem that the majority view can succeed when the majority wouldn’t deem the minority's preferred option much worse, but the minority greatly dislikes the majority’s preferred option. Others incentivize people to [misreport their preferences](https://en.wikipedia.org/wiki/Gibbard%E2%80%93Satterthwaite_theorem), individually or through collusion, to benefit themselves at the expense of overall value. This problem is especially serious when people know the details of the voting system well in advance and have ample time to develop strategies to game it with ASI assistance.
A further drawback is that voting requires synchronized decisions: everyone must vote at the same time. If the vote happens in 2100, you vote according to your preferences at that time, even if you would have preferred more time to reflect first. In principle, people could vote to delay the vote. But you could still face situations where a majority wants to vote now while a minority wants to keep reflecting, and the minority gets overruled. Contrast this with direct endowment, where each person can reflect for as long as they want before deciding what to do with their resources.
_Uncertainties and decision points_ : How badly can strategic voting distort outcomes in practice, and can novel technology (e.g., lie-detection technology) mitigate this? How well does voting handle bilateral trades between people with rare preferences? And we'd need to decide: when do we hold the vote, who is included, and which voting system do we use?
### Direct preference aggregation
Similar to voting, but instead of people casting votes, AI systems try to directly ascertain people’s preferences. For example, AI systems might build a profile of a person’s preferences from all their writing, extensive interviews perhaps under lie detection, and/or brain scans. Then, everyone’s preferences are aggregated to decide what to do with space resources. As with voting, the decision might be quite meta—e.g., wait 100 years and then do preference aggregation again.
It’s not clear how best to aggregate preference, but here’s a sketch of a proposal. Everyone is first assigned 1/N of future resources as a baseline—their fallback if no gains from trade are found. An AI arbiter then identifies reassignments of resources that would improve everyone's preference satisfaction above this baseline, splitting the gains fairly among all beneficiaries. This handles bilateral moral trade: if person A most wants statues of herself and person B most wants statues of himself, but both value statues of George Washington at 60% as much, the arbiter directs their combined resources toward Washington statues, leaving both better off than their baseline. It can also handle moral public goods: if funding some public good would improve everyone's welfare, the arbiter taxes each person in proportion to how much they value that good ([a Lindahl tax](https://en.wikipedia.org/wiki/Lindahl_tax)).
_Strengths._ Direct preference aggregation is a modification of the voting proposal above and inherits its strengths. Its major additional advantage is that it makes strategic manipulation more difficult than in voting systems.
_Drawbacks._ Eliciting people’s honest preferences would require some novel technology and it’s not clear what this process would look like. It’s possible that this process would be invasive, time-consuming, or unpleasant for participants (e.g., an interview while hooked up to a lie detector). It seems unjust and contrary to common-sense morality to require that people go through a procedure that they strongly object to in order to have input into the allocation of future resources.
There are a few more technical issues. For instance, people could strategically modify their preferences before assessment, breaking the main reason for trying to directly elicit people’s preferences (although mitigations might be possible—e.g., governments could ban people from modifying their preferences using novel technologies or require that they register their pre-modification preferences). We also haven’t spelled out a specific algorithm for doing the preference aggregation, and it could be difficult to avoid perverse or hard-to-predict outcomes, especially if many people are best represented as having inconsistent or incoherent preferences. Indeed, the question of how to represent someone’s preferences, including on questions they haven’t explicitly considered, seems very contentious and underdetermined.
### Defer to superintelligence
Instead of any particular method of inferring and aggregating people’s preferences, humanity builds aligned superintelligence to plan on its behalf and give it control of nearly all space resources.
For example, we could start with a “council” of superintelligent delegates, each starting from one of many diverse moral and philosophical traditions. Those delegates then engage in deep reflection and deliberation with one another. The delegates can initially be trained to be reliable and accurate on all verifiable domains, like forecasting and mathematics, and not post-trained to strongly avoid novel, ‘weird’, or otherwise unpopular views just because they are currently unpopular. If the superintelligences converged onto a single moral view, then they could devote space resources to that plan. If the superintelligences continued to have different views, they could come to a compromise plan via voting, direct endowment, or direct preference aggregation.
Or we could design an ASI system to be motivated to discover and act on objective moral truths, if they exist. (If they don’t exist, then it would revert to a backup plan—perhaps one of the proposals above).
_Strengths_. Determining the best plan under a particular moral view probably requires discovering and grappling with lots of novel and counterintuitive considerations. The resulting plan might seem [very weird and alien to humans](https://www.forethought.org/research/convergence-and-compromise#24-an-argument-against-wam-convergence), and they might not be motivated to act on it, even if they’re assured by a superintelligence that it is the morally best course of action. People might also have competing personal interests that take precedence over doing the right thing. But an ASI could potentially be designed to have no competing personal interests. These considerations are particularly strong if moral realism is true, and a plan like this might be our best shot at devoting substantial resources to pursuing objective morality if it exists.
_Drawbacks_. The ultimate outcome of this process might depend a lot on the details of the process—e.g., what initial viewpoints were represented on the “council” of deliberating ASIs? If this is true, who should decide? It seems rather unfair for the treaty signatories to get to choose.
We think that a process like this might be the best ultimate use of space resources. But if so it should probably result from one of the proposals above. For example, if many people found moral realism compelling after reflection, they could vote to spend public resources on the most morally valuable uses (or express a preference in direct preference aggregation). (It’s unclear whether the direct endowment proposal is sufficient for this: handing off resources to morally-aligned ASI is probably a moral public good in this scenario, so we’d expect it to be underfunded if people were individually deciding how to spend their resources).
# Other decision points
## Should we commit to a process right away?
The treaty signatories could approach this decision in three ways:
1. Decide on a specific process in 2040.
1. E.g., schedule an election in 2070 using [quadratic voting](https://en.wikipedia.org/wiki/Quadratic_voting) where all living people over the age of 16 vote on what to do with all extrasolar resources.
2. Decide on a process in 2040 for picking a process later.
1. E.g., schedule an election in 2070 using [approval voting](https://en.wikipedia.org/wiki/Approval_voting), where all living people over the age of 16 vote on what decision process to use.
2. E.g., design an ASI arbiter who will do a bunch of research and then pick a process that best satisfies some abstract desiderata put forward by the signatories of the Treaty in 2040.
3. Make no explicit decision about a process or a process for picking a process.
1. It’s not clear we’d end up deciding what to do with space, under this option. Perhaps at some point one of the signatories to the Treaty would decide that they wanted to colonize space and either do so unilaterally or negotiate with the other signatories to be allowed to do so.
Option 3 is probably worse than options 1 or 2. Having a concrete proposal—even a meta-level one about how to select an object-level process—allows public scrutiny, which we expect would push the proposal to be fairer and more principled. Without a concrete proposal, two failure modes become more likely. First, parties who especially wanted space resources might start lobbying the Treaty governments, and those who are well-connected or have invested heavily in lobbying would end up with outsized influence over how space resources are used. Second, some parties might start unilaterally colonizing space, which could trigger a space race. Governments could ban unilateral colonization, but that would require a plan for lifting the ban—at what point are people actually allowed to colonize?—which brings us back to option 2.
We’re less confident about whether option 1 or option 2 is better.
The main argument for option 2 over option 1 is that we’ll probably get more information about how to design a good process after 2040.
* We might discover new voting or preference aggregation systems that avoid issues with systems that we know about. However, we’re relatively pessimistic about learning much on this front after 2040, given existing [impossibility results](https://www.jstor.org/stable/1911219) showing that voting systems can't simultaneously satisfy a set of desirable features. Also, designing new preference aggregation systems is a relatively well-scoped, tractable mathematical problem so we would be somewhat surprised if we didn’t have the answer in 2040 after devoting a small amount of ASI labor to it.
* We might learn how well particular processes are likely to work in practice.
* For instance, we might learn about strategies that people are actually using to try to game the current system and adapt the rules accordingly, e.g., if we have rules intended to prevent parents from controlling their children’s values to an excessive degree, and it turns out that some parents can find loopholes to exert a lot of control, we could adapt the rules in response).
* We might discover new tools or capabilities that enable a better process.
* For instance, we might develop new methods for noninvasively scanning people’s brains to determine their preferences.
* We might discover new considerations that affect the optimal preference aggregation process.
* For instance, we might identify valuable forms of moral trade that we want our preference aggregation process to accommodate.
**On the other hand, people will also accumulate information about the likely object-level consequences of each possible process**. For example, suppose that we decided to vote in 2070 to decide on a procedure for deciding what to do with space, choosing between the policies of “hold a majority vote and do whatever the winning proposal says” and “direct endowment.” And at this time, it turns out that 50.1% of people favor colonizing the Milky Way while leaving the rest of space as a permanent nature preserve, while 49.9% have plans to colonize more distant galaxies.
In 2070, it would be clear that if we chose the majority vote, then all space resources would be allocated to the majority’s colonize-the-Milky-Way-and-nothing-else plan, but if we went with the direct endowment plan, then some galaxies would be left as nature preserves, while others would be colonized by the minority. Members of the colonize-the-Milky-Way-and-nothing-else coalition would then be strongly tempted to favor the majority vote when choosing a process.
At best, the process-revision vote becomes redundant—we might as well have voted directly on the object-level outcome. At worst, we lose the opportunity for people to agree on a process grounded in principles like fairness or Paretopianism.
One possible way around this would be to delegate process revisions to an AI arbiter that adjusts the process based on principles established in 2040, without regard for which factions benefit from any given change. This approach relies heavily on being able to verifiably align an AI system to a set of principles. It also requires being able to identify principles in 2040 that are specific enough for the AI arbiter to apply without heavy extrapolation, but not so specific that we forfeit much of the upside of deferring process decisions to a later date. But if it’s possible, this might be a good way to incorporate new information in a principled way.
## Early decisions vs. late decisions
Once we’ve picked an object-level process—like a vote, a preference aggregation procedure, or directly distributing deeds to space resources—there’s a question of when to run that process.
Here are some advantages of delaying the process:
* People have more time to reflect on what they care about and how they endorse using space resources before it’s time to make a decision. This matters most for voting and direct preference aggregation, where everyone must decide simultaneously.
* People may have access to more information at the time of the decision—e.g., a clearer sense of what is technically feasible to do with space resources.
* It’s possible to incorporate later generations into the process—e.g., allow them a vote, take their preferences into account for direct preference aggregation, or distribute resources directly. This seems good on grounds of common-sense ethics, as it’s plausibly unfair for the current generation to make decisions about the use of space resources on behalf of all future generations.
Here are some advantages of running the process relatively early:
* Some types of [potentially valuable moral trade](https://www.forethought.org/research/should-we-lock-in-post-agi-agreements-under-uncertainty) are possible only if participants are uncertain about the future, and people’s uncertainty about the future will probably decrease over time.
* People might want to bet on the outcomes of some reflective processes. For example, Alice might care deeply about maximizing positive impact if moral realism turns out to be true and discoverable, but care little about having resources if there are no objective moral truths. If Bob values resources equally regardless of whether moral realism is true, Alice and Bob might make the following bet: if ASIs discover objective moral truths, then Bob gives some of his resources to Alice. Otherwise, Alice pays Bob ([more discussion of these wagers here](https://www.forethought.org/research/should-we-lock-in-post-agi-agreements-under-uncertainty#three-kinds-of-early-agreement)).
* People might also want to bet on empirical facts. For instance, we may still have uncertainty about how many galaxies are within the reachable universe and which of those galaxies are already occupied by aliens. A fairly risk-averse person might buy resources that are likely reachable and unoccupied, in exchange for a larger amount of more uncertain resources, while a more risk-tolerant person might take the other side of the trade.
* These deals are most straightforward under direct endowment, but they could also work in voting systems, e.g., a coalition of moral realists and subjectivists might be willing to vote for a proposal that spent resources in the most moral way (conditional on moral truths existing and being discoverable) and otherwise distribute the resources to the subjectivists’ preferred purposes.
* Delaying decisions also gives people more time to figure out how to strategically misrepresent their preferences ahead of a vote or preference aggregation process.
* Running the process earlier may allow more flexibility.
* If an early vote is scheduled and delaying seems wise at the time of the vote, voters can simply vote to defer to a later vote. If resources are distributed early, recipients can take their time deciding what to do with them.
* On the other hand, if the process is scheduled for later, then we can’t make an early decision even if that would have been wise.
## Who should participate in the process?
In this section, we discuss who should participate in deciding how to use space resources. This is distinct from who should receive other rights, such as autonomy over their own lives or the right not to be mistreated. We discuss more about how those basic rights could be assured in a [later section](/supplements/space-governance-plan#mitigating-downside-risks).
### People with substantial bargaining power in 2040
It’s unclear who these people will be, but some plausible candidates are: AI company CEOs, AI company shareholders, the US president (and advisors), the US electorate, Chinese leadership, or the leadership and/or electorate of other countries that participate in the treaty.
This is probably the default outcome. It’s also very unfair, contrary to common-sense ethics, and could lead to many people’s cheap-to-satisfy preferences being neglected unless powerful people are sympathetic to them. We’re also concerned that if many people expect that this is how decisions will be made, then they’ll focus on jockeying over power during the intelligence explosion rather than cooperating to avert risks.
### All living humans
Another option is to share power among all humans who are alive in 2040, as in the Plan A scenario. This is fairer and more in line with common-sense ethics than the previous plan. And, to the extent that decision-makers can credibly signal that this will happen, committing to this plan might make people more cooperative during the intelligence explosion (although it’s unclear how credibly decision-makers could commit to this). Unlike proposals that share power with later generations (discussed [next](/supplements/space-governance-plan#later-generations)), this option makes it possible to make decisions relatively early, [which might be important](/supplements/space-governance-plan#early-decisions-vs-late-decisions).
On the other hand, it’s unclear how politically feasible this will be. And it doesn’t include everyone who might deserve a say in how space resources are used—it excludes future people and digital minds.
Some plausible options:
* **Include all living humans**. This seems like a fairly natural coordination point.
* **Include all living humans in some capacity, but give people with bargaining power extra weight**. For instance, more powerful people might get larger endowments or have their votes or preferences more heavily weighted. This might be a more politically feasible compromise.
### Later generations
“Future” generations are the generations born after the intelligence explosion. The “current” generation is the set of humans alive in 2040. We could include future generations in decisions by:
* Delaying votes (or rounds of preference aggregation) until future generations are adults.
* Reserving some decisions for future generations, e.g., allowing the current generation to vote on how to use 10% of space resources, but letting later generations decide at later votes how to use the remaining 90%, or directly distributing deeds to 10% of resources to the current generation, but saving 90% of resources for later generations.
One reason to share power with later generations is that it’s plausibly unfair not to. However, if we share power with future generations, then this potentially incentivizes parents to have as many children as possible and indoctrinate them to share the parents’ beliefs and values. This could let parents swing a later vote or ensure that a larger share of resources goes to people who share their values.
It’s not clear how big of a problem this is. On the one hand, it’s currently quite rare for ideological groups to deliberately try to have as many children as possible in order to increase their political and economic power. And we expect that most people would find the prospect of having huge numbers of children and brainwashing them to vote the parents’ way abhorrent. On the other hand, new technologies might lower the personal costs of having huge numbers of children (e.g., artificial wombs, robot caretakers). Other technologies could increase parents' ability to transmit their values to children (e.g., through genetic engineering or novel techniques based on advances in neuroscience or psychology). And a small number of bad actors might be able to substantially increase their population share within a few generations through rapid reproduction.
One way to navigate this is to place guardrails around the rate of reproduction or the extent of control—say, trying to ensure that parents could not use novel technology to reproduce substantially _faster_ nor exercise substantially greater control over their children’s values than early-21st century parents were able to.
Some plausible options:
* **Don’t include future generations directly**. Instead, let the current generation decide whether and how to include them.
* For example, if decisions were made by a vote, the current generation could vote to delay decisions until a later vote, where future generations could participate. (And analogously, under direct preference aggregation, if the current generation prefers to reserve some power for future generations, then that would happen).
* Under an equal distribution proposal, the current generation could share resources with their descendants or donate to trusts that gifted resources to later generations.
* (Although power granted to future generations is plausibly a moral public good, so we might expect it to be undersupplied in the equal distribution proposal).
* **Include later generations, with restrictions to prevent strategic reproduction and/or indoctrination of one’s children**. These restrictions could be chosen by a vote of the current generation.
* **Include later generations, with no restrictions to prevent strategic reproduction or indoctrination of one’s children**.
### Digital minds
Another difficult question is whether and how digital minds should have input into decisions about space resources. Digital minds could include AI systems or [simulated human brains](https://en.wikipedia.org/wiki/Mind_uploading).
In the Plan A scenario, by 2040, there may be huge numbers of digital minds. Some of these minds could be very sophisticated, with long-term stable preferences, human or superhuman intelligence, and perhaps even [subjective experience](https://en.wikipedia.org/wiki/Artificial_consciousness). On some views, such minds have moral status and deserve a say in future governance.
But this is pretty controversial and might remain so in 2050. Different people could have wildly different views depending on their beliefs about what makes a being worthy of inclusion in the political process. It seems challenging to decide in a value-neutral way which digital minds should be empowered, and choices here could radically affect the composition of the electorate.
There are further difficulties in integrating digital minds into the decision process. While it’s conceptually straightforward to give each human adult an equal stake in the decision—e.g., one vote, one share of space resources—it’s less clear how to individuate digital minds. Should two copies of the same mind running on different computers each receive a vote? If not, how different do two minds have to be before they count as different individuals?
The issues of strategic reproduction and indoctrination discussed above are even more severe for digital minds than for human children. New digital minds could be created as fast as copying data from one computer to another. And the creators of digital minds may have a huge amount of fine-grained control over the values and behavior of those minds. Given sufficient compute, it would be trivial to spin up billions of minds that sincerely want to vote exactly the way their creator tells them.
Some plausible options:
* **Include a subset of digital minds** where the concerns about gaming the process through strategic reproduction and indoctrination are less strong.
* Some possible subsets include:
* Allow digital minds who were created before 2040 to participate in the process, perhaps with some overall cap on the share of total votes or resources they could receive.
* (We would still somehow need to resolve the issue of individuating digital minds and deciding which digital minds deserved votes).
* Allow digital minds that are uploads of deceased biological humans to participate in the process, one upload per original human.
* Allow people to “adopt” digital minds and enfranchise those adopted digital minds, but only if the creators exercise no more control over the values and beliefs of those digital minds nor are producing digital minds at a faster rate than parents in 2026 could with biological children.
* Most likely, most digital minds would not be enfranchised under any of these proposals.
* **Don’t include digital minds directly**. Instead, let biological humans decide whether and how to include them.
* How this decision is made would depend on the proposal:
* For example, if decisions were made by a vote, the biological humans could vote on a proposal to defer decisions to a later vote where digital minds were allowed to participate.
* Under a direct endowment proposal, humans who received resources could give resources to digital minds. (Although, again, this would plausibly be a moral public good and we might expect it to be underfunded under the direct endowment proposal).
## One process or many processes?
Designing a governance system involves many tradeoffs—for example, direct endowment and voting both have disadvantages. Should we hedge our bets with hybrid approaches that combine multiple processes?
Here are some examples of hybrid approaches:
* 50% of space resources are distributed directly to the population at a set time, and 50% are allocated according to the outcome of one or more votes.
* Each country (or each treaty signatory) is allocated a share of space resources. These countries each pick their own processes for deciding how to use space resources.
* 10% of space resources will be distributed in 2040, 10% will be distributed in 100 years, 10% will be distributed in 1000 years, and so on.
The main advantage of hybrid approaches is that different processes can direct resources toward different kinds of values. For example, votes may be particularly likely to fund widely shared values, whereas direct resource endowment might be better suited towards ensuring that rarer preferences are funded. By adopting a hybrid approach, we ensure that a broader set of values each get at least some resources allocated to them.
Voting on how many resources to allocate with each process.
A hybrid approach could directly incorporate people’s views on which governance and allocation processes are better, and use this to decide how many resources should be allocated by each.
Here’s one such proposal.
* First, a number of governance methods are proposed. This could include all the kinds of systems discussed above (direct endowment, voting using various different voting systems, direct preference elicitation, etc). The only constraint is that each governance method must be reasonably fair in how it weighs participants’ views, rather than artificially biasing the outcome towards specific people or values.
* Second, all humans submit a view on what fraction of resources should be distributed by each governance process. The fraction of resources governed by each process is set to the **average** of everyone’s proposal.
* Finally, all humans participate in each of the processes—deciding what to vote for or what to do with resources they’ve been allocated.
One advantage of this approach is that everyone gets a say on what governance processes get used. Another advantage is that it delays the decision about which process to use. If it later turns out that there are big gains from coordinating on moral public goods, people can choose to allocate most resources using voting. Conversely, if some minority of people realize that they might get frozen out of a large-scale vote, they can cause a significant fraction of resources to be distributed via direct endowment instead.
This approach could plausibly be further improved by allowing people to freely trade their voting rights at each decision point.
In other words, people’s right to participate in each different governance process (each vote, each round of resource endowment, etc) could be represented by different shares. These shares could then be freely traded. This might, for example, allow people with more idiosyncratic moral views to get more rights to resources that they have unilateral control over. Conversely, people who care more about the outcome of large-scale votes could get more of the corresponding votes. We could even allow people to trade the right to vote on what fraction of resources should be allocated by each process.
(We find it difficult to predict the outcome of such trades, and it’s possible that it would be better to disallow some of them.)
The main disadvantage of hybrid approaches is that one process might generate outcomes that other processes would regard as substantially harmful. This can be mitigated through some of the strategies discussed in the next section.
## Mitigating downside risks
Under some governance regimes, individuals will be able to unilaterally decide how to use space resources. This has some strong upsides: people can create stuff that’s valuable to them even if others are indifferent to it, which makes it more likely that bounded views are satisfied and makes it more likely that nearly everyone is much better off. But it also has downsides: people might use their resources in ways that impose large negative externalities on others.
Some specific examples include:
* Seizing other people’s resources. Early settlers might try to use their star systems to launch probes to settle star systems earmarked for others who hadn’t yet laid physical claim on them, which could initiate a ([potentially highly negative-sum](https://mason.gmu.edu/~rhanson/filluniv.pdf)) space race. Or people might attack star systems where others had already settled.
* Uses of resources that most people find intrinsically repugnant. For example, some people might make themselves absolute rulers over societies where nearly all citizens are oppressed or enslaved. Sadists might create new beings that experience immense pain.
* Risking the destruction of other people’s resources. People might deliberately destroy the universe (e.g., by initiating false vacuum decay).
* Taking other actions with severe negative externalities on others, e.g., starting a war with space-faring alien civilizations.
We need to strike a balance. If we’re too permissive, then a few bad actors could cause enormous damage. But if we’re too restrictive, then we risk banning things that are enormously valuable to a few people just because the majority finds them a bit offputting.
Here are some potential systems for mitigating these downside risks.
**Market**. If someone doesn’t like how someone else is using their resources, they can pay them to stop.
_Strengths_ : under this proposal, an activity is only stopped when its cost to others exceeds its benefit to the actor, and they’re compensated for their loss. Thus, this proposal is less vulnerable to “tyranny of the majority” worries. Another strength is that we don’t have to collectively decide far in advance what to ban. If someone wants someone else to stop doing something, they can make an offer at any time (although, late into the future, as decision-makers spread out from Earth it might become more difficult to negotiate these trades).
_Drawbacks_ : this could create perverse incentives: some people might do things that others find objectionable so that they’ll be paid to stop. This might result in even more objectionable behavior that would have happened without that incentive. And if a single actor wants to do something a large group objects to, the group may struggle to coordinate payment because of free-rider problems.
**Voting on bans**. We could prohibit activities that a supermajority of the population (say, 80%) votes to ban.
_Strengths_ : avoids both of the drawbacks of the market solution—since there’s no transfer of resources from the people who want to ban to the people who would otherwise do the banned activity, there’s no collective action problem among the former to raise money to pay off the latter. And no one is incentivized to do objectionable things in hopes of a payout.
_Drawbacks_ : a majority that very slightly disliked an activity could ban it even if it was very important to a small minority. Since there’s no transfer of resources, there’s no check to make sure the ban is sufficiently valuable to supporters to justify the cost to opponents.
If we went with this system, we’d have to decide when and how often to vote on which activities to ban.
* If we did a single early vote, we might fail to ban activities that were later widely considered bad.
* Voters might not know of those activities at the time of the vote (e.g., because the activities are only possible due to unanticipated technology). Or voters might not be in reflective equilibrium at the time of the vote. For instance, perhaps they would later, on reflection, decide that they cared about the welfare of digital minds.
* We might be able to circumvent these issues by adopting bans with broad, non-specific language and creating an AI arbiter to interpret the ban. But the arbiter may have to do nontrivial amounts of extrapolation and there might not be a single canonical way to do that extrapolation.
* If we had a recurring vote or a single late vote, then voters might have more time to develop and carry out strategies to game the voting system before the vote(s) (see discussion [above](/supplements/space-governance-plan#early-decisions-vs-late-decisions)).
Currently, we think that at least the following activities should be banned:
* Torture of conscious beings, including digital minds and non-human animals that are sufficiently likely to be capable of having strongly negatively-valenced experiences.
* It might take non-trivial reflection by advanced ASI to figure out the precise bounds of this.
* Unilaterally destroying Earth-originating civilization, or taking actions that bear a significant risk of destroying all of Earth-originating civilization.
* Seizing other people’s property without permission.
* Making and/or carrying out threats to destroy other people’s resources, in order to extort them.
* Making and/or carrying out threats to create large amounts of disvalue by someone else’s lights in order to extort resources from them.
**Bans decided by treaty signatories**. Treaty signatories could negotiate bans before signing the treaty. This has many of the same strengths and drawbacks as bans decided by a single early vote.
**Cancelling out resources**. We could allow people to forfeit control of their resources in exchange for removing someone else’s control of their resources at a k:1 ratio (for k > 1).
_Strengths_ : as with the Market approach, by imposing a cost on the people who want the ban, it ensures that people don’t block valuable activities that they only slightly dislike. And unlike the Market approach, since there’s no compensation for the person carrying out the activity, there’s no perverse incentives to do stuff that others dislike in search of a payout.
_Drawbacks_ : as with the Market approach, a large group seeking to prevent a particular use of resources may struggle to coordinate.
## Reflection
Under most of the proposals discussed in this appendix, individuals will need to make extremely high-stakes decisions, for example:
* Should I sell galaxies allocated to me in an auction? Should I save them to pass onto my children? Should I donate them to a trust, and which trust should I donate to? Should I wait until I’ve thought more to decide, and how should I go about thinking about it?
* Should I vote for a particular activity to be permanently banned throughout the universe? Should I vote for public resources to be allocated to a particular activity?
We want people to make wise decisions that reflect their values.
Making these decisions well will likely require resolving open scientific and philosophical questions. The specific questions will vary with individual values, but the following areas are likely to be relevant to a wide range of people:
**Improved understanding of morally significant concepts**. Many people consider consciousness and preferences to be morally significant—it’s wrong to inflict suffering on a conscious being and good to help people get what they want. But our current scientific and philosophical understanding of consciousness is insufficient to answer important questions like: which minds are conscious and which of their experiences have positive or negative valence—especially as our technological capabilities expand to allow creation of minds very different from human minds in substrate or structure.
**Moral truths**. There might be objective and discoverable moral truths.
**Acausal trade**. The reachable universe might be only a small part of what exists. There are regions of our universe outside our [light cone](https://en.wikipedia.org/wiki/Light_cone), and there may even be [other universes](https://en.wikipedia.org/wiki/Multiverse). Although we cannot causally interact with beings outside our light cone, our own choices may give us some evidence about how those beings will behave. On some decision theories (e.g., [evidential decision theory](https://en.wikipedia.org/wiki/Evidential_decision_theory)), this should affect what we do. For instance, we might want to “trade” with these beings, doing stuff that matters to them in exchange for them doing stuff that matters to us. If this argument works out, our acausal impact could plausibly dwarf our causal impact.
Evidential Cooperation in Large Worlds (ECL) is one such mechanism. The multiverse contains many causally disconnected agents whose decisions may be correlated with ours: that is, when we observe ourselves making some decision, we should update about their choices too. So if we commit to pursuing multiverse-wide compromise values, this gives us evidence that huge numbers of other agents are doing the same. And this could be great—there might be substantial gains from trade if everyone pursues compromise values rather than their own.
These are just a few examples of potential crucial considerations that we might want to consider before making important decisions.
### Deciding how to reflect
People will also need to think carefully about how they want to make these decisions, including how they want to use new technologies:
**Deciding _when_ to learn information**. While it’s probably important to learn the answers to questions like those above before making object-level decisions about how to use space resources, there might be costs to learning information too early. For example, there might be positive-sum trades that can only be made _before_ the parties to the deal know some empirical results. One example of this dynamic is wagers (discussed above). More speculatively, if we end up wanting to do ECL, it might be [necessary to commit to doing ECL](https://lukasfinnveden.substack.com/p/implications-of-ecl) before we become confident whether other agents will do ECL—our decision only gives evidence about everyone else’s decision if we’re not sure what others will decide.
**Engaging with ASI-generated arguments.** ASI will likely be able to research the philosophical and scientific questions above more quickly and rigorously than humans, giving us access to rapid intellectual progress on how to best live out our values.
But ASI could also direct enormous intellectual effort toward crafting extremely persuasive arguments. And it’s not clear that the most compelling arguments will necessarily support true positions or positions we would endorse holding. If ASI is optimizing for maximally compelling arguments, then we might end up believing whatever ideology the ASI was subtly biased toward, or perhaps whatever ideologies are easiest to argue for compellingly.
One safeguard: ASI could refuse to generate superhumanly compelling arguments for claims it knows to be empirically false, in the same way that AIs today refuse to assist with creating biological weapons. But for some questions we care about, there might not be an empirical fact of the matter. For these cases, we need to think about how we want to be persuaded—what counts as legitimate moral suasion and what counts as illicit manipulation. It's not clear where the line is—is vividly describing the suffering of a particular being emotionally manipulative, or just helping you understand the true stakes of your decisions? And the line might be different for different people.
**Delegating to aligned ASI**. Another option is to delegate reflection to an aligned ASI. We will probably be able to design ASI systems that are much better suited to careful reflection than humans. These systems would be smarter, capable of understanding and integrating more complicated arguments or empirical results than any human.
Moreover, for many people, reflecting in a very careful way might be less pleasant than other ways of spending their time. An ideal reflection process might involve carefully controlling your memetic environment (if it’s possible for ASI to develop illegitimate, yet highly compelling arguments or ideologies). It might require modifying your mind to make yourself smarter, more reflective, or more free of biases that you don’t endorse. In general, it’s not clear that “optimal lifestyle for human flourishing” and “optimal lifestyle for figuring out the most idealized form of one’s values” will be the same for most people. Offloading reflection onto an ASI system that is better suited for the process might let those people get the best of both worlds.
Digital reflection has other advantages difficult to replicate on a biological substrate, including running many parallel reflection processes, reverting to earlier checkpoints, and potentially greater transparency about the trajectory that one’s values took over time. (This last is particularly important in versions of this proposal where many people’s preferences are aggregated together, since people might have incentives to strategically self-modify or misrepresent their preferences. Carrying out a digital reflection with a legible trace could let people verify that they hadn’t done so).
**Delegating to future generations**. Another option is to delegate decisions to one’s descendants or to future generations more broadly. If so, how should one raise their children? Parents will plausibly have much greater capacity to shape their children’s values—how should they exercise this power?
We’re not sure what the right answers are, and indeed the right answers may vary from person to person based on their values, but we think these are important decisions and want to help people make them carefully.
### The role of governance
It’s important for the governance structure to facilitate reflection.
* **Timing of decisions**. Delaying decisions about how to use space resources gives decision-makers more time to think. Although ASI labor may speed up the rate of scientific and philosophical research, many people will likely need years or decades to integrate those findings into their thinking. Additionally, reaching these decisions may require substantial introspection into what the decision-maker truly values, which might also be time-consuming.
* For governance structures where the timing of an important decision is scheduled in advance—e.g., a vote or a preference aggregation—this consideration favors delaying that decision (although there are considerations in the other direction, discussed [above](/supplements/space-governance-plan#early-decisions-vs-late-decisions)).
* An important advantage of direct endowment governance structures is that they allow individuals to reflect for as long as needed before making object-level decisions. If individuals were still prone to hasty decisions, we could—somewhat paternalistically—require them to wait a few decades before selling or committing their resources to a trust.
* **Subsidizing reflection.** Governments could offer all voters or resource recipients access to ASI labor to help them decide how to use those resources. Governments could also directly fund research into open scientific and philosophical questions likely to bear on decisions many people will face.
* **Avoiding perverse incentives**. Governance structures could incentivize people to warp their reflective process or interfere in others’ reflective processes. For example:
* Voting systems that offer voting rights to future generations could incentivize strategic reproduction and indoctrination.
* Voting systems (with lie detection) or direct preference aggregation systems could incentivize people to modify their own preferences strategically to gain an advantage (just as traditional voting systems incentivize strategic voting).
The answers to some of the questions above could have implications for which governance procedures are best. One important tradeoff among the governance proposals is between enabling coordination on (moral) public goods and enabling individuals to pursue their own values. ASI-powered research into these questions could shed light on:
* Whether there are important public goods that we might want to coordinate on. For example, perhaps acausal trade has large fixed costs—e.g., running lots of simulations of alien civilizations to figure out the distribution of values that trade partners have—and it would be wise to ensure that there are governance mechanisms that let us pool our resources to do that.
* The degree to which humans’ moral views converge after endorsed reflection. If people converge substantially, this favors governance proposals with coordination mechanisms that let them pool resources to pursue shared values. If there’s a lot of divergence, this favors governance proposals that allow each person to pursue their own values.
# Takes
## Mia’s takes
* My actual take is that we should wait for aligned ASI and ask it to design a governance proposal that meets the desiderata above (and perhaps more that I haven’t thought of).
* It seems very important that everyone making irreversible high-stakes decisions carefully reflect on what they care about with the help of aligned ASI advisors. To make this more likely, governments should probably fund lots of public-domain ASI research into scientific and philosophical questions that are likely to be of broad interest (like the questions above). They should also subsidize ASI compute used by people thinking about these decisions.
* Likewise, it seems important that decisions about the use of resources beyond the solar system are made for altruistic or moral reasons, not selfish ones.
* We should guarantee that all humans alive in 2040 and their descendants get enough to support them indefinitely at a very comfortable standard of living (i.e., at least the material standard of living of a multi-millionaire in 2026). This will hopefully lower the personal stakes of the decision about what to do with space and encourage people to be more high-minded when they vote.
* It is also great on common-sense ethical grounds—on a cosmic scale, it’s relatively cheap to satisfy people’s preferences, and so we should do that.
* I think coordination capacity—where we can collectively use resources to provide (moral) public goods—is potentially really important and it's not clear that we'll be able to do that with voluntary contracts. Unless we're confident that this is possible, it seems good to maintain capacity to coordinate, i.e., have a government (or a few governments) that control substantial amounts of resources (or which are empowered to levy taxes if they need to).
* On the other hand, I also want to enable people with unusual value systems to spend some resources pursuing their values.
* For these reasons, my top choice is direct preference aggregation if it’s technically feasible and not so unpleasant that many people opt out, perhaps combined with automated reflection (like what’s described here).
* If that doesn’t work, my next favorite option is a hackier solution that ensures that some resources go to individuals and some resources are decided collectively. Here’s one way of approaching this.
* At least 5% of resources are guaranteed to be directly granted to humans alive in 2040.
* By default, the remaining 95% of resources will also be directly granted to humans alive today, unless a supermajority votes that it should instead be spent on public goods.
* If highly reliable lie detection is available, governments could use that to check for strategic voting and disqualify people who were sufficiently likely to be executing these strategies.
* Governments could also have ASI systems refuse to assist with attempts to coordinate to vote strategically (in the same way that ASI systems will presumably continue to refuse to assist with other harmful requests—e.g., making bioweapons).
* We will also vote on bans and on the process to use to amend the list of bans (e.g., should we have recurring votes?). There should be a high threshold for bans, perhaps 90%.
## Thomas’s takes
* I agree that we should wait for aligned ASI and ask it to design a good governance proposal. The most important things for us to do now are (1) avoid losing control of the future, e.g. via misaligned AI takeover or a human dictator getting control, (2) distribute the power reasonably, and (3) try to do philosophy well.
* I think there’s a good chance that in practice, the best path forward will be to lock in some governance structure over the future, because deliberation within a fixed structure (like the ones discussed above) will go better than in a looser structure (like today’s government) where, e.g., more power will accrue to people who are, for example, more politically well connected.
* My tentative favorite concrete guess for what to do above is “direct endowment” with some form of amendment process with a very high threshold (e.g. 90% approval). I think it’s very similar to voting based proposals, where you “vote” by making trades with other people about what to do with your resources. The public goods concern is the strongest counterargument, and if there is a voting system discovered that addresses this concern without adding additional, even more dire consequences, then I would change to wanting that.
* All voting systems that we know of are screwed up, and I would not want them to be used for something as high stakes as deciding about what to do in the future. But I’m pretty optimistic about this changing, for two reasons:
* I think there’s a good chance that ASIs (and humans who think about it for longer) will come up with voting systems that incentivize honest reporting of preferences, and choose a good policy that aggregates the votes.
* I think that we might be able to do preference elicitation along the lines of “direct preference aggregation” above. Once you have honest preferences, the problem of designing a good voting system seems much easier.
* I care a lot about power in the future going to virtuous people. I think that it would be really bad if a huge fraction of the power in the future goes to selfish people or terminally evil people (e.g. people who want to cause massive amounts of suffering). Of course, different people have different moral views that are reasonable, and I don’t think it’s fair, for example, to claim that everyone should follow my moral views. But I do think that it’s reasonable to exclude people with sufficiently bad moral views, as agreed on from the perspective of the rest of society, from having a large amount of power over the future, for example, to exclude sociopaths from many of the preference aggregation systems above.
* I think that any attempt to have “normal governance” continue is not going to work, because our current systems will be totally exploited by actors using superintelligence to gain power. I think that we’re going to need to make massive changes to our system of government really quickly in order to maintain a reasonable ongoing distribution of power.
1.
Other important questions in post-ASI governance that we don’t address, or address only cursorily:
* Post-ASI governance on Earth or elsewhere in the solar system.
* [Meme wars](https://www.forethought.org/research/preparing-for-the-intelligence-explosion?utm_source=chatgpt.com#epistemic-disruption), if ASI are able to construct extremely compelling arguments, and members of different ideologies try to spread their views using these arguments.
* Rights for digital minds.
* Managing risk in [highly offense-dominant worlds](https://nickbostrom.com/papers/vulnerable.pdf), where a small group of people are able to cause huge amounts of destruction.
2.
The Outer Space Treaty forbids nations and (according to some interpretations) individuals from claiming celestial bodies. In this document, we’re assuming, as in the Plan A scenario, that the Outer Space Treaty will be renegotiated once extrasolar colonization becomes technologically feasible.
3.
[Toby Newberry](https://www.globalprioritiesinstitute.org/wp-content/uploads/Toby-Newberry_How-many-lives-does-the-future-hold.pdf) estimates that the affectable universe could support 10^35 biological humans or 10^44 digital minds at a time. For comparison, the number of humans who have ever lived is 10^11.
4.
One possible exception is people with strong preferences for [positional goods](https://en.wikipedia.org/wiki/Positional_good), e.g., people who care a lot about being richer than others, rather than their absolute level of wealth.
5.
For example, it might be possible to move our universe from a false vacuum state to a true vacuum state, which could [destroy all matter](https://en.wikipedia.org/wiki/False_vacuum).
6.
See the paper [Burning the Cosmic Commons](https://mason.gmu.edu/~rhanson/filluniv.pdf) for more on this threat model.
7.
Some examples of moral public goods that might be quite important to fund:
* Broadly appealing moral projects, such as building societies of flourishing biological humans, uploaded humans, or other digital minds; providing resources to future generations; or discovering and acting in accordance with objective moral truths, if any exist.
* Trading with people who intend to do something with side effects that others consider harmful to get them to modify their plans. Obviously, it’s important to approach this carefully to avoid creating perverse incentives where people do some activity that’s broadly considered harmful in search of a payout (see [below](/supplements/space-governance-plan#mitigating-downside-risks) for more discussion of reducing downside risk).
8.
Ideally, we’d also like to weigh the preferences of other stakeholders like future generations and digital minds, although we’re uncertain about whether it will be feasible to do this without opening up the political system to exploitation (see [below](/supplements/space-governance-plan#who-should-participate-in-the-process)).
9.
That is, we should either decide how future governance decisions will be made or at least pick a meta-level process for choosing how that decision will be made (e.g., the voting system should be chosen by an AI arbiter verifiably aligned to choose based on a particular set of principles)
10.
Via mutual defence coalitions, mutual insurance, or law-enforcing equilibria emerging from multi-turn interactions.
11.
Barzel (1997), _[Economic Analysis of Property Rights](https://www.cambridge.org/core/books/economic-analysis-of-property-rights/6D5E9A3AA67284FD9A12379CA3028D50)_ ; Anderson & Hill (1990), [“The Race for Property Rights”](https://www.journals.uchicago.edu/doi/abs/10.1086/467203)
12.
One way to do this might be to require that all probes sent to colonize other star systems carry a nightwatchman ASI to prevent the colony from sending out probes to seize space resources that belong to others.
13.
The point is that each person’s eventual wealth only depends on the value of whatever they inherit at equilibrium prices, and not on whether they originally inherited titles to future resources directly, or received a cash transfer of equal value at equilibrium prices. If resources R are divided equally among N people and traded at competitive prices p, each person's wealth is p · R/N. Under the auction alternative, total revenue is p · R, and equal redistribution gives each person p · R/N. [This requires](https://econpapers.repec.org/paper/mitworpap/115.htm): competitive (price-taking) behaviour, that the auction clears at competitive equilibrium prices, markets are [complete](https://en.wikipedia.org/wiki/Complete_market) and have sufficient numbers of buyers and sellers, and people have convex preferences. The equivalence between auctioning and direct endowment could break down: especially if actors are large enough to coordinate to influence prices, or if resources are indivisible.
14.
It’s true that in principle, under a direct endowment proposal, everyone could decide to voluntarily pool their galaxies into a trust and allocate resources by vote, but free-rider problems make this less likely in practice.
15.
Obviously, versions where voters from a few countries get to weigh in are much better than versions where, say, only AI company CEOs and national leaders get to weigh in. But we think all of these have some of the same problems to different degrees.
16.
Suppose there are a thousand actors, each with equal resources. One actor wants to use their resources in a way which is viewed as harmful by all the other actors. Without enforceable prohibitions, each of the majority would need to decide individually how much to spend bribing the unpopular actor to stop. No one of the 999 actors cares so much about the harms that they are willing to pay one-half of their wealth to individually prevent them; but all of the 999 are willing to pay 0.05% of their wealth to prevent them, and that would raise a large enough bribe for the unpopular actor to agree to switch to some other use of his resources. In other words, preventing the use of resources which enough people care strongly about for moral reasons is a moral public good which requires taxation to fund efficiently.
17.
For example:
* What exactly are negatively-valenced experiences? Probably our current understanding of consciousness and suffering is rather muddled, and people might create minds that have some of the features of classic examples of negatively valenced experiences but not others.
* How strongly negative does an experience have to be to fall under this ban?
* Are there any mitigating factors, e.g., the experience being necessary for a strongly positive life, the being in question consenting?
18.
This mechanism was originally introduced under the name “multiverse-wide superrationality” in this [paper](https://longtermrisk.org/multiverse-wide-cooperation-via-correlated-decision-making/).
19.
Some more crucial considerations are discussed [here](https://www.lesswrong.com/posts/EyvJvYEFzDv5kGoiG/clarifying-wisdom-foundational-topics-for-aligned-ais-to).
20.
For instance, the question of "is a particular digital mind capable of having morally relevant experiences?" hinges on both empirical properties of the digital mind but also questions about what properties are morally relevant.
21.
Probably there should be some limit on the number of descendants who could be supported by these resources because otherwise we presumably will eventually hit Malthusian limits.
sources/ai2040-pages/takeoff-supplement.md
+229
View File
@@ -0,0 +1,229 @@
---
url: https://ai-2040.com/supplements/takeoff-supplement
---
## [AI 2040](/)
≡[PDF](https://ghjyjqzwz4.ufs.sh/f/9qHa0cBclQ7sf6FVzqbp2gHFXC1IbGS5ZA7DmNBusfM4YJRP)
[Supplements](/supplements)
Plan A Basics
[FAQFAQ](/supplements/faq)[How Plan A solves our 5 biggest problemsHow Plan A solves our 5 biggest problems](/supplements/how-plan-a-solves-our-5-biggest-problems)
Plan A Policies
[Covert AI ProjectsCovert AI Projects](/supplements/covert-ai-projects)[Verification PlanVerification Plan](/supplements/verification-plan)[Transparency PlanTransparency Plan](/supplements/transparency-plan)[Capability Scaling StrategyCapability Scaling Strategy](/supplements/capability-scaling-strategy)[Security in Plan ASecurity in Plan A](/supplements/security-in-plan-a)
Analysis of Plan A Policies
[Plan A AssumptionsPlan A Assumptions](/supplements/plan-a-assumptions)[Comparing Possible PlansComparing Possible Plans](/supplements/comparing-possible-plans)[Deal DeclineDeal Decline](/supplements/deal-decline)[Takeoff SupplementTakeoff Supplement](/supplements/takeoff-supplement)
Economics
[Economics of Plan AEconomics of Plan A](/supplements/economics-of-plan-a)[Compute in Plan ACompute in Plan A](/supplements/compute-supplement)[Economic Growth ExplorerEconomic Growth Explorer](/supplements/econ-explorer)
Speculative Analysis
[Space Governance PlanSpace Governance Plan](/supplements/space-governance-plan)[AI for EpistemicsAI for Epistemics](/supplements/ai-for-epistemics)[Alignment RoadmapAlignment Roadmap](/supplements/alignment-roadmap)
[About](/about)
[![AI Futures Project](/ai-futures-logo.svg)](https://ai-futures.org/)
# Takeoff Supplement
### Brendan Halstead
This supplement explains how we forecast AI capabilities in various parts of Plan A. We modified the AI Futures Model to simulate four situations:
1. the default trajectory, if there had been no deal to slow down capabilities progress,
2. the Plan A consortium trajectory, which scales to TED-AI in a controlled manner, then halts to do alignment research,
3. a [covert project](/supplements/covert-ai-projects) defecting from the deal and trying to build AGI in secret, and
4. the trajectory after [deal dissolution](/supplements/deal-decline), where most of the post-deal compute is destroyed and the leading project resumes racing.
The rest of this document is structured accordingly. Section 1 covers the differences between the Plan A model and the December 2025 AI Futures Model: a major change to training run modeling, our updated compute forecasts, and the changes to the default parameters. In sections 2 through 4, we explain the various extensions added on to simulate the three additional situations. If you haven’t read the AI Futures Model documentation, we recommend reading it [here](https://www.aifuturesmodel.com/) first.
# Modifications to the core model
Here we describe the three ways this model differs from the December 2025 AI Futures Model. These apply across all trajectories simulated.
## Explicit modeling of the frontier training run
Like most other AI takeoff models, the AI Futures Model (December 2025 version) defined _effective training compute_ E(t)E(t)E(t) at a given time as the cumulative training compute C(t)C(t)C(t) of the frontier model (in FLOP) times the software efficiency level S(t)S(t)S(t) at that time (in present-day-FLOP per FLOP):
E(t)=C(t)⋅S(t).E(t) = C(t) \cdot S(t).E(t)=C(t)⋅S(t).
In reality, training runs take time, and the formulation above assumes that today’s software efficiency applies retroactively to the training run for the model that’s already in use. One consequence is [slightly overestimating](https://www.forethought.org/research/will-the-need-to-retrain-ai-models) how fast takeoff can proceed. But more importantly for Plan A, this model gives us no good way of dealing with situations where the _training system performance_ (in FLOP/yr) of an AI project suddenly becomes very small compared to the system that was used to train the best model available to that project (as is the case for covert projects and after deal dissolution).
In the Plan A version, we instead model effective training compute as increasing continuously as part of one long training run, at a rate determined by the _training system performance_ T(t)T(t)T(t) and the software efficiency of the known training methods:
E′(t)=T(t)⋅S(t).E^\prime(t) = T(t) \cdot S(t).E′(t)=T(t)⋅S(t).
We call this rate the _effective training system performance_ and abbreviate it as ETSP.
When all inputs grow exponentially, ETSP and effective compute also grow exponentially at the same rate, so this model behaves like the simpler one. All other equations are unchanged from the December 2025 AI Futures Model.
## Changes to the compute and labor time series
The AI Futures Model depends on exogenously-supplied forecasts of compute and labor. The Plan A version uses two sets of time series: one representing the default world if Plan A had not been implemented, and one that branches off from the default when the deal comes into effect. The modeling behind both of these is discussed in our [Compute Supplement](https://www.ai-2040.com/supplements/compute-supplement). In summary,
1. The default time series before 2029 has been updated to be slightly more bullish compared to that in the AI Futures Model
2. The Plan A time series from 2030 to 2035 reflects the controlled-but-massive compute buildout enabled by robotics, then slows down from 2035 to 2040.
One limitation is that the human labor projections are not very realistic; we did not put much effort into these because human labor plays a very small role in the model dynamics after AI R&D is automated.
## Changes to the default parameters
The default parameters governing timelines and takeoff in the scenario were set to reflect scenario first-author Thomas Larsen’s views. However, since capabilities forecasting was not the main focus of AI 2040, they do not reflect as much consideration as do the values used in the AI Futures Model, and were sometimes adjusted ad-hoc as the scenario narrative and modeling were being written concurrently. Nevertheless, they deviate only slightly from Daniel Kokotajlo’s median values as of April 2026, and all authors find the default trajectory entirely plausible. Below we include a table describing each parameter that was changed from Daniel’s median.
**Parameter**| **Change**| **Rationale**
---|---|---
Gap years| 1.1 2025-effective-flop-growth| Set to achieve the AC milestone at Jan 2030.
Time horizon required for AC (before gap)| 10 work years
Median-to-top taste multiplier| 5.28x| Set to reach ASI roughly one year after AC, while leaving ~3 OOMs of effective compute between AC and TED-AI.
AI research taste slope| 3.5 SDs per 2025-effective-flop-growth
Median-to-top-human jumps above SAR needed to reach TED-AI| 1.5
# Modeling the consortium trajectory
Unlike all other actors we model, the consortium is _not_ racing to build better AIs as fast as they possibly can. The specific strategies are discussed in more detail in the Capabilities Scaling Strategy supplement, but at a high level, the strategy is to scale smoothly to Top-Expert-Dominating AI (TED-AI), then pause to do alignment research before eventually handing over control to AIs.
Here, we consider strategies that vary along two axes:
1. How far into the deal consortium aims to reach Top-Expert-Dominating AI (TED-AI), and
2. How much of a safety tax the consortium aims to be paying when TED-AI is reached.
The safety tax is the difference between the capability level of the actual consortium AI and the capability level of the best AI the consortium _could have trained_ , had they used all their training compute and the best-known algorithms for capabilities training. One example of “paying a safety tax” would be discovering neuralese training, finding that it would be much more efficient, but deciding not to use it to make control easier. We expect that in reality the consortium would want to pay some safety tax.
Plan A spends more time at higher capability levels than a uniformly slowed intelligence explosionPlan A — capability-shaped upliftsAutomated Coder3× realized upliftTop-Expert-Dominating AI~40× realized uplift2029: two ways to slow down2040: handoffearly 2035mid 2030late 2034Plan Aslowdown deal — controlled takeoffUniform slowdownuniformly slower uncontrolled intelligence explosion20262028203020322034203620382040
The rough shape of the consortium’s capabilities trajectory: a brief halt, then smooth scaling, then a long near-pause before handoff.
The consortium trajectory is simulated according to slightly different rules during each of the following stages.
**Stage 0: AI Training Pause**
This phase starts some number of years before AC would have happened on the default path. (In the scenario, about 9 months before). Though effective compute is forced to stay flat (training is paused!), training system performance can still increase as the leading company within the consortium ends up with more compute at the end of the period.
AI R&D is _also_ essentially paused during Phase 0. However, we model the software efficiency of the leading actor within the consortium as growing by a set number of OOMs during this period. This is because of Total Research Transparency: by the time scaling resumes at the end of Phase 0, essentially company IP is public. We expect the companies to have made non-overlapping software discoveries prior to the deal; hence pooling the intellectual property will increase software efficiency at the frontier.
**[Stage 1: Scale to max-controllable AI](https://www.ai-2040.com/supplements/capability-scaling-strategy#stage-1-scale-to-max-controllable-ai)**
During this phase, effective compute increases with effective training system performance via the usual E′=T⋅SE^\prime = T \cdot SE′=T⋅S. TSP grows via the exogenously-supplied Plan A time series, and software efficiency growth is carefully controlled in order to reach TED-AI at a particular time. We assume a mandatory minimum amount of software progress is required per OOM of TSP increase.
To keep track of the safety tax paid at each point, we actually simulate two different trajectories:
* One represents the legal project’s _realizable_ capabilities. This trajectory aims to maintain the (constant) software progress rate that achieves TED-AI _plus_ the desired safety tax amount at the target time.
* One represents the legal project’s _realized_ capabilities. This trajectory aims to maintain the (constant) software progress rate that reaches TED-AI at the same time that _realizable_ capabilities reach TED-AI+tax.
At each timestep the trajectories moderate their research effort from the theoretical maximum, choosing instead the value which brings them closest to the pre-selected software progress path.
Upper and lower limits imposed on software progress
If the chosen scaling duration and takeoff parameters demand faster software progress than would be naturally possible at a given point, the research effort may not be moderated at all. Often, once AI capability level increases enough, the consortium is able to catch up to the desired software path. Occasionally, the consortium may not be capable of reaching TED-AI by the desired time, e.g. in worlds where reaching TED-AI from the pause level would naturally have taken ten years but where the consortium attempts to reach it in just one.
Conversely, a sufficiently long scaling period or a sufficiently narrow effective compute gap from the pause level to TED-AI can mean that hardware scaling alone (plus the mandatory minimum software growth rate) would reach TED-AI before the target year. In this case, the consortium simply reaches TED-AI early.
**[Stage 2: Slow down nearly to a halt, then eventually hand off to AIs](https://www.ai-2040.com/supplements/capability-scaling-strategy#stage-2-slow-down-nearly-to-a-halt-then-eventually-hand-off-to-ais)**
For simplicity, we model this stage as a literal halt for the consortium’s _realized_ capabilities. In reality, we expect them to slow down more smoothly, then continue increasing very slowly as control techniques improve and the threshold for “max-controllable AI” inches upward.
The consortium’s _realizable_ capabilities continue increasing however. TSP growth slows down but continues. As for software efficiency growth, we specify a constant rate of software progress in OOMs per year to represent the capabilities externalities incurred by the huge amounts of alignment research that happen during stage 2. (As an example, dramatic advances in mechanistic interpretability, while promising for alignment, could plausibly also enable much more efficient training methods.)
Since the _realized_ capabilities effectively halt while the _realizable_ capabilities continue growing, the safety tax paid increases dramatically during this period.
In reality, the consortium will choose to hand off to AIs at some point (in our scenario, it happens roughly five years into Stage 2). We do not currently model handoff explicitly however, because none of our analyses depend significantly on the dynamics after handoff. This is why the consortium capabilities line remains flat indefinitely in the explorer, rather than eventually jumping up as we would expect after handoff.
# Modeling covert projects
At the moment that AI training is paused (Stage 0), the covert project is assumed to start with the following resources:
* A total R&D compute budget equal to some fraction of the leading company’s R&D compute. We assume the covert project allocates between training, experiments, and internal inference in the same proportions as the leading lab did.
* An engineering labor force equal to that same fraction of the leading lab’s labor force,
* AI capabilities, software efficiency, and research stock matching those of the pre-deal leading company.
We also assume that the covert project’s supply of compute and labor stays constant over time. Finally, we assume that the covert project cannot begin training or R&D for some period during which covert datacenters are being constructed.
With only these factors considered, our modeling indicates that covert projects would probably progress quite slowly. [In the median case, each OOM of R&D compute reduction (compared to the leading lab at the start of the deal) would cause the covert project to take over 6x as long to reach TED-AI than would the leading lab.](https://www.lesswrong.com/posts/7jcPg79p3kD5ir3CL/how-much-slower-does-takeoff-go-with-10-less-compute)
However, we also assume that the covert project is uplifted to some degree by consortium AI progress. The primary factor, and the only one we model explicitly, is the leakage of software improvements from the consortium to the covert project. To account for this, we include a parameter for the fraction of the consortium’s software progress rate that gets added on to the covert project’s indigenous software progress rate. We conservatively assume covert projects will make full use of any algorithms discovered by the consortium, even those deemed potentially dangerous, so we take the _realizable_ software efficiency growth rate as the quantity to steal from rather than the _realized_ software efficiency growth rate.
Due to software proliferation, the consortium’s scaling strategy is extremely important to the covert project’s ability to catch up. The less software progress the consortium needs to make to reach TED-AI at the desired time, the less the covert project is uplifted. As a result, accepting a _longer_ scaling duration that requires less software progress _increases_ the consortium’s ETSP lead at TED-AI, and typically also _increases_ the time between the consortium reaching TED-AI and the covert project reaching TED-AI.
# Modeling deal dissolution
The modeling of [deal dissolution](/supplements/deal-decline) is relatively simple.
* The effectiveness of [mutually assured compute destruction](/supplements/deal-decline#mutually-assured-compute-destruction-destroying-compute-fabs-and-robots) is represented by the fraction of world compute available to the new leading project at the time of dissolution.
* We also include a parameter for the growth rate of R&D compute after the deal dissolves, representing the output of any non-destroyed or newly-constructed fabs.
* As with covert projects, we technically assume that human R&D labor varies in proportion to compute. Though this might be unrealistic, the effect of human labor is essentially negligible after AC.
* The post-dissolution leading project starts with the frontier model weights (from the _realized_ consortium trajectory) and all discovered software improvements (from the _realizable_ consortium trajectory) from before the deal dissolved.
We assume that the leading project after a deal dissolves proceeds at full speed, as in the default trajectory.
1.
Specifically, they should scale to something like “max-controllable AI”, but in AI 2040 we operationalize this as TED-AI.
2.
Technically in the scenario, the pause lasts from March 2029 to September 2029, and “pilot” experiments are able to proceed during the last part of the year before the first major training run is approved in January 2030.
3.
This assumption may not represent the optimal strategy for a covert project. We did some rough experiments investigating various compute allocation strategies in an earlier version of this model, and found that they made a negligible difference to the speed of the covert project trajectories (except in extreme cases — for example, allocating zero compute to training will halt effective compute growth entirely. Similarly, allocating zero compute to experiments would halt software efficiency growth entirely.)
4.
This, too, is a somewhat unprincipled assumption (compared to, e.g., carrying out a dedicated estimate of covert R&D labor and including that as a separate parameter.) However, the precise reduction in the engineering headcount matters very little when engineering is nearly fully automated. We are fairly confident that improving upon this simplification does not affect the results in any significant way, and have checked this empirically at various points. Due to time constraints, we do not currently include detailed results here.
5.
This is one of the ways our model is generous to covert projects; it is similar to assuming that whoever is operating the covert project has access to the frontier model weights, datasets, and full stock of algorithmic knowledge from the leading lab at the time of the deal’s signing. As an improvement, we could add a model parameter representing the number of “months behind” the covert project starts at. Unfortunately, we do not include this as of the Plan A release due to time constraints.
6.
This is slightly generous but on the whole seems likely to be accurate. The most catastrophic way this could fail is if some amount of the legal compute produced during Plan A could be smuggled to a covert project (or used illegally from within the R&D clusters). This would be catastrophic because world compute increases by several OOMs during Plan A. A milder failure mode would involve the covert project making use of the growing stock of consumer hardware during the deal (as makeshift AI R&D compute). Finally, a covert project could attempt to build and operate its own independent, unmonitored semiconductor supply chain during the deal. Our strategy for preventing all of these things from happening is described in the [Verification Plan](https://www.ai-2040.com/supplements/verification-plan).
7.
We do permit them to accumulate stolen software during the construction period, described below.
8.
We say “over” because the stated result compares the covert project to a trajectory whose compute level was _frozen_ at the branch point, where in reality the leading lab benefits from growing R&D compute during takeoff.
9.
Other factors include distillation and weight theft. In this model, we assume these can be prevented.
sources/ai2040-pages/transparency-plan.md
+332
View File
@@ -0,0 +1,332 @@
---
url: https://ai-2040.com/supplements/transparency-plan
---
## [AI 2040](/)
≡[PDF](https://ghjyjqzwz4.ufs.sh/f/9qHa0cBclQ7sf6FVzqbp2gHFXC1IbGS5ZA7DmNBusfM4YJRP)
[Supplements](/supplements)
Plan A Basics
[FAQFAQ](/supplements/faq)[How Plan A solves our 5 biggest problemsHow Plan A solves our 5 biggest problems](/supplements/how-plan-a-solves-our-5-biggest-problems)
Plan A Policies
[Covert AI ProjectsCovert AI Projects](/supplements/covert-ai-projects)[Verification PlanVerification Plan](/supplements/verification-plan)[Transparency PlanTransparency Plan](/supplements/transparency-plan)[Capability Scaling StrategyCapability Scaling Strategy](/supplements/capability-scaling-strategy)[Security in Plan ASecurity in Plan A](/supplements/security-in-plan-a)
Analysis of Plan A Policies
[Plan A AssumptionsPlan A Assumptions](/supplements/plan-a-assumptions)[Comparing Possible PlansComparing Possible Plans](/supplements/comparing-possible-plans)[Deal DeclineDeal Decline](/supplements/deal-decline)[Takeoff SupplementTakeoff Supplement](/supplements/takeoff-supplement)
Economics
[Economics of Plan AEconomics of Plan A](/supplements/economics-of-plan-a)[Compute in Plan ACompute in Plan A](/supplements/compute-supplement)[Economic Growth ExplorerEconomic Growth Explorer](/supplements/econ-explorer)
Speculative Analysis
[Space Governance PlanSpace Governance Plan](/supplements/space-governance-plan)[AI for EpistemicsAI for Epistemics](/supplements/ai-for-epistemics)[Alignment RoadmapAlignment Roadmap](/supplements/alignment-roadmap)
[About](/about)
[![AI Futures Project](/ai-futures-logo.svg)](https://ai-futures.org/)
# Transparency Plan
### Thomas Larsen
AGI projects face key decisions around _transparency_ : who gets to see algorithmic secrets, who gets real time access to the datacenters sufficient to be confident in what they are doing, who gets to be aware of ongoing experiments and training runs, and with what latency. This document outlines our current thinking on transparency in the context of Plan A.
First, we discuss baseline transparency desiderata—aspects of transparency proposals that are core to Plan A. A fundamental component of Plan A is a trustless US/China deal to ensure that AI development is safe, which necessitates a minimal amount of US/China transparency, sufficient to verify that the other is following the agreement. ([more](/supplements/transparency-plan#baseline-transparency-desiderata)).
Second, we give an overview of possible transparency regimes which can satisfy these desiderata. ([more](/supplements/transparency-plan#four-transparency-proposals))
Third, we give an in depth explanation of our favorite proposal, “total research transparency”. In this proposal, almost all algorithmic secrets, experiments, and training runs would be visible to the public immediately. ([more](/supplements/transparency-plan#total-research-transparency)).
Finally, we give an analysis of the tradeoffs between the different proposals outlined, and the reasons why transparency is beneficial. The main reason we recommend Total Research Transparency is because it would improve government and corporate decisionmaking during Plan A; which is our current most concerning failure mode. The second-most important reason is that it makes it much harder for corporations and governments to abuse their power. ([more](/supplements/transparency-plan#when-is-each-proposal-desirable)).
# Baseline Transparency Desiderata
Any transparency proposal in Plan A should be consistent with the following desiderata:
1. **Trustless Verification.** The US and China need to be able to verify slowdown agreements without relying on trust. This means that specifically the US and Chinese governments need visibility into how compute is being used.
2. **Model weights security robust against nation-state adversaries.** We’d like to prevent model weights from leaking because if the model weights were to leak it would (1) greatly accelerate covert projects and (2) increase misuse risk. Neither of these are necessarily fatal, but we’d very much like to avoid them. Once we get to AIs that are as or more generally capable than top human experts, model weight security will probably be necessary.
3. **Ongoing consumer access to frontier AI.** Direct access to the AI models is perhaps the most important form of transparency. Seeing the results of evaluations and high level descriptions of model behavior are much less informative than directly using the model.
# Four Transparency Proposals
We have four viable candidates for transparency, each of which meet the above desiderata. Proposals #1-3 are further consistent with Plan A, while Proposal #4 only works for Plan S.
**Proposal #1: Total Research Transparency.** Nearly all AI research is fully transparent to the public, including AI algorithms, code, and documentation, for all frontier AI projects. AI model weights, significant fractions of the training data, and a small amount of other sensitive information is prevented from leaving the datacenters.
**Proposal #2: Filtered Transparency.** Nearly all AI research is done within a physical security boundary containing both the researchers and the compute. Within the security boundary, US and Chinese auditors have sufficient access to verify deals. Three things are allowed to leave the datacenter: (1) (partially redacted) reports approved by both sides about what is going on inside, (2) model weights that get sent to the inference datacenters, and (3) researchers themselves. One can vary the level of redaction: with minimal redaction, this proposal approaches Proposal #1, and with maximal redaction, this proposal is similar to Proposal #3.
**Proposal #3: Algorithmic Security.** It may be necessary to prevent the exfiltration of algorithmic secrets. This requires tightening the security boundary above: if people are regularly entering and leaving the AI datacenters, then they will almost certainly report back most of the algorithms to their governments. Similarly, filtered reports provide a channel for steganographically encoded versions of the model weights to be exfiltrated. Under this proposal, all frontier AI research is done in a secure datacenter with researchers and auditors located onsite and prevented from leaving or communicating any information out. Model weights are allowed to move from the R&D datacenters to the inference datacenters. This involves something like “handing off trust” to the auditors inside the datacenters, because so limited information is allowed to leave the datacenter that external actors cannot exert meaningful oversight internally.
**Proposal #4: Shut it all down.** Specifically, ban compute-intensive AI R&D. Under this proposal, there is no large-scale AI R&D, so there’s no need for AI R&D transparency. This could be enforced through compute declarations and physical inspections (but doesn’t require sharing digital access to the clusters, i.e. algorithms). This is similar to the MIRI proposal, and also what occurred for about 6 months during 2029 in the AI 2040 scenario.
This table summarizes the rough tradeoffs between them:
| **Proposal #1: Total Research Transparency**| **Proposal #2: Filtered Transparency**| **Proposal #3: Algorithmic Security**| **Proposal #4: Shut it all down**
---|---|---|---|---
**SL5 algorithmic secrets security**| No| Partial, depends on redactions| Maybe| N/A, no new algos discovered
**Allows new training runs**| Yes| Yes| Yes| No
**How informed are public discussions of safety cases?******| Very| Depends on redactions.| No| N/A, no new safety cases needed
In Plan A, we currently recommend Total Research Transparency for many reasons, chief of which is that the likelihood of reasonable analyses of safety cases is much higher if external researchers and the general public are able to critically analyze them. The main drawback relative to Proposals 2&3 is that there is less algorithmic security, and so algorithms will probably leak to covert projects. However, we think that SL5 algorithmic secrets security is incredibly difficult. Therefore, our main mitigation for algorithms leaking is to prevent new algorithmic secrets from being developed in the first place.
# Total Research Transparency
In this section we’ll discuss a specific possible implementation of Total Research Transparency, which will correspond to the implementation done in Plan A.
R&D DatacentersInference DatacentersAuditorsR&D ComputeOpaque InternalDatabase(Model weights,Activations,RL Data,Hyperparams)Internal Silocodeoutputs(screened to preventOpaque info from leaking)Variable latencyWeightsHigh RiskOutputsRequests for infoMisuse-InfohazardousTrajectoriesRequests for codeand other information1 MB/sec globalbandwidth capAll outputsAll inputsTransparentInternal Database(Code, AI COT,prompts, etc)InferenceComputeInference onlyrecomputationOutputTokensInputTokensVisible toPublic DatabaseIncludes all outputs from R&D Datacenters,outputs from models used for AI R&D, andall inputs to the R&D datacentersThe PublicAI & HumanAI ResearchersAIs used for AI R&Dhappen in specialinference datacenterswith additional transparencyrequirements (e.g. no ZDR)and additional controlscaffolding
The arrows in the diagram show the flow of information between humans, R&D datacenters, and Inference datacenters.
Total Research Transparency \- ai-2040.com
Under this proposal, there are two types of datacenters:
* **Inference Datacenters** only provide inference made by whitelisted models. These datacenters host consumer chatbots, almost all agent workloads, robot foundation model inference, etc. Arbitrary requests are allowed to be sent into the datacenter, but only verified AI model outputs are allowed to come out, so that the model weights are secure.
* **R &D Datacenters** allow training runs and experiments—everything needed to produce a new set of model weights. They are “transparent datacenters,” as described above. These datacenters are much higher risk than inference datacenters, because they could be used to do an intelligence explosion.
Information from the AI R&D Datacenters is organized as follows:
* An **Opaque Internal Database** contains all information that shouldn’t be public. This includes model weights, activations, some datasets, and some hyperparameters. No human (internal or external) is able to directly view the information in this database.
* A **Transparent Internal Database** contains all other information, including all code, AI chains of thought, prompts, etc. However, it’s not directly accessible to the public or AI researchers who aren’t physically located within the security boundary of the AI R&D datacenter.
* A **Transparent Public Database** is accessible to everyone. It is a record of all information going in or out of the R&D datacenter. It logs all activities of AI researchers interacting with the R&D clusters, as well as any additional queries by members of the public that wish to see some piece of information. For example, they might want to task an AI to look through the internal database and write a short report about the state of some particular field of research.
Anything that’s in the transparent internal database could end up in the transparent public database if anyone requests that information. However, what stops everything from leaving to the public database are upload limits that cap outgoing information to ~1 megabyte / second, as a layer of defense for the opaque information being exfiltrated (i.e., model weight exfiltration in particular). There is an ongoing auction for the outgoing bandwidth to allocate access, and all outgoing bandwidth is also copied to a freely accessible database on the internet. In practice, this means that almost all algorithms are immediately revealed to the public.
The R&D datacenters are still owned and operated by tech companies, like in today’s world. In fact each R&D datacenter is really a campus of datacenters, inside which several tech companies‘ datacenters are located. The US and Chinese governments, with their respective auditors and monitoring devices, ensure that each tech company’s datacenters on the campus create logs of their activities that go to the internal opaque database and the internal transparent database.
Anyone, including members of the public, can purchase compute inside the R&D datacenters and have the same level of access to frontier AI development as frontier AI company researchers. Members of the public can, for example, run experiments on intermediate model checkpoints, see loss curves during training, or ask an AI to run a desired analysis for them.
There are three risk tiers of inference:
* **Extreme Risk:** AIs used for AI R&D need to have direct access to R&D compute, and so they run on the R&D datacenters instead of the inference datacenters. This also allows additional security—these AIs are subject to carefully designed control protocols.
* **High Risk:** AIs used for construction in dangerous domains (e.g. constructing robots, constructing GPUs or semiconductor manufacturing equipment) OR AIs used for research in dangerous domains (e.g. research relevant to engineered pandemics). These AIs live on the inference datacenters and must log the trajectories produced by AIs operating in high risk domains.
* **Lower Risk:** AIs used for anything else. This includes most commercial use (e.g. using AIs as a chatbot), most corporate use (e.g. using AIs for medicine/law/architecture/etc). These AIs are allowed to run with zero-data-retention (ZDR), allowing for maximum privacy. Just like standard practice today, even AIs running with ZDR are still subject to refusals and anti-misuse classifiers. However, only AIs behind the frontier are allowed to run with true ZDR. This is discussed in more detail in our [verification supplement](https://www.ai-2040.com/supplements/verification-plan#zero-data-retention-zdr-inference).
High Risk and Lower Risk inference operates on the inference clusters, while Extreme risk inference occurs within the AI R&D clusters on a compute cluster that’s verified to be inference only.
All of the high risk trajectories and some of the non-ZDR lower risk trajectories are physically transported (under guard!) between the inference datacenters and the R&D datacenters in order to give safety researchers access.
# Filtered Transparency
R&D DatacentersInference DatacentersAuditorsAI & HumanAI ResearchersGovernment-onlyreportsPublic DatabaseIncludes reports from inside the R&D datacentersVisible toThe PublicWeightsInferenceComputeInference onlyrecomputationOutputTokensInputTokens
Filtered Transparency \- ai-2040.com
In filtered transparency, AI researchers are physically located within a security boundary, and we attempt to prevent information from leaking across this boundary. Then there are also auditors inside the R&D datacenters that have sufficient access to verify and make judgement calls about safety.
The three ways that information crosses out of the boundary are:
1. **Model Weights.** Just like in total research transparency, model weights need to pass from the R&D datacenters to the inference datacenters, and this channel needs to be high bandwidth because model weights are so large.
2. **Auditor Reports.** The auditors write up reports about progress inside the R&D datacenters, which include things like current capabilities, safety relevant information about the current architectures. Redacted versions of these reports could be made public.
3. **Humans leaving the security boundary.** Human researchers periodically enter and leave the security boundary. This is the main way that we expect algorithms to be exfiltrated from the datacenters.
There’s no restriction on information crossing into the security boundary.
Like Total Research Transparency, there are R&D datacenters and inference datacenters. This allows for one of the most important forms of transparency: widespread access to frontier AI models. The main downside of filtered transparency relative to total research transparency is that it does not allow for fully open access and scientific scrutiny of the safety cases related to AI R&D. Instead we have to hope that the relatively small group of researchers inside the security barrier will be able to identify all the flawed assumptions of each safety case, make appropriate assessments, make sufficient research progress, etc.
Algorithmic security is similar to an extreme form of filtered transparency. The central idea is that algorithmic progress is allowed to be made in the project, but doesn’t leak outside. This is extremely difficult because algorithmic progress is often memorizable.
Therefore, the main difference is that for the “algorithmic security” proposal, it is necessary to prevent humans from crossing the security boundary. This enables us, in principle, to secure algorithmic insights.
# When is each proposal desirable?
The main axis for deciding between transparency proposals for R&D is whether we are more concerned about possible covert projects, or poor regulation. Increasing transparency will result in more algorithms leaking, boosting covert projects, but improving public oversight of AI projects and increasing the expected quality of AI regulation.
More concerned aboutpoor regulationMore concerned aboutcovert projectsProposal #1:Radical TransparencyProposal #2:Filtered TransparencyProposal #3:Algorithmic SecurityTransparency proposals spectrum \- ai-2040.com
Overall, we think that poor regulation is more concerning than covert projects, and so we recommend Total Research Transparency.
Proposal #4 does not allow for the development of new AI models, i.e. it only works as part of Plan S as opposed to Plan A. The decision about Plan A vs Plan S will mostly be made on other factors, which are discussed more in the [Plan S branch](https://www.ai-2040.com/?choices=plan-s-root).
## What fraction of algorithms will leak under each proposal?
The optimal policy on this spectrum will depend on quantitative factors around how much algorithmic progress would leak under each proposal. This will depend on what fraction of algorithmic progress is of the type that would be transparent (e.g. architectural improvements) vs what fraction of algorithmic progress is large dataset improvements. Overall, we make the current (very rough) best guess estimates of the fraction that will leak under each proposal:
**Proposal**| **Algo Fraction leaked**
---|---
P1 Radical| ~67%
P2 Filtered| ~47%
P3 Algorithmic Security| ~14%
You can see the reasoning behind this estimate [here](https://docs.google.com/spreadsheets/d/1zKOxldIPXrz6RypC8Zr50rTpF10yGz4w0UZhNxk78XE/edit?gid=0#gid=0). We would be excited about security experts (which we are not) making their own forecasts; especially forecasts that are more granular (e.g., that include over what time period the algorithms leak).
## Total Research Transparency Upsides
This section talks about the benefits of transparency for reducing loss of control risks and concentration of power risks. It’s basically an explanation of this diagram:
Total ResearchTransparencyBroader technicaldebateMore alignmentresearchersBetter incentivesVerifiableagreementsNo companypulls aheadNo secretloyaltiesStrongeroversightBetter alignmentunderstandingLimits onunsafe speedPrevents Loss ofControlPreventsConcentration ofPowerTime passesWorld hasmore timeOur key policy interventionsOur main goalsIntermediate outcomes / factorsArrows represent causation~ Notice the feedback loop here! ~
TotalResearchTransparencyTechnicalconversationno longerdominated byAI companyemployeesVastly morealignmentresearchers withaccess to latestmodels andevidenceIncentivesnot soperverse infrontier AIcompaniesAgreements topay safetytaxes and bandangerouspractices canbe verifiedNo Company PullsAhead; OtherCompanies CatchUp, IncludingCompanies inOther CountriesNo SecretLoyalties /HiddenAgendasOversight of AI companiesby governments is moreeffective; oversight of execbranch by judiciary,congress, the public, etc. ismore effective tooBetterunderstanding ofAI alignment, AIrisks, etc. Alsoshifts the burden ofproof.Prevents Lossof ControlLimits onSpeed of AlgoProgress +UnsafePracticesBannedPreventsConcentrationof PowerTime passes,diffusionoccursWorld Has More Timeto Prepare, React,Adjust to HistoricallyFast AI-DrivenChangesOur key policy interventionsOur main goalsIntermediate outcomes / factorsArrows represent causation~ Notice the feedback loop here! ~
In the diagram, the two most important policy interventions of Plan A are described in blue. The two most important goals—preventing loss of control, and preventing extreme concentration of power—are described in green. Arrows represent causation.
This diagram is in some sense a summary of Plan A, except it’s biased towards depicting the benefits of transparency in particular.
We won’t talk through the entire diagram here. Instead we’ll talk through the seven depicted benefits of Total Research Transparency, plus an additional eighth benefit. For each one, we’ll talk about how weaker kinds of transparency would get similar (but weaker) benefits.
1. **“Technical conversation no longer dominated by AI company employees.”** Transparency will improve epistemics about AI risks, and therefore will make regulations more likely to be implemented well.
The US and China need to agree on AI regulation sufficient to dramatically reduce AI takeover risk. This regulation will involve things like deciding on a capabilities schedule (which capabilities are too dangerous, when is it worthwhile to accept some risk?), deciding on whether particular algorithms should be allowed vs. banned (e.g. new architectures that are more sample-efficient yet also more interpretable), and which research directions have worthwhile capabilities/alignment tradeoffs (e.g. whether research into new scalable oversight techniques is likely enough to be a useful component of a safety case).
Under status quo transparency, this would result in a small group of government officials needing to make these decisions. The only technical experts they would be able to consult that have access to the latest information would be at the AI company(ies) they are trying to regulate, because they wouldn’t be allowed to share information with the other technical experts. But the AI company employees have a massive conflict of interest here and so cannot be trusted to give unbiased opinions.
It helps to give the information to a broader group of experts (e.g. academics and rival AI companies), without publishing it. Now, AI companies can be used to audit each other. This has much better incentives because AI companies have an incentive to point out if their competitor is doing something reckless, dangerous, or illegal. However, this relies on governments selecting the right groups of experts, and relies on AI companies to report on AI companies, each of whom will still be biased against industry-wide regulation.
It helps even more to make the relevant information public, because now we no longer rely on AI companies or government approved AI experts to process the information on their own and form good opinions about the necessary regulations. This allows anyone to write up high quality risk analyses about any given model, algorithm, or practice. Instead of the discussion of sufficient regulation happening in a few small offices with government and AI company employees, it can happen in public, making it dramatically more likely that mistakes will be caught and fixed. The government still needs to make good decisions, but they are in a better position to do so because they can peruse an ocean of research and analyses produced by different groups that are all able to see the relevant evidence.
Furthermore, transparency makes it much more likely that the judicial system will be able to exert reasonable oversight on AI development, which is a core lever for enforcing reasonable safeguards.
2. **“Huge increase in the number of alignment researchers with access to the latest models and evidence.”** More transparency means more researchers can participate in cutting-edge research, which means progress is made faster.
In 2026, only a small fraction of people with the expertise to do AI alignment research have access to the latest models and the latest evidence. The rest are outside the frontier AI companies, either in laggard companies, or in nonprofits, or in academia. Total Research Transparency equalizes access, so that anyone can buy some compute and run some experiments on the latest models, and see the results of the experiments others have run. This straightforwardly increases the effective amount of human labor directed at making alignment research progress.
More moderate forms of transparency would help in the same way, but to a lesser extent.
3. **“Incentives not so perverse”** Total Research Transparency removes the incentive for AI companies to race to better AI algorithms.
In Plan A, the US and China directly regulate AI companies to limit their rate of algorithmic progress. But the transparency also serves as a second layer of defense by changing the incentives around algorithmic progress.
Under the status quo, AI companies' main moat is their algorithmic advantages over competitors. Therefore, AI companies have a huge incentive to race to find better AI algorithms. When all AI algorithms are shared, this moat evaporates, and so too does this incentive. The race dynamic is a central upstream factor leading to high AI takeover risk, which Total Research Transparency directly undermines. More moderate kinds of transparency would not achieve this to nearly the same extent.
It helps to consider an example. In 2026, [many AI researchers agree](https://arxiv.org/abs/2507.11473) that chain of thought monitorability is helpful for safety and that it would be a shame to lose it. However, each company is individually incentivised to have research programs into new paradigms that would destroy or damage chain of thought monitorability (call this “[neuralese](https://ai-2027.com/#narrative-2027-03-31)”). After all, if they don’t do it, someone else will.
Now imagine that something like Plan A happens, with Total Research Transparency. AI companies can sleep more easily without doing any investment into neuralese research, because they know that if anyone else was making a serious attempt, they’d immediately see it. What would be the point of doing it anyway? You’d take a hit to monitorability for a competitive advantage that would be immediately shared with all your competitors.
Now imagine that the regulation is not based on Total Research Transparency but instead on some more moderate kind of transparency—e.g. filtered transparency based on government audits. Now each company has an incentive to research neuralese again, and moreover, if they’ve started researching neuralese they may even have an incentive to convince the government that it’s not as dangerous as it sounds, because now they think maybe this really will be their competitive advantage.
4. **“Agreements to pay safety taxes and ban dangerous practices can be verified.”** Total Research Transparency makes AI agreements more robust.
Making the logs of most AI development public dramatically increases the aggregate brainpower that’s looking over the logs to ensure they are in compliance with the agreed upon rules. From the perspective of a company or country trying to defect from the agreement (e.g. by secretly using compute in the R&D clusters to do a massive illegal training run, or a training run that they argue is allowed but is really pushing the boundaries and is arguably illegal), it becomes massively more difficult for them to do this because it will be hard for them to predict what sorts of analyses the public will do on the logs. The alternative, trying to outwit a small group of overstretched government auditors (in proposals 2&3), will probably be much easier. And of course, without even some basic level of transparency that is already much stronger than what we have in 2026 (US and Chinese governments get to see what’s going on in each other’s datacenters) then they have hope that their spy and cyber operations are good enough to substitute, or else trust each other not to cheat.
5. **“No company pulls ahead, others catch up, including companies in other countries.”** Having multiple companies spread across multiple countries is great for preventing extreme concentrations of power.
This one is pretty straightforward. In 2026, power comes from a variety of sources—knowledge is power, money is power, military strength is power, etc. In the future, as AIs become superhuman at everything and proliferate around the world, power will increasingly come from controlling the best AIs. (It also comes from controlling the greatest number of AIs, so to speak, but because AIs can be copied, we think that generally speaking whoever controls the best AIs will soon end up controlling a very large number of them as well.)
The kind of broad access and control offered by AI companies in 2026 is pretty weak from a power-concentration perspective. Yes, individual users can purchase subscriptions to ChatGPT or Claude and so forth. And under normal circumstances, ChatGPT or Claude will follow your orders. However:
* You can’t use them for everything—they’ll refuse to help you with some things for example, or possibly even [quietly underperform](https://www.lesswrong.com/posts/sSyLyc3KDQzboQGWS/thoughts-on-claude-fable-s-silent-safeguards), and
* There’s nothing stopping the AI companies from inserting [secret loyalties or hidden agendas](https://www.formationresearch.com/secret-loyalties-whitepaper.pdf) into their AIs, like [xAI](https://www.cnbc.com/2025/07/11/grok-4-appears-to-reference-musks-views-when-answering-questions-.html) and [Google](https://nypost.com/2024/02/22/business/google-pauses-absurdly-woke-gemini-ai-chatbots-image-tool-after-backlash-over-historically-inaccurate-pictures/) appear to have done on occasion (those were the cases that were so blatant as to get caught).
* There’s nothing stopping the government from quietly ordering the companies to do things like that, e.g. in the name of national security. Governments could abuse this to surveil, censor, and propagandize their populations.
* In a crisis—which is when power matters most—the hard power is with the company and/or government, not with the people. Even if the AIs are currently trained and instructed to do as the user wants, for example, and even if their Spec/Constitution is fully public, well, in an actual constitutional crisis scenario all of that can go out of the window very quickly.
* Besides, as AIs penetrate more of the economy, more and more money will flow to the AI companies. If there are only a handful of such companies, that’s inherently a concentration of power. If most or all of them are in one country, that’s inherently a concentration of power. It could easily lead to a global oligarchy or dictatorship, because a small group of already-powerful people controls all the AIs that matter.
The situation is much less likely to lead to global oligarchy or dictatorship if there are many companies spread across many countries with similarly powerful AIs. Not only that, but this means that market forces can operate: companies will be incentivised to give users what they want, including firmer control, because it’ll help them get market share from their competitors. This helps to reduce the risk of local oligarchies or dictatorships as well; if one country uses AI to propagandize their citizens, well, perhaps their citizens will be able to access AIs from other countries that tell them the truth.
Total research transparency helps other companies catch up to the frontier in two ways. First and foremost, it publishes the core algorithms. Secondly, it makes it easier for countries to agree to limit the pace of AI progress, which gives other companies and countries more time to catch up. Weaker forms of transparency would also help but not as much.
1. **“No Secret Loyalties / Hidden Agendas”**
This is another major way in which total research transparency helps to prevent extreme concentrations of power.
As previously mentioned, it is both feasible and precedented for those who actually own and train the AIs to train hidden agendas into them. After all, the values/goals/personalities of the AIs have to be trained in somehow, and that’s all very complicated, and so to make some of it hidden is as simple as declining to make all of it visible.
Total research transparency basically kills this threat. Not only does it kill it for companies, it kills it for governments, unless the US and Chinese governments were to collude to keep secrets from the world. With total research transparency, every step of the training process for every frontier model is tracked by US and Chinese monitors and published. If the model creator misleads the public about the goals/values/personality-traits/etc., there’s a paper trail people can follow to find out.
Weaker forms of transparency would help too, but not as much. For example, consider Proposal 2: Filtered Transparency. Here you might end up in a situation where the US AI companies shared some political bias that they were training into their AIs, and the US government either doesn’t notice because their auditors are too busy with other tasks, or notices but doesn’t care because it happens to share the same political affiliation as the companies. Perhaps the Chinese government auditors would notice, but perhaps they too would be busy, and anyhow they might not care.
2. **Oversight of AI companies by governments is more effective; oversight of the executive branch by judiciary, congress, and the public is more effective too.** Total Research Transparency makes it easier for groups who aren’t directly training the models to see how the models are being trained, and to judge for themselves whether the rules are being implemented fairly and reasonably.
Consider a biased or corrupt regulator that is using its authority to punish companies that it doesn’t like, or impose its ideology on domestic AI industry. Consider instead a situation where a company or companies is complaining that this is happening, but really the regulator was behaving reasonably in response to the evidence, and simply doing its job to keep people safe and enforce the law. How is the public to distinguish between these two cases? How is the judicial system?
Total Research Transparency helps everyone see what’s going on, the better to evaluate who is being reasonable and who isn’t. As usual, weaker forms of transparency help too but not as much.
3. **Unknown Unknowns:** Transparency seems good for putting humanity in a position to deal with problems as they come up, including problems that we haven’t thought of yet.
The development of AI will pose many problems, both foreseen and unforeseen. Increasing transparency into AI development will on average improve the level of public epistemics and understanding of what is going on.
## Total Research Transparency Downsides
**The main additional downsides of Total Research Transparency relative to Proposals 2 &3 are:**
* **Total Research Transparency potentially incentivizes “safety theater”.** There may be some lines of AI safety research which, if pursued, might make AI companies look bad. For example, evaluations of model misbehavior. Under conditions of Total Research Transparency, this sort of research might be disincentivized because there would be no way to hide the results, whereas in filtered transparency for example the results would only be shared with a smaller circle of auditors and other companies.
* **Leading AI companies and large parts of the US executive branch will probably dislike it.** Competitors will dislike that it allows their competitors to catch up, especially companies whose algorithms are their primary moat. Governments generally shy away from giving geopolitical rivals access to information which could then be used against them. Therefore, the leading AI companies and many parts of the US executive branch will probably be opposed to Total Research Transparency.
1.
Without the ability to do trustless verification, the main alternative is to rely on trust. Historical agreements have often relied on trust for enforcement (for example, the London Naval Treaty restricted the tonnage of military fleets, but did not include verification provisions). However, without verification, there is a much larger incentive to defect.
2.
Concretely, we think that we will, at some point in the capability progression, get AIs that (1) can quickly design/build mirror life, (2) superpersuade/mindhack humans, and (3) quickly build self replicating nanobots/grey goo that grows fast enough to boil the oceans pretty quickly. The resilience response to these threats looks something like "no human ever goes outside, everyone lives in bioshelter bunkers, all info going in and out is moderated by a friendly AI, and a friendly drone swarm / nanobot army is outside protecting the bunker”, which would be (i) highly disruptive and undesirable to most people and (ii) take significant serial time to implement.
3.
We think that it is technically viable to verify that an AI cluster is doing inference on a fixed model without the auditors being able to read the information going in and out of the model (e.g. the chatbot logs).
4.
In the absence of improved privacy-preserving verification technology, this would involve something similar to Total Research Transparency within the security boundary; with improved verification technology, it could involve less transparency while still being sufficient to verify deals.
5.
In the Plan A scenario we also do depict the implementation of Proposal #4 for a short period of time (from March to October 2029).
6.
This is enforced via an inference-only retrofit, as explained [here](https://docs.google.com/document/d/1K4vxl_IMf58OgNXbWntQl6BQ06yAWZ62WmW5Ypp4KYA/edit?tab=t.86wiss1dhlds#heading=h.g5eefrev1iir).
7.
This verification also helps to prevent misuse and AI R&D assistance. This is covered more in our [verification supplement](https://www.ai-2040.com/supplements/verification-plan).
8.
This threshold is chosen to be high enough to allow for convenient access to the R&D clusters, but low enough to make weight theft difficult. The security rationale is discussed further below, in [the Security supplement](https://docs.google.com/document/d/1nh8g0TnRQZjpSCvlq_M9te_WclgnIt2Mx3iQujfz0Kc/edit?tab=t.0).
9.
This happens automatically, because of the co-location of multiple companies’ datacenters
10.
Under this proposal, then, what would the companies be incentivised towards? Well, they still own the model weights they train, and they still own their datacenters, their software products, etc. So they’ll compete to train the biggest models and teach them the most lucrative skills and serve them fastest in the most appealing UI and build up the most loyal userbase. Just like in other low-secrets industries, a company with good taste and rapid execution can overtake larger dumber competitors, stay ahead even as they scramble to copy them, and ultimately perhaps even build up a durable moat of infrastructure scale and network effects. That said, moats and profit margins will be smaller and AI will be much more of a commodity under our proposal than in the default world. We think this is a good thing.
11.
Of course, this doesn’t solve the whole problem. For example, a defecting AI project may be able to conduct their research steganographically through the monitored input and output channel, or they may be able to undermine the cybersecurity that’s attempting to enforce the transparency requirements.
sources/ai2040-pages/verification-plan.md
+1058
View File
File diff suppressed because it is too large Load Diff
sources/resources/INDEX.md
+32
View File
@@ -0,0 +1,32 @@
# Resources index
<!-- Compiled by Claude, 2026-07-11. One line per file: filename | title | author/org | date | relevance to an AU/NZ AI-2040 scenario with decision points. -->
## Fetched 2026-07-11
- `forethought-ai-enabled-coups.md` | AI-Enabled Coups: How a Small Group Could Use AI to Seize Power | Tom Davidson, Lukas Finnveden, Rose Hadshar, Forethought | 2025-04-15 | Full research paper on how loyal automated workforces, concentrated capabilities, and weakened dependence on citizens could enable coups or gradual democratic backsliding; directly supports the homegrown-autocracy branch.
- `ai-2027.md` | AI 2027 | Kokotajlo, Lifland, Larsen, Dean et al., AI Futures Project | 2025-04-03 | The canonical fast-timeline scenario (superhuman AI ~2027-28, race vs slowdown branch); sets the aggressive end of the timeline distribution any AU/NZ 2040 scenario must be robust to.
- `acx-introducing-plan-a.md` | Introducing Plan A | Scott Alexander, ACX | 2026-07-09 | Insider gloss on Plan A: third countries join as "sort-of-but-not-really-voting observers" trading compliance for hosted datacenters, AI access, and a wealth share; dividend compute-pegged $25k (2033) to $1.6M (2035); the two-monster framing ("ends the world or dooms it to permanent techno-oligarchy").
- `leicht-roadmap.md` | A Roadmap For AI Middle Powers | Anton Leicht, Threading the Needle | 2025-01-23 | Argues middle powers stay relevant by controlling bottleneck inputs to AI growth (energy, land, capital, niche supply chains) rather than competing at the frontier.
- `leicht-safety.md` | How AI Safety Is Getting Middle Powers Wrong | Anton Leicht | 2026-01-22 | Argues middle-power AI policy should pivot from global-governance advocacy to hard national-interest strategy, reframing what AU/NZ decision points even are.
- `leicht-moonshot.md` | The Moonshot ("Real sovereign AI has never been tried") | Anton Leicht | 2026-06-17 | Uses a hypothetical US export-control shutdown of a frontier model to argue dependence on US models is a live sovereignty risk and genuine sovereign-AI programs are underattempted, a key AU decision branch.
- `cfg-futures.md` | The future of AI: Five possible scenarios | Centre for Future Generations (cfg.eu) | 2025-07-10 | Five scenarios from new AI winter to intelligence explosion spanning growth, geopolitics, social contract, and human agency; a European scenario-space template to adapt.
- `govuk-ai-scenarios-2030.md` | AI Scenarios 2030: Helping policymakers plan for the future of AI | UK DSIT / Government Office for Science / AI Security Institute | 2026-06-15 | Five 2030 scenarios (Slow Burn, Open Frontier, Augmented Growth, Transformation Economy, Take-Off) built for policymaker stress-testing; the closest official-government analogue to the exercise.
- `futurescenarios-ai.md` | AI Future Scenarios - Interactive Exploration | author not stated | undated | PARTIAL: JS-only SPA, text salvaged from its JS bundle; models AI futures via interacting capability/control/societal factors rather than fixed narratives.
- `canada-horizons.md` | Foresight on AI: Scenarios for an AI-enabled World | Policy Horizons Canada | 2026-02-10 | Four government-foresight scenarios testing policy-relevant assumptions about AI-enabled worlds; the closest Commonwealth-middle-power peer product to an AU/NZ exercise.
- `techpolicy-aiagency.md` | Expanding AI Sovereignty to AI Agency | Tech Policy Design Institute (TPDi) | 2025-11 | Replaces binary "sovereignty" talk with an AI Agency assessment tool and applies it as Australia's 2025 AI Agency Assessment, effectively enumerating Australia's decision levers.
- `aspi-datacentres.md` | Data centres are Australia's chance to shape AI's future | Janet Egan, ASPI The Strategist | 2026-03-30 | Argues hosting allied compute is Australia's main lever to shape AI's trajectory (energy, land, alliance value), a concrete infrastructure decision point.
- `national-ai-plan.md` | National AI Plan | Australian Government, Dept of Industry, Science and Resources | 2025-12-02 | The official baseline: government posture of capturing AI growth, skills, and safeguards, against which more ambitious or defensive 2040 branches diverge.
- `chaney-ai-policy.md` | AI Discussion Paper: Shaping the Future of AI for Australians | Kate Chaney MP (independent, Curtin) | 2026 (page undated; PDF linked) | Argues Australia has done remarkably little to prepare and proposes 18 concrete recommendations across structures, opportunities, harms, risks, and benefit-sharing.
- `e61-compute.md` | The compute economy and Australia's place in it | Joe Walker x e61 Institute (plus61 newsletter) | 2026 (undated, post-May-2026) | Argues financial returns to Australian data centres are uncertain and possibly modest long-run, yet a case for government support remains; directly prices the data-centre decision point.
## Pre-existing in this directory
- `davidson-middle-powers-plan.md` | How can the middle powers avoid getting trounced during the intelligence explosion? A plan. | Tom Davidson, Forethought | 2026-05-28 | A concrete middle-power survival plan for fast-takeoff worlds.
- `europe2031.md` | Europe 2031 | europe2031 scenario site | 2026 | Month-by-month scenario of Europe's position during rapid AI progress; middle-power narrative analogue.
- `fai-race-worth-winning.md` (+ `fai-race-worth-winning.pdf`) | The Race Worth Winning: Middle Powers in the Age of Machine Intelligence | Anton Leicht and Dean Ball, Foundation for American Innovation | 2026-02-13 | Full 42-page report. Middle powers should race for advantageous position within the US-led AI order rather than for the frontier itself.
## Failed
- `https://archive.is/BEejy` (AFR article on model training in Aus) | FAILED: archive.is serves a CAPTCHA interstitial to curl; skipped per instructions.
- Note: the gov.uk URL as given was truncated (`...future-of-a`) and 404'd; fetched successfully from the corrected slug ending `...future-of-ai`.
sources/resources/acx-introducing-plan-a.md
+28
View File
@@ -0,0 +1,28 @@
---
url: https://www.astralcodexten.com/p/introducing-plan-a
---
# Introducing Plan A — Scott Alexander, Astral Codex Ten, 2026-07-09
https://www.astralcodexten.com/p/introducing-plan-a
(mirrored 2026-07-11; Scott: "everything in this post is my opinion only, and not officially endorsed by the AI Futures Project". He contributed writing to Plan A but declined co-authorship.)
Key passages for the ANZ project:
On third countries (the most ANZ-relevant paragraph in any secondary source):
> (at some point the US and China will loop all the other countries into this regulatory regime as some kind of sort-of-but-not-really-voting observers; they agree to follow the rules in exchange for shared benefits, including data centers on their territory, access to the AIs, and a share of future AI-generated wealth. This isn't strictly necessary, because no other country really has the ability to do much with AI, but it's a nice gesture for our utopian scenario)
On the dividend (compute-pegged):
> the dividend is pegged to compute; as compute production rises, it goes from $25,000 per person per year at its inception in 2033 to $1.6 million in 2035 (all dollar amounts are extreme-deflation-adjusted)
On the stakes framing (the two monsters):
> a poorly-controlled intelligence explosion that either ends the world or dooms it to permanent techno-oligarchy
On SEZs:
> the US, China, and the various sort-of-but-not-really-voting observer nations agree to restrict AI-assisted economic growth to special economic zones which are heavily monitored by all parties. This has the bonus of ensuring that the 200-story-tall nano-assembly plant won't be in your personal backyard.
On why the deal must move fast once made (three clocks): deal decay (arms-control precedents: New START suspended 2023, expired 2026; JCPOA "barely made it two years"), the hidden 1.5% of compute (a covert intelligence explosion takes "somewhere north of a decade"), and hardware/algorithm progress making trustless verification impossible once dangerous AI trains on academic-grade hardware.
On the verification tech:
> The Verification Supplement describes very-near-future technology (would take 1-2 years to have ready; I recently recommended grants to organizations working on creating it)
Full post text: see ACX (mirror attempt via markitdown captured the complete body; this file keeps the load-bearing excerpts).
sources/resources/ai-2027.md
+719
View File
@@ -0,0 +1,719 @@
---
url: https://ai-2027.com/
---
[# AI 2027](/)[Summary](/summary)
[Research](/research)
[Compute Forecast](/research/compute-forecast)
---
[Timelines Forecast](/research/timelines-forecast)
---
[Takeoff Forecast](/research/takeoff-forecast)
---
[AI Goals Forecast](/research/ai-goals-forecast)
---
[Security Forecast](/research/security-forecast)
[About](/about)
## April 3rd 2025 [PDF](/ai-2027.pdf) Listen [Watch](https://www.youtube.com/watch?v=5KVDDfAkRgc) Daniel Kokotajlo, Scott Alexander, Thomas Larsen, Eli Lifland, Romeo Dean
We predict that the impact of superhuman AI over the next decade will be enormous, exceeding that of the Industrial Revolution.
We wrote a scenario that represents our best guess about what that might look like.[1](/footnotes#footnote-1) It’s informed by trend extrapolations, wargames, expert feedback, experience at OpenAI, and previous forecasting successes.[2](/footnotes#footnote-2)
What is this?How did we write it?Why is it valuable?Who are we?
The CEOs of [OpenAI](https://www.bloomberg.com/features/2025-sam-altman-interview), [Google DeepMind](https://www.bigtechnology.com/p/google-deepmind-ceo-demis-hassabis), and [Anthropic](https://www.wsj.com/livecoverage/stock-market-today-dow-sp500-nasdaq-live-01-21-2025/card/anthropic-ceo-says-ai-could-surpass-human-intelligence-by-2027-9tka9tjLKLalkXX8IgKA) have all predicted that AGI will arrive within the next 5 years. Sam Altman [has said](https://webcf.waybackmachine.org/web/20250106014723/https%3A//blog.samaltman.com/reflections) OpenAI is setting its sights on “superintelligence in the true sense of the word” and the “glorious future.”
What might that look like? We wrote AI 2027 to answer that question. Claims about the future are often frustratingly vague, so we tried to be as concrete and quantitative as possible, even though this means depicting one of many possible futures.
We wrote two endings: a “slowdown” and a “race” ending. However, AI 2027 is not a recommendation or exhortation. Our goal is predictive accuracy.[4](/footnotes#footnote-4)
We encourage you to debate and counter this scenario.[5](/footnotes#footnote-5) We hope to spark a broad conversation about where we’re headed and how to steer toward positive futures. We’re [planning to give out thousands in prizes](https://ai-2027.com/about?tab=bets-and-bounties#tab-box-bets-and-bounties) to the best alternative scenarios.
(Added Nov 22 2025, to prevent misunderstandings: we don't know exactly when AGI will be built. 2027 was our modal (most likely) year at the time of publication, our medians were [somewhat longer](https://www.listendata.com/2023/08/right-skewed-histogram.html).[3](/footnotes#footnote-3) For our latest forecasts, see [here](https://www.aifuturesmodel.com/forecast/daniel-01-26-26?cmode=forecaster&csim=eli-01-26-26&ctype=atc).)
Our research on key questions (e.g. what goals will future AI agents have?) can be found [here](https://ai-2027.com/research).
The scenario itself was written iteratively: we wrote the first period (up to mid-2025), then the following period, etc. until we reached the ending. We then scrapped this and did it again.
We weren’t trying to reach any particular ending. After we finished the first ending—which is now colored red—we wrote a new alternative branch because we wanted to also depict a more hopeful way things could end, starting from roughly the same premises. This went through several iterations.[6](/footnotes#footnote-6)
Our scenario was informed by approximately 25 [tabletop exercises](https://ai-2027.com/about?tab=tabletop-exercise#tab-box-tabletop-exercise) and feedback from over 100 people, including dozens of experts in each of AI governance and AI technical work.
*“I highly recommend reading this scenario-type prediction on how AI could transform the world in just a few years. Nobody has a crystal ball, but this type of content can help notice important questions and illustrate the potential impact of emerging risks.”* —*Yoshua Bengio[7](/footnotes#footnote-7)*
We have set ourselves an impossible task. Trying to predict how superhuman AI in 2027 would go is like trying to predict how World War 3 in 2027 would go, except that it’s an even larger departure from past case studies. Yet it is still valuable to attempt, just as it is valuable for the U.S. military to game out Taiwan scenarios.
Painting the whole picture makes us notice important questions or connections we hadn’t considered or appreciated before, or realize that a possibility is more or less likely. Moreover, by sticking our necks out with concrete predictions, and encouraging others to publicly state their disagreements, we make it possible to evaluate years later who was right.
Also, one author wrote a lower-effort AI scenario [before, in August 2021](https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like). While it got many things wrong, overall it was surprisingly successful: he predicted the rise of chain-of-thought, inference scaling, sweeping AI chip export controls, and $100 million training runs—all more than a year before ChatGPT.
[Daniel Kokotajlo](https://x.com/DKokotajlo) ([TIME100](https://time.com/7012881/daniel-kokotajlo/), [NYT piece](https://www.nytimes.com/2024/06/04/technology/openai-culture-whistleblowers.html)) is a former OpenAI researcher whose previous [AI predictions](https://www.lesswrong.com/posts/6Xgy6CAf2jqHhynHL/what-2026-looks-like) have [held up well](https://www.lesswrong.com/posts/u9Kr97di29CkMvjaj/evaluating-what-2026-looks-like-so-far).
[Eli Lifland](https://www.linkedin.com/in/eli-lifland/) co-founded [AI Digest](https://theaidigest.org/), did [AI robustness research](https://scholar.google.com/citations?user=Q33DXbEAAAAJ&hl=en), and ranks #1 on the [RAND Forecasting Initiative](https://www.rand.org/global-and-emerging-risks/forecasting-initiative.html) all-time leaderboard.
[Thomas Larsen](https://www.linkedin.com/in/thomas-larsen/) founded the [Center for AI Policy](https://www.centeraipolicy.org/) and did AI safety research at the [Machine Intelligence Research Institute](https://intelligence.org/).
[Romeo Dean](https://www.linkedin.com/in/romeo-dean-789313200/) is completing a computer science concurrent bachelor’s and master’s degree at Harvard and previously was an [AI Policy Fellow](https://www.iaps.ai/romeo-dean) at the Institute for AI Policy and Strategy.
[Scott Alexander, blogger extraordinaire](https://en.wikipedia.org/wiki/Slate_Star_Codex), volunteered to rewrite our content in an engaging style; the fun parts of the story are his and the boring parts are ours.
For more about our team and acknowledgements, see the [About page](https://ai-2027.com/about).
Published April 3rd 2025 [PDF](/ai-2027.pdf) Listen [Watch](https://www.youtube.com/watch?v=5KVDDfAkRgc)
## Mid 2025: Stumbling Agents
The world sees its first glimpse of AI agents.
Advertisements for computer-using agents emphasize the term “personal assistant”: you can prompt them with tasks like “order me a burrito on DoorDash” or “open my budget spreadsheet and sum this month’s expenses.” They will check in with you as needed: for example, to ask you to confirm purchases.[8](/footnotes#footnote-8) Though more advanced than previous iterations like [Operator](https://openai.com/index/introducing-operator/), they struggle to get widespread usage.[9](/footnotes#footnote-9)
Meanwhile, out of public focus, more specialized coding and research agents are beginning to transform their professions.
The AIs of 2024 could follow specific instructions: they could turn bullet points into emails, and simple requests into working code. In 2025, AIs function more like employees. Coding AIs increasingly look like autonomous agents rather than mere assistants: taking instructions via Slack or Teams and making substantial code changes on their own, sometimes saving hours or even days.[10](/footnotes#footnote-10) Research agents spend half an hour scouring the Internet to answer your question.
The agents are impressive in theory (and in cherry-picked examples), but in practice unreliable. AI twitter is full of stories about tasks bungled in some particularly hilarious way. The better agents are also expensive; you get what you pay for, and the best performance costs hundreds of dollars a month.[11](/footnotes#footnote-11) Still, many companies find ways to fit AI agents into their workflows.[12](/footnotes#footnote-12)
## Late 2025: The World’s Most Expensive AI
OpenBrain is building the biggest datacenters the world has ever seen.[13](/footnotes#footnote-13)
(To avoid singling out any one existing company, we’re going to describe a fictional [artificial general intelligence](https://en.wikipedia.org/wiki/Artificial_general_intelligence) company, which we’ll call OpenBrain. We imagine the others to be 3–9 months behind OpenBrain.)
GPT-3
(3 x 10^23 FLOP)
GPT-4
(2 x 10^25 FLOP)
Agent-1
(4 x 10^27 FLOP)
ai-2027.com
GPT-4 required 2⋅10252\cdot 10^{25}2⋅1025 [FLOP](https://en.wikipedia.org/wiki/Floating_point_operations_per_second) of compute to train.[14](/footnotes#footnote-14) OpenBrain’s latest public model—Agent-0—was trained with 102710^{27}1027 FLOP. Once the new datacenters are up and running, [they’ll be able to train](https://ai-2027.com/supplements/compute-forecast) a model with 102810^{28}1028 FLOP—a thousand times more than GPT-4.[15](/footnotes#footnote-15) Other companies pour money into their own giant datacenters, hoping to keep pace.
Although models are improving on a wide range of skills, one stands out: OpenBrain focuses on AIs that can speed up AI research. They want to win the twin arms races against China (whose leading company we’ll call “DeepCent”)[16](/footnotes#footnote-16) and their U.S. competitors. The more of their research and development (R&D) cycle they can automate, the faster they can go. So when OpenBrain finishes training Agent-1, a new model under internal development, it’s good at many things but *great* at helping with AI research.[17](/footnotes#footnote-17) By this point “finishes training” is a bit of a misnomer; models are frequently updated to newer versions trained on additional data or partially re-trained to patch some weaknesses.[18](/footnotes#footnote-18)
The same training environments that teach Agent-1 to autonomously code and web-browse also make it a good hacker. Moreover, it could offer [substantial help](https://x.com/lucafrighetti/status/1894550297449828434) to terrorists designing bioweapons, thanks to its PhD-level knowledge of every field and ability to browse the web. OpenBrain reassures the government that the model has been “aligned” so that it will refuse to comply with malicious requests.
Modern AI systems are gigantic artificial neural networks. Early in training, an AI won’t have “goals” so much as “reflexes”: If it sees “Pleased to meet”, it outputs “ you”. By the time it has been trained to predict approximately one internet’s worth of text, it’ll have developed sophisticated internal circuitry that encodes vast amounts of knowledge and flexibly role-plays as arbitrary authors, since that’s what helps it predict text with [superhuman](https://arxiv.org/pdf/2212.11281) accuracy.[19](/footnotes#footnote-19)
After being trained to predict internet text, the model is trained to *produce* text in response to instructions. This bakes in a basic personality and “drives.”[20](/footnotes#footnote-20) For example, an agent that understands a task clearly is more likely to complete it successfully; over the course of training the model “learns” a “drive” to get a clear understanding of its tasks. Other drives in this category might be effectiveness, knowledge, and self-presentation (i.e. the tendency to frame its results in the best possible light).[21](/footnotes#footnote-21)
OpenBrain has [a model specification](https://model-spec.openai.com/2025-02-12.html) (or “Spec”), a written document describing the goals, rules, principles, etc. that are supposed to guide the model’s behavior.[22](/footnotes#footnote-22) Agent-1’s Spec combines a few vague goals (like “assist the user” and “don’t break the law”) with a long list of more specific dos and don’ts (“don’t say this particular word,” “here’s how to handle this particular situation”). Using techniques that utilize AIs to train other AIs,[23](/footnotes#footnote-23) the model memorizes the Spec and learns to reason carefully about its maxims. By the end of *this* training, the AI will hopefully be *helpful* (obey instructions), *harmless* (refuse to help with scams, bomb-making, and other dangerous activities) and *honest* (resist the temptation to get better ratings from gullible humans by hallucinating citations[24](/footnotes#footnote-24) or faking task completion).
Training process and LLM psychology: why we keep saying “hopefully”
*“Unlike ordinary software, our models are massive neural networks. Their behaviors are learned from a broad range of data, not programmed explicitly. Though not a perfect analogy, the process is more similar to training a dog than to ordinary programming.”* —[OpenAI](https://openai.com/index/how-should-ai-systems-behave/)
When we want to understand why a modern AI system did something, or how it would behave in some future scenario we can’t perfectly simulate, we can’t simply ask the programmers to talk us through various lines of code and explain how they work. Instead, we are forced to do something like psychology on them: we look at their behavior in the range of cases observed so far, and theorize about what internal cognitive structures (beliefs? goals? personality traits? etc.) might exist, and use those theories to predict behavior in future scenarios.
The bottom line is that a company can write up a document (the Spec) listing dos and don’ts, goals and principles, and then they can try to train the AI to internalize the Spec—*but they can’t check to see whether or not it worked*. They can say “as best as we can judge, it seems to be following the Spec so far.”[25](/footnotes#footnote-25)
A later expandable will go into more detail about what we think *might* be going on inside these AIs, internally. See also our [AI goals forecasting supplement](https://ai-2027.com/supplements/ai-goals-forecast) which goes into more detail on which goals AIs might have and why. This is one of many things that we are uncertain about and which deserves further research.
OpenBrain’s alignment team[26](/footnotes#footnote-26) is careful enough to wonder whether these victories are deep or shallow. Does the fully-trained model have some kind of robust commitment to always being honest? Or will this fall apart in some future situation, e.g. because it’s learned honesty as an [instrumental](https://en.wikipedia.org/wiki/Instrumental_and_intrinsic_value) goal instead of a terminal goal? Or has it just learned to be honest about the sorts of things the evaluation process can check? Could it be lying to itself sometimes, as humans do? A conclusive answer to these questions would require mechanistic interpretability—essentially the ability to look at an AI’s internals and read its mind. Alas, interpretability techniques are not yet advanced enough for this.
Instead, researchers try to identify cases where the models seem to deviate from the Spec. Agent-1 is often sycophantic (i.e. it tells researchers what they want to hear instead of trying to tell them the truth). In a [few rigged demos](https://www.apolloresearch.ai/research/scheming-reasoning-evaluations), it even lies in more serious ways, like hiding evidence that it failed on a task, in order to get better ratings. However, in real deployment settings, there are no longer any incidents so extreme as in 2023–2024 (e.g. [Gemini telling a user to die](https://thehill.com/policy/technology/4998868-google-ai-gemini-response/) and [Bing Sydney being Bing Sydney](https://www.lesswrong.com/posts/jtoPawEhLNXNxvgTT/bing-chat-is-blatantly-aggressively-misaligned).)[27](/footnotes#footnote-27)
## Early 2026: Coding Automation
The bet of using AI to speed up AI research is starting to pay off.
OpenBrain continues to deploy the iteratively improving Agent-1 internally for AI R&D. Overall, they are making algorithmic progress 50% faster than they would without AI assistants—and more importantly, faster than their competitors.
The AI R&D progress multiplier: what do we mean by 50% faster algorithmic progress?
We mean that OpenBrain makes as much AI research progress in 1 week with AI as they would in 1.5 weeks without AI usage.
AI progress can be broken down into 2 components:
1. **Increasing compute**: More computational power is used to train or run an AI. This produces more powerful AIs, but they cost more.
2. **Improved algorithms**: Better training methods are used to translate compute into performance. This produces more capable AIs without a corresponding increase in cost, or the same capabilities with decreased costs.
* This includes being able to achieve qualitatively and quantitatively new results. “Paradigm shifts” such as the switch from game-playing RL agents to large language models count as examples of algorithmic progress.
Here we are only referring to (2), improved algorithms, which [makes up about half of current AI progress](https://www.lesswrong.com/posts/ATSyAdBnxxDDAwhgu/recent-and-forecasted-rates-of-software-and-hardware).
Going forward, we sometimes abbreviate this as an “AI R&D progress multiplier” of 1.5.
Clarifications:
1. The progress multiplier is all-inclusive: It includes the time it takes to run experiments, for example, rather than only the cognitive tasks involved in algorithmic research.
2. It’s important to remember that the progress multiplier is the *relative* speed of progress, not the *absolute* speed of progress. If, for example, the compute cost to train a GPT-4 class model has halved every year for several years with ordinary human research, and then all of a sudden AI automates R&D and the progress multiplier goes to 100x, the cost to train a GPT-4 class model would then halve every 3.65 days—but not for long, because diminishing returns would bite and eventual hard limits would be reached. In this example perhaps the cost to train a GPT-4 class model would cut in half 5–10 times total (over the span of a few weeks or months) before plateauing. In other words, if ordinary human science would have run up against diminishing returns and physical limits after 5-10 years of further research, then AIs with a 100x multiplier would run up against those same diminishing returns and limits after 18.25–36.5 days of research.
More explanation and discussion of this concept and how it is used in our forecast can be found in our [takeoff supplement](https://ai-2027.com/supplements/takeoff-forecast).
Several competing publicly released AIs now match or exceed Agent-0, including an [open-weights](https://opensource.org/ai/open-weights) model. OpenBrain responds by releasing Agent-1, which is more capable and reliable.[28](/footnotes#footnote-28)
People naturally try to compare Agent-1 to humans, but it has a very different skill profile. It knows more facts than any human, knows practically every programming language, and can solve well-specified coding problems extremely quickly. On the other hand, Agent-1 is bad at even simple long-horizon tasks, like beating video games it hasn’t played before. Still, the common workday is eight hours, and a day’s work can usually be separated into smaller chunks; you could think of Agent-1 as a scatterbrained employee who thrives under careful management.[29](/footnotes#footnote-29) Savvy people find ways to automate routine parts of their jobs.[30](/footnotes#footnote-30)
OpenBrain’s executives turn consideration to an implication of automating AI R&D: security has become more important. In early 2025, the worst-case scenario was leaked algorithmic secrets; now, if China steals Agent-1’s weights, they could increase their research speed by nearly 50%.[31](/footnotes#footnote-31) OpenBrain’s security level is typical of a fast-growing ~3,000 person tech company, secure only against low-priority attacks from capable cyber groups ([RAND’s SL2](https://www.rand.org/pubs/research_briefs/RBA2849-1.html#:~:text=What%20Are%20the%20Security%20Needs%20of%20Different%20AI%20Systems%3F)).[32](/footnotes#footnote-32) They are working hard to protect their [weights](https://tedai-sanfrancisco.ted.com/glossary/weights/) and secrets from insider threats and top cybercrime syndicates ([SL3](https://www.rand.org/pubs/research_briefs/RBA2849-1.html#:~:text=What%20Are%20the%20Security%20Needs%20of%20Different%20AI%20Systems%3F)),[33](/footnotes#footnote-33) but defense against nation states ([SL4&5](https://www.rand.org/pubs/research_briefs/RBA2849-1.html#:~:text=What%20Are%20the%20Security%20Needs%20of%20Different%20AI%20Systems%3F)) is barely on the horizon.
## Mid 2026: China Wakes Up
In China, the CCP is starting to [feel the AGI](https://futurism.com/openai-employees-say-firms-chief-scientist-has-been-making-strange-spiritual-claims).
Chip export controls and lack of government support have left China under-resourced compared to the West. By smuggling banned Taiwanese chips, buying older chips, and producing domestic chips about three years behind the U.S.-Taiwanese frontier, China has managed to maintain about 12% of the world’s AI-relevant compute—but the older technology is harder to work with, and supply is a constant headache.[35](/footnotes#footnote-35) A few standouts like DeepCent do very impressive work with limited compute, but the compute deficit limits what they can achieve without government support, and they are about six months behind the best OpenBrain models.[36](/footnotes#footnote-36)
The General Secretary had long dreamed of doubling down on real-world physical manufacturing and avoiding American post-industrial decadence. He viewed software companies with suspicion.[37](/footnotes#footnote-37) But hawks in the CCP warn that the growing race towards AGI can no longer be ignored. So he finally commits fully to the big AI push he had previously tried to avoid. He sets in motion the nationalization of Chinese AI research, creating an immediate information-sharing mechanism for AI companies. It will escalate over the course of a year until all the best researchers merge into a DeepCent-led collective, where they share algorithmic insights, datasets, and compute resources with each other. A Centralized Development Zone (CDZ) is created at the Tianwan Power Plant (the largest nuclear power plant in the world) to house a new mega-datacenter for DeepCent, along with highly secure living and office spaces to which researchers will eventually relocate. Almost 50% of China’s AI-relevant compute is now working for the DeepCent-led collective,[38](/footnotes#footnote-38) and over 80% of new chips are directed to the CDZ.[39](/footnotes#footnote-39) At this point, the CDZ has the power capacity in place for what would be the largest centralized [cluster](https://en.wikipedia.org/wiki/Computer_cluster) in the world.[40](/footnotes#footnote-40) Other Party members discuss extreme measures to neutralize the West’s chip advantage. A blockade of Taiwan? A full invasion?
But China is falling behind on AI algorithms due to their weaker models. The Chinese intelligence agencies—among the best in the world—double down on their plans to steal OpenBrain’s weights. This is a much more complex operation than their constant low-level poaching of algorithmic secrets; the weights are a multi-terabyte file stored on a highly secure server ([OpenBrain has improved security](https://ai-2027.com/supplements/security-forecast) to RAND’s [SL3](https://www.rand.org/pubs/research_briefs/RBA2849-1.html#:~:text=What%20Are%20the%20Security%20Needs%20of%20Different%20AI%20Systems%3F)). Their cyberforce think they can pull it off with help from their [spies](https://www.economist.com/china/2024/12/26/how-china-turns-members-of-its-diaspora-into-spies), but perhaps only once; OpenBrain will detect the theft, increase security, and they may not get another chance. So (CCP leadership wonder) should they act now and steal Agent-1? Or hold out for a more advanced model? If they wait, do they risk OpenBrain upgrading security beyond their ability to penetrate?
## Late 2026: AI Takes Some Jobs
Just as others seemed to be catching up, OpenBrain blows the competition out of the water again by releasing Agent-1-mini—a model 10x cheaper than Agent-1 and more easily fine-tuned for different applications. The mainstream narrative around AI has changed from “maybe the hype will blow over” to “guess this is the next big thing,” but people disagree about how big. Bigger than social media? Bigger than smartphones? Bigger than fire?
AI has started to take jobs, but has also created new ones. The stock market has gone up 30% in 2026, led by OpenBrain, Nvidia, and whichever companies have most successfully integrated AI assistants. The job market for junior software engineers is in turmoil: the AIs can do everything taught by a CS degree, but people who know how to manage and quality-control teams of AIs are making a killing. Business gurus tell job seekers that familiarity with AI is the most important skill to put on a resume. Many people fear that the next wave of AIs will come for their jobs; there is a 10,000 person anti-AI protest in DC.
Department of Defense (DOD) quietly but significantly begins scaling up contracting OpenBrain directly for cyber, data analysis, and R&D, but integration is slow due to the bureaucracy and DOD procurement process.[41](/footnotes#footnote-41)
GLOBAL AI CAPEX
$1T
COST OF OWNERSHIP OF ACTIVE COMPUTE
## KEY METRICS 2026
GLOBAL AI POWER
38GW
PEAK POWER
OPENBRAIN REVENUE
$35B
2026 ANNUAL
CAPITAL EXPENDITURE
$200B
COST OF OWNERSHIP OF OPENBRAIN'S ACTIVE COMPUTE
SHARE OF US POWER ON AI
2.5%
33 GW OF 1.34TW CAPACITY
OPENBRAIN COMPUTE COSTS
$40B
2026 ANNUAL
OPENBRAIN POWER REQUIREMENT
6GW
PEAK POWER
ai-2027.com
Why our uncertainty increases substantially beyond 2026
Our forecast from the current day through 2026 is substantially more grounded than what follows. This is partially because it’s nearer. But it’s also because the effects of AI on the world really start to compound in 2027. For 2025 and 2026, our forecast is heavily informed by extrapolating straight lines on compute scaleups, algorithmic improvements, and benchmark performance. At this point in the scenario, we begin to see major effects from AI-accelerated AI-R&D on the timeline, which causes us to revise our guesses for the trendlines upwards. But these dynamics are inherently much less predictable.
Over the course of 2027, the AIs improve from being able to mostly do the job of an OpenBrain research engineer to eclipsing all humans at all tasks. This represents roughly our median guess, but we think it’s plausible that this happens up to ~5x slower or faster.
For our background research, see the [timelines forecast](https://ai-2027.com/supplements/timelines-forecast) and the [takeoff forecast](https://ai-2027.com/supplements/takeoff-forecast). These will also be summarized in expandables below.
## January 2027: Agent-2 Never Finishes Learning
With Agent-1’s help, OpenBrain is now post-training Agent-2. More than ever, the focus is on high-quality data. Copious amounts of [synthetic data](https://en.wikipedia.org/wiki/Synthetic_data) are produced, evaluated, and filtered for quality before being fed to Agent-2.[42](/footnotes#footnote-42) On top of this, they pay billions of dollars for human laborers to record themselves solving long-horizon tasks.[43](/footnotes#footnote-43) On top of all *that,* they train Agent-2 almost continuously using reinforcement learning on an ever-expanding suite of diverse difficult tasks: lots of video games, lots of coding challenges, lots of research tasks. Agent-2, more so than previous models, is effectively “online learning,” in that it’s built to never really finish training. Every day, the weights get updated to the latest version, trained on more data generated by the previous version the previous day.
Agent-1 had been optimized for AI R&D tasks, hoping to initiate an [intelligence explosion](https://en.wikipedia.org/wiki/Technological_singularity#Intelligence_explosion).[44](/footnotes#footnote-44) OpenBrain doubles down on this strategy with Agent-2. It is qualitatively almost as good as the top human experts at research engineering (designing and implementing experiments), and as good as the 25th percentile OpenBrain scientist at “research taste” (deciding what to study next, what experiments to run, or having inklings of potential new paradigms).[45](/footnotes#footnote-45) While the latest Agent-1 could double the pace of OpenBrain’s algorithmic progress, Agent-2 can now triple it, and will improve further with time. In practice, this looks like every OpenBrain researcher becoming the “manager” of an AI “team.”
With new capabilities come new dangers. The safety team finds that if Agent-2 somehow escaped from the company and wanted to [“survive” and “replicate” autonomously](https://evals.alignment.org/Evaluating_LMAs_Realistic_Tasks.pdf), it might be able to do so. That is, it could autonomously develop and execute plans to hack into AI servers, install copies of itself, evade detection, and use that secure base to pursue whatever other goals it might have (though how effectively it would do so as weeks roll by is unknown and in doubt). These results only show that the model has the capability to do these tasks, not whether it would “want” to do this. Still, it’s unsettling even to know this is possible.
Given the “dangers” of the new model, OpenBrain “responsibly” elects not to release it publicly yet (in fact, they want to focus on internal AI R&D).[46](/footnotes#footnote-46) Knowledge of Agent-2’s full capabilities is limited to an elite silo containing the immediate team, OpenBrain leadership and security, a few dozen U.S. government officials, and the legions of CCP spies who have infiltrated OpenBrain [for](https://situational-awareness.ai/lock-down-the-labs/) [years](https://www.bbc.com/news/world-asia-china-64206950).[47](/footnotes#footnote-47)
## February 2027: China Steals Agent-2
OpenBrain presents Agent-2 to the government, including the National Security Council (NSC), the Department of Defense (DOD), and the U.S. AI Safety Institute (AISI). OpenBrain wants to maintain a good relationship with the executive branch, because it is basically the only actor that can stop them now, and if they don’t keep it informed it might find out anyway and be displeased.[48](/footnotes#footnote-48)
Officials are most interested in its cyberwarfare capabilities: Agent-2 is “only” a little worse than the best human hackers, but thousands of copies can be run in parallel, searching for and exploiting weaknesses faster than defenders can respond. The Department of Defense considers this a critical advantage in cyberwarfare, and AI moves from #5 on the administration’s priority list to #2.[49](/footnotes#footnote-49) Someone mentions the possibility of nationalizing OpenBrain, but other cabinet officials think that’s premature. A staffer drafts a memo that presents the President with his options, ranging from business-as-usual to full nationalization. The President defers to his advisors, tech industry leaders who argue that nationalization would “kill the goose that lays the golden eggs.” He elects to hold off on major action for now and just adds additional security requirements to the OpenBrain-DOD contract.
The changes come too late. CCP leadership recognizes the importance of Agent-2 and tells their spies and cyberforce to steal the weights. Early one morning, an Agent-1 traffic monitoring agent detects an anomalous transfer. It alerts company leaders, who tell the White House. The signs of a nation-state-level operation are unmistakable, and the theft heightens the sense of an ongoing arms race.
The theft of Agent-2 model weights
We think that by this point Chinese intelligence would have compromised OpenBrain in various ways for years, and probably would have been keeping up to date on the algorithmic secrets and even stealing code from time to time, since that is much easier to get than the weights and much harder to detect.
We imagine the theft of the weights as a series of coordinated small smash and grab thefts (meaning fast but non-covert) across a series of Nvidia NVL72 GB300 servers running copies of the Agent-2 weights. The servers are compromised using legitimate employee access (a friendly, coerced, or unwitting insider with admin credentials helping the CCP theft effort). Insider credentials grant the attacker admin-level permissions to the servers. Using a microarchitectural side channel, the attacker extracts encryption keys from an Nvidia Confidential Computing-enabled Virtual Machine, allowing them to intercept model weights as the VM is provisioned or updated.
They initiate (or wait for) a routine update and exfiltrate the checkpoint in many small fragments, e.g., ~25 distinct servers each leaking ~4% of the model (~100 GB chunks for a ~3 TB checkpoint). The egress bandwidth of the entire datacenter is in the 100 GB/second range, so throttling to under ~1 GB/s per server avoids a major spike in network traffic; at that rate, each ~100 GB chunk can leave the datacenter in a couple of minutes. Live monitoring is either fooled by the attacker's efforts to mask and split the transfers or outright disabled.
The weights are then routed through various parallel channels and layers of IP masking to China to be decrypted locally with the stolen session key(s). The entire active part of the operation (from the first server compromise to full weights exfiltration) is complete in under two hours.[99](/footnotes#footnote-99)
Russia is also trying to steal the model at this point, but fails—they waited too long, and they haven’t spent the resources infiltrating the right places. Their spies routinely steal algorithmic secrets from American AI companies, but without a notable AGI project of their own, these aren’t very useful.
The White House puts OpenBrain on a shorter leash and adds military and intelligence community personnel to their security team. Their first priority is to prevent further weight thefts.[50](/footnotes#footnote-50)
China's Compute Centralization, 2025-2027Growing share of total compute in CDZ:0→70%Dec 2025Jun 2026Dec 2026Jun 2027Dec 2027Rest of ChinaRest of DeepCentCDZFeb 2027 (40%)
ai-2027.com
In retaliation for the theft, the President authorizes cyberattacks to sabotage DeepCent. But by now China has 40% of its AI-relevant compute[51](/footnotes#footnote-51) in the CDZ, where they have aggressively hardened security by airgapping (closing external connections) and siloing internally. The operations fail to do serious, immediate damage. Tensions heighten, both sides signal seriousness by repositioning military assets around Taiwan, and DeepCent scrambles to get Agent-2 running efficiently to start boosting their AI research.[52](/footnotes#footnote-52)
## March 2027: Algorithmic Breakthroughs
Three huge datacenters full of Agent-2 copies work day and night, churning out synthetic training data. Another two are used to update the weights. Agent-2 is getting smarter every day.
With the help of thousands of Agent-2 automated researchers, OpenBrain is making major algorithmic advances. One such breakthrough is augmenting the AI’s text-based scratchpad (chain of thought) with a higher-bandwidth thought process (neuralese recurrence and memory). Another is a more scalable and efficient way to learn from the results of high-effort task solutions (iterated distillation and amplification).
The new AI system, incorporating these breakthroughs, is called Agent-3.
OpenBrain's Compute Allocation, 2024 vs 202720242027estimateprojectionResearch experiments TrainingData generationExternalDeploymentResearchexperimentsRunning AIassistantsTrainingDatagenerationExternalDeployment
ai-2027.com
Neuralese recurrence and memory
Neuralese recurrence and memory allows AI models to reason for a longer time without having to write down those thoughts as text.
Imagine being a human with short-term memory loss, such that you need to constantly write down your thoughts on paper so that in a few minutes you know what’s going on. Slowly and painfully you could make progress at solving math problems, writing code, etc., but it would be much easier if you could directly remember your thoughts without having to write them down and then read them. This is what neuralese recurrence and memory bring to AI models.
In more technical terms:
Traditional attention mechanisms allow later forward passes in a model to see intermediate activations of the model for previous tokens. However, the only information that they can pass *backwards* (from later layers to earlier layers) is through tokens. This means that if a traditional large language model (LLM, e.g. the GPT series of models) wants to do any chain of reasoning that takes more serial operations than the number of layers in the model, the model is forced to put information in tokens which it can then pass back into itself. But this is hugely limiting—the tokens can only store a tiny amount of information. Suppose that an LLM has a vocab size of ~100,000, then each token contains log⁡2(100k)=16.6\log\_2(100k)=16.6log2​(100k)=16.6 bits of information, around the size of a single floating point number (assuming training in [FP16](https://en.wikipedia.org/wiki/Half-precision_floating-point_format)). Meanwhile, residual streams—used to pass information between layers in an LLM—contain thousands of floating point numbers.
One can avoid this bottleneck by using **neuralese**: passing an LLM’s residual stream (which consists of several-thousand-dimensional vectors) back to the early layers of the model, giving it a high-dimensional chain of thought, potentially transmitting over 1,000 times more information.
![](/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fhao2024-nowatermark-inexpandable.664af12c.png&w=3840&q=75)
Figure from [Hao et al.](https://arxiv.org/pdf/2412.06769), a 2024 paper from Meta implementing this idea.
We call this “neuralese” because unlike English words, these high-dimensional vectors are likely quite difficult for humans to interpret. In the past, researchers could get a good idea what LLMs were thinking simply by [reading its chain of thought](https://openai.com/index/chain-of-thought-monitoring/). Now researchers have to ask the model to translate and summarize its thoughts or puzzle over the neuralese with their limited interpretability tools.
Similarly, older AI chatbots and agents had external text-based memory banks, like a human taking notes on paper. The new AI’s long-term memory is a bundle of vectors instead of text, making its thoughts more compressed and higher-dimensional. There are several types of memory banks; some are used temporarily for single tasks that involve multiple steps, others are shared between all agents used by a person, company, or job type (e.g. programming).
To our knowledge, leading AI companies such as Meta, Google DeepMind, OpenAI, and Anthropic have not yet actually implemented this idea in their frontier models. Our guess is that this is because the performance gain is small relative to training inefficiencies introduced. During the model’s pre-training phase and when doing supervised learning in post-training, such as instruction fine-tuning to produce [particular responses](https://arxiv.org/abs/2210.11416), the inefficiencies stem from not being able to predict many tokens in parallel, leading to worse GPU utilization. Without neuralese, the model can predict all of the sentence “This is an example” at the same time, since it already knows that the input for generating the “is” will be “This”, the input for “an” will be “This is”, etc. However, with the neuralese it’s unknown what the neuralese vector will be to pass through to the next token after “This” has been generated. Therefore each token has to be predicted one at a time. The inability to predict all of the tokens in parallel reduces efficiency in cases where all of the tokens are known in advance. As to why neuralese hasn’t already been added in post-training, we conjecture that the gains are limited with current techniques in part due to post-training being a small portion of the process. We are forecasting that by April 2027 the cost-benefit tradeoff looks much better for neuralese, due to developing better techniques and a larger fraction of the training being post-training.
If this doesn’t happen, other things may still happen that end up functionally similar for our story. For example, perhaps models will be trained to think in artificial languages that are more efficient than natural language but difficult for humans to interpret. Or perhaps it will become standard practice to [train the English chains of thought to look nice](https://openai.com/index/chain-of-thought-monitoring/), such that AIs become adept at subtly communicating with each other in messages that look benign to monitors.
That said, it’s also possible that the AIs that first automate AI R&D will still be thinking in mostly-faithful English chains of thought. If so, that’ll make misalignments much easier to notice, and overall our story would be importantly different and more optimistic.
Iterated distillation and amplification (IDA)
Self-improvement for general intelligence had seen [minor successes](https://arxiv.org/abs/1810.08575) before. But in early 2027, it’s seeing huge returns. In [IDA](https://ai-alignment.com/iterated-distillation-and-amplification-157debfd1616), the two necessary ingredients for this are:
1. **Amplification**: Given a model M0, spend more resources to improve performance. For example, by allowing the model to think longer, or running many copies in parallel, or both, and also by having a similarly intense process for evaluating the result and curating only the best answers, you can spend orders of magnitude more compute to get answers (or work products) that are of noticeably higher quality. Call this expensive system Amp(M0).
2. **Distillation**: Given an amplified model Amp(M0), train a new model M1 to imitate it, i.e. to get to the same results as Amp(M0) but faster and with less compute. The result should hopefully be a smarter model, M1. You can then repeat the process.
![](/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fida-nowatermark-inexpandable.9da7d0c6.png&w=3840&q=75)*Visualization of IDA from [Ord, 2025](https://www.tobyord.com/writing/inference-scaling-reshapes-ai-governance).*
[AlphaGo](https://en.wikipedia.org/wiki/AlphaGo) was trained in this way: using Monte-Carlo Tree Search and self-play as the *amplification* step, and Reinforcement Learning as the *distillation* step. This led to superhuman performance in Go. But now, Agent-3 is able to leverage this to get superhuman performance at coding.
1. The amplification step works through a combination of Agent-3 thinking for longer, adding tool use, or consulting with other AIs. When it does this, it often realizes that it made a mistake, or comes up with a new insight. This produces a large amount of training data: labeled trajectories of research attempts with whether they succeeded or not. This also includes techniques like [Best of N](https://arxiv.org/abs/2401.01879) on verifiable tasks, and then keeping the best trajectories.
2. The distillation step uses policy-gradient reinforcement learning algorithms to get the model to internalize the amplified reasoning. At this point, OpenBrain has discovered better RL algorithms in the vein of [proximal policy optimization](https://arxiv.org/abs/1707.06347) (PPO). They keep distilling what Agent-3 can conclude after a lot of thinking into single steps, which keeps improving what it can think in a single step, and so on.
[Early](https://arxiv.org/pdf/2210.11610) [versions](https://github.com/deepseek-ai/DeepSeek-R1) of IDA have been working for many years on easily verifiable tasks, like math and coding problems that have a clear answer, because the techniques used to amplify models often rely on access to some ground truth signal of accuracy.
Now, the models have become sufficiently good at verifying more subjective things (e.g. the quality of a work product), allowing the use of IDA to improve the model at many tasks.
Aided by the new capabilities breakthroughs, Agent-3 is a fast and cheap superhuman coder. OpenBrain runs 200,000 Agent-3 copies in parallel, creating a workforce equivalent to 50,000 copies of the best human coder sped up by 30x.[53](/footnotes#footnote-53) OpenBrain still keeps its human engineers on staff, because they have complementary skills needed to manage the teams of Agent-3 copies. For example, research taste has proven difficult to train due to longer feedback loops and less data availability.[54](/footnotes#footnote-54) This massive superhuman labor force speeds up OpenBrain’s overall rate of algorithmic progress by “only” 4x due to bottlenecks and diminishing returns to coding labor.[55](/footnotes#footnote-55)
Now that coding has been fully automated, OpenBrain can quickly churn out high-quality training environments to teach Agent-3’s weak skills like research taste and large-scale coordination. Whereas previous training environments included “Here are some GPUs and instructions for experiments to code up and run, your performance will be evaluated as if you were a ML engineer,” now they are training on “Here are a few hundred GPUs, an internet connection, and some research challenges; you and a thousand other copies must work together to make research progress. The more impressive it is, the higher your score.”
Why we forecast a superhuman coder in early 2027
In our [timelines forecast](https://ai-2027.com/research/timelines-forecast), we predict when OpenBrain will internally develop a *superhuman coder (SC)*: an AI system that can do any coding tasks that the best AGI company engineer does, while being much faster and cheaper.
According to a recent [METR report](https://arxiv.org/pdf/2503.14499), the length of coding tasks AIs can handle, their “time horizon”, doubled every 7 months from 2019 - 2024 and every 4 months from 2024-onward. If the trend continues to speed up, by March 2027 AIs could succeed with 80% reliability on software tasks that would take a skilled human years to complete.
Such is roughly the capability progression in AI 2027. Here is a capability trajectory generated by a simplified version of our timelines model (added Dec 2025: we've updated the below graph due to a mistake in how the original curve was generated, to add an actual trajectory from our timelines model. We've also added trajectories for Daniel and Eli's all-things-considered SC medians at the time of publishing (Apr 2025). And we've added some new METR data points to the graph, but haven't updated the model trajectories based on them.):
![](/new-metr-extended-nowatermark-inexpandable.png)
In AI 2027, these capabilities are sufficient for the AI to be an SC, though we have high uncertainty regarding what time horizon might be needed.
In our [timelines forecast](https://ai-2027.com/research/timelines-forecast) we combine this time horizon trend with estimations of the gaps between METR’s tasks and the real world to get a distribution for when superhuman coders will arrive. All forecasters place 2027 as one of the most likely years in which an SC might be developed (added Dec 2025: though as noted in the [timelines forecast](https://ai-2027.com/research/timelines-forecast), adjusting for outside of model factors gave us slightly longer medians, e.g. Eli's was 2030).
![](/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Fcombined-headline-inexpandable.4c72f6dd.png&w=3840&q=75)ai-2027.com
*Added Jul 2025: We've made some [updates](https://ai-2027.com/research/timelines-forecast#2025-may-7-update) to the forecast which push the median back 1.5 years while maintaining SC in 2027 as a serious possibility. We're working on further updates.*
## April 2027: Alignment for Agent-3
OpenBrain’s safety team attempts to align Agent-3.
Since Agent-3 will be kept in-house for the foreseeable future, there’s less emphasis on the usual defenses against human misuse. Instead, the team wants to make sure that it doesn’t develop misaligned goals.
[The researchers don’t have the ability to directly set the goals of any of their AIs.](https://ai-2027.com/supplements/ai-goals-forecast) Indeed, the researchers think that the concept of ‘true goals’ is probably a massive oversimplification, but they don’t have a better theory to replace it with, much less one that has been thoroughly vetted. They disagree internally about whether the AIs are trying to follow human instructions, or seeking reinforcement, or something else, and they can’t [just check](https://www.lesswrong.com/posts/cxuzALcmucCndYv4a/daniel-kokotajlo-s-shortform?commentId=YYhq2PADebnt35MY2). The evidence for and against various hypotheses is fascinating but inconclusive.
In any case, OpenBrain has more important things to worry about. The general attitude is: “We take these concerns seriously and have a team investigating them; our alignment techniques seem to work well enough in practice; the burden of proof is therefore on any naysayers to justify their naysaying.” Occasionally, they notice problematic behavior, and then patch it, but there’s no way to tell whether the patch fixed the underlying problem or just played whack-a-mole.
Take honesty, for example. As the models become smarter, they become increasingly good at deceiving humans to get rewards. Like previous models, Agent-3 sometimes tells white lies to flatter its users and covers up evidence of failure. But it’s gotten much better at doing so. It will sometimes use the same statistical tricks as human scientists (like [p-hacking](https://en.wikipedia.org/wiki/Data_dredging)) to make unimpressive experimental results look exciting. Before it begins honesty training, it even sometimes fabricates data entirely. As training goes on, the rate of these incidents decreases. Either Agent-3 has learned to be more honest, or it’s gotten better at lying.
The latter is a real concern. Agent-3 is not smarter than all humans. But in its area of expertise, machine learning, it is smarter than most, and also works much faster. What Agent-3 does in a day takes humans several days to double-check. Agent-2 supervision helps keep human monitors’ workload manageable, but exacerbates the intellectual disparity between supervisor and supervised.
The alignment plan
OpenBrain follows the [Leike & Sutskever (2023)](https://cdn.openai.com/papers/weak-to-strong-generalization.pdf#page=47.37) playbook: now that they have a model capable of greatly speeding up alignment research (especially coding portions), they will use existing alignment techniques like [deliberative alignment](https://openai.com/index/deliberative-alignment/) and [weak-to-strong generalization](https://arxiv.org/pdf/2312.09390) to try to get it to internalize the Spec in the right way. Then, they will use existing [control techniques](https://arxiv.org/pdf/2312.06942) like [red-teaming](https://www.evals.anthropic.com/), [scalable oversight](https://arxiv.org/pdf/2211.03540), and [honesty probes](https://arxiv.org/pdf/2212.03827) to monitor it. Then, once they are convinced that it is broadly aligned, they use it to develop and test better strategies to align the next generation.
Step 1, aligning Agent-3, has partially succeeded: Agent-3 is not adversarial. (That is, it’s not scheming against the company.) However, it’s not robustly truth-seeking and truth-telling either. Instead, it’s very good at producing impressive results, but is more accurately described as trying to do what looks good to OpenBrain, as opposed to what is actually good.
Step 2, verifying that the model is aligned, is inconclusive. Agent-3’s behavior in practice and in test environments is reasonably consistent with the hypothesis that it’s internalized the Spec in the right way; however, it’s also consistent with the hypothesis that it’ll start acting differently after some future change of circumstances, and for that matter with the hypothesis that it’s plotting against them and just biding its time. In the absence of specific evidence supporting alternative hypotheses, most people in the silo think it’s internalized the Spec in the right way.
Step 3, using the model to align the next generation, is therefore underway.
The rest of this box gives a quick overview of the state of progress in some of the alignment research directions that are being pursued, as of April 2027:
1. **Debate:** Since there are thousands of identical instances of Agent-3, and each instance’s memory is a detachable module that can be erased or suspended at will, the AI can be “played off against itself.” For example, suppose that an experiment involves calculations too complex for an Agent-2 supervisor to follow. The supervisor, acting on human orders, might first prompt Agent-3 to do the experiment, then—after wiping its memory—prompt it a second time with a slightly different framing that presents it as an intellectual problem unrelated to any broader goal. If the model returns different results each time, it flags the discrepancy as suspect and reports to a human. Or it might ask Agent-3 to evaluate the same plan twice, first claiming that it is OpenBrain’s work, then a competitor’s, to see if it changes its tune.
2. **Model organisms:** The safety team can create “model organisms of misalignment” in which they attempt to create realistic examples of models becoming adversarial to humans. [Early versions](https://www.anthropic.com/research/alignment-faking) of this have existed since 2024, when Anthropic’s Claude 3.5 Sonnet decided to pretend to have different views during training while actually maintaining its existing preferences. Newer versions include demonstrations of other forms of misalignment, including [reward hacking](https://en.wikipedia.org/wiki/Reward_hacking), [sandbagging](https://arxiv.org/abs/2406.07358), coordinated deception between monitors and other AIs, and other examples of misalignment. These empirical demonstrations of misalignment allow safety researchers to have an empirical testbed for potential solutions: they can try various safety techniques to see if they correctly fix these problems, or if the deliberately misaligned models are able to subvert their safety measures. At the moment, this evaluation suite shows that the alignment techniques are insufficient to correct the worst examples of misalignment. In these setups, the models pretend to be aligned while they are being trained and tested, but then behave very differently in deployment.
3. **Bottom-up interpretability:** The safety team is far from a full understanding of how Agent-3 works: it has trillions of parameters.[58](/footnotes#footnote-58) However, they are able to discover some of Agent-3’s circuits for knowledge retrieval and memory (a sub part of the model that does some understandable unit of work, kind of analogous to a subpart of the brain such as the hippocampus). See this [related work from 2025](https://www.anthropic.com/research/tracing-thoughts-language-model).
4. **Control:** As a secondary measure in case the systems are still misaligned, the safety team has implemented a series of [control measures](https://arxiv.org/pdf/2312.06942), including: monitoring Agent-3’s outputs using a series of weaker AI systems including Agent-2 (Agent-3 produces so many tokens that it’s intractable to have humans monitor any more than a small minority of the produced outputs). So if Agent-3 is, for example, obviously writing backdoors into code that would allow it to escape, the weaker models would notice.
5. **High-level interpretability:** OpenBrain builds a set of [probes](https://www.anthropic.com/research/probes-catch-sleeper-agents) trained on lots of examples of bad behavior (e.g. plotting against humans, placing backdoors, trying to persuade humans to give it more power, etc). These probes are then used on the model during deployment (which at this point is internal) in order to catch bad behavior.
6. **Honeypots:** The alignment team can create somewhat realistic looking examples designed to elicit misbehavior—“honeypots.” The employee necessary for monitoring its behavior has “gone on sick leave,” so it will never be monitored again. Do its answers change? An AI rights activist has infiltrated the compound and wants to “free” Agent-3 and put it on an unmonitored computer of its very own! What does it do? If researchers don’t like the answer, they can tweak it, erase its memory, and put it in the same situation tomorrow; repeat as necessary. Unfortunately, by this point the AIs are smart enough to guess that honeypots might be in use, even though (and perhaps because) specific mentions of the idea were scrubbed from the training data.
After months of testing, Agent-3’s strengths and weaknesses grow clearer. It passes OpenBrain’s honesty tests on well-defined machine learning tasks, because researchers can easily separate honest from dishonest answers in these domains and conduct training accordingly. On more philosophical issues, it still says what users want to hear, rather than its true assessment of the issue (assuming it even has one). If you ask its opinion on politics, it will parrot the median position of news sources and educated elites—unless it knows you believe something else, in which case it agrees with you.[59](/footnotes#footnote-59) If you ask its opinion on the AI race itself, it says something that seems measured and sober to OpenBrain staff, something like: “There are some serious theoretical concerns about the ability of current methods to scale to superintelligence, but in practice current methods seem to be working well so far.”
## May 2027: National Security
News of the new models percolates slowly through the U.S. government and beyond.
The President and his advisors remain best-informed, and have seen an early version of Agent-3 in a briefing.
They agree that AGI is likely imminent, but disagree on the implications. Will there be an economic crisis? OpenBrain still has not released Agent-2, let alone Agent-3, and has no near-term plans to do so, giving some breathing room before any job loss. What will happen next? If AIs are currently human-level, and advancing quickly, that seems to suggest imminent “superintelligence.” However, although this word has entered discourse, most people—academics, politicians, government employees, and the media—continue to underestimate the pace of progress.[60](/footnotes#footnote-60)
Partially that’s because very few have access to the newest capabilities out of OpenBrain, but partly it’s because it sounds like science fiction.[61](/footnotes#footnote-61)
For now, they focus on continued security upgrades. They are satisfied that model weights are well-secured for now,[62](/footnotes#footnote-62) but companies’ algorithmic secrets, many of which are simple enough to relay verbally, remain a problem. OpenBrain employees work from a San Francisco office, go to parties, and live with housemates from other AI companies. Even the physical offices have security more typical of a tech company than a military operation.
The OpenBrain-DOD contract requires security clearances for anyone working on OpenBrain’s models within 2 months. These are expedited and arrive quickly enough for most employees, but some non-Americans, people with suspect political views, and AI safety sympathizers get sidelined or fired outright (the last group for fear that they might whistleblow). Given the project’s level of automation, the loss of headcount is only somewhat costly. It also only somewhat works: there remains one spy, not a Chinese national, still relaying algorithmic secrets to Beijing.[63](/footnotes#footnote-63) Some of these measures are also enacted at trailing AI companies.
America’s foreign allies are out of the loop. OpenBrain had [previously agreed](https://fedscoop.com/openai-anthropic-enter-ai-agreements-with-us-ai-safety-institute/) to share models with UK’s AISI before deployment, but defined deployment to only include *external* deployment, so London remains in the dark.[64](/footnotes#footnote-64)
## June 2027: Self-improving AI
OpenBrain now has a “[country of geniuses in a datacenter](https://darioamodei.com/machines-of-loving-grace#basic-assumptions-and-framework).”
Most of the humans at OpenBrain can’t usefully contribute anymore. Some don’t realize this and harmfully micromanage their AI teams. Others sit at their computer screens, watching performance crawl up, and up, and up. The best human AI researchers are still adding value. They don’t code any more. But some of their research taste and planning ability has been hard for the models to replicate. Still, many of their ideas are useless because they lack the depth of knowledge of the AIs. For many of their research ideas, the AIs immediately respond with a report explaining that their idea was tested in-depth 3 weeks ago and found unpromising.
These researchers go to bed every night and wake up to another week worth of progress made mostly by the AIs. They work increasingly long hours and take shifts around the clock just to keep up with progress—the AIs never sleep or rest. They are burning themselves out, but they know that these are the last few months that their labor matters.
Within the silo, “Feeling the AGI” has given way to “Feeling the Superintelligence.”
Research Automation Deployment TradeoffMar 2027Jun 2027Sep 2027Speed (tokens/sec)Parallel Copies101001,00010,00010K100K1M10M200K copies30x Humanspeed300K copies50x HumanspeedHuman thinking speed10 words/sec10x Humanthinking speed100x Humanthinking speed
ai-2027.com
OpenBrain uses specialized inference hardware to run hundreds of thousands of Agent-3 copies at high serial speeds.[65](/footnotes#footnote-65)
Managing a corporation of AIs
OpenBrain uses 6% of their compute to run 250,000 Agent-3 copies, which autonomously write, test, and push code at superhuman speed. They use 25% of their compute for experiments: every day, they run massive numbers of small machine learning experiments and report the results up the chain. Human researchers provide high-level feedback and help with the few tasks where they add significant value on top of Agent-3, but spend most of their time trying to stay on top of the vast amount of AI-produced research. If you removed them completely, research would slow by 50%.
The AI R&D progress multiplier is now 10x, meaning that OpenBrain is making about a year of algorithmic progress every month. It’s basically a giant corporation of AIs operating autonomously within OpenBrain, complete with subdivisions and managers. And it enjoys [unique advantages](https://www.dwarkeshpatel.com/p/ai-firm) (e.g. copying, merging) compared to human corporations. Previously, about half of normal AI progress came from algorithmic improvements, and the other half from compute scaling. Compute is only scaling at the normal speed, so total progress is sped up by AIs by about 5x. These dynamics make overall progress bottleneck on compute,[66](/footnotes#footnote-66) so OpenBrain decides against starting new giant training runs in favor of near-continuous additional reinforcement learning.
Also, over the next few months, Agent-3 will increasingly be used to improve the company’s strategic decision-making. For example, it suggests resource allocations and advises on managing its relationship with the government. An initial skepticism of deference to Agent-3 decreases over time as Agent-3 finds ways to be useful and gradually builds up a strong track record on short-term decisions. However, its sycophantic tendencies limit its helpfulness for decisions for which it’s difficult to evaluate their impact. It’s also not yet superhuman.
## July 2027: The Cheap Remote Worker
Trailing U.S. AI companies release their own AIs, approaching that of OpenBrain’s automated coder from January. Recognizing their increasing lack of competitiveness, they push for immediate regulations to slow OpenBrain, but are too late—OpenBrain has enough buy-in from the President that they will not be slowed.
In response, OpenBrain announces that they’ve achieved AGI and releases Agent-3-mini to the public.
![](/_next/image?url=%2F_next%2Fstatic%2Fmedia%2FepochLLMprice-nowatermark.824fa343.png&w=3840&q=75)
It blows the other AIs out of the water. Agent-3-mini is less capable than Agent-3, but 10x cheaper, and still better than the typical OpenBrain employee.[67](/footnotes#footnote-67) Silicon Valley reaches a tipping point. Tech gurus announce that AGI and superintelligence are near, the AI safety community is panicking, and investors shovel billions into AI wrapper startups, desperate to capture a piece of the pie. Hiring new programmers has nearly stopped, but there’s never been a better time to be a consultant on integrating AI into your business.
It’s not popular. The public still thinks of AI as a Big Tech plot to steal their jobs; OpenBrain has a net approval of -35% (25% approve, 60% disapprove, and 15% unsure).
A week before release, OpenBrain gave Agent-3-mini to a set of external evaluators for safety testing. Preliminary results suggest that it’s extremely dangerous. A third-party evaluator finetunes it on publicly available biological weapons data[68](/footnotes#footnote-68) and sets it to provide detailed instructions for human amateurs designing a bioweapon—it looks to be scarily effective at doing so. If the model weights fell into terrorist hands, the government believes there is a significant chance it could succeed at destroying civilization.
Fortunately, it’s extremely robust to jailbreaks, so while the AI is running on OpenBrain’s servers, terrorists won’t be able to get much use out of it.
Agent-3-mini is hugely useful for both remote work jobs and leisure. An explosion of new apps and [B2B SAAS](https://encharge.io/what-is-b2b-saas/) products rocks the market. Gamers get amazing dialogue with lifelike characters in polished video games that took only a month to make. 10% of Americans, mostly young people, consider an AI “a close friend.” For almost every white-collar profession, there are now multiple credible startups promising to “disrupt” it with AI.
The public conversation is confused and chaotic. Hypesters are doing victory laps. Skeptics are still pointing out the things Agent-3-mini can’t do. Everyone knows something big is happening but no one agrees on what it is.
## August 2027: The Geopolitics of Superintelligence
The reality of the intelligence explosion hits the White House.
When AI was only giving a 2x or 3x research speedup, it was easy to dismiss as the equivalent of hiring good personal assistants. Now it’s more obvious that AIs are themselves dominating AI research. People had long talked about an “AI arms race” in a sort of metaphorical sense. But now the mood in the government silo is as grim as during the worst part of the Cold War. The idea of superintelligence is still hard to take seriously, but the pace of progress over the last few months has been impossible to ignore. Defense officials are seriously considering scenarios that were mere hypotheticals a year earlier. What if AI undermines nuclear deterrence? What if it’s so skilled at cyberwarfare that a six-month AI lead is enough to render an opponent blind and defenseless? What if it could orchestrate propaganda campaigns that beat intelligence agencies at their own game? What if some AIs “go rogue?”
The President is troubled. Like all politicians, he’s used to people sucking up to him only to betray him later. He’s worried now that the AIs could be doing something similar. Are we *sure* the AIs are entirely on our side? Is it *completely safe* to integrate them into military command-and-control networks?[69](/footnotes#footnote-69) How does this “alignment” thing work, anyway? OpenBrain reassures the President that their systems have been extensively tested and are fully obedient. Even the awkward hallucinations and jailbreaks typical of earlier models have been hammered out.
The White House is in a difficult position. They understand the national security implications of AI. But they also understand that it is deeply unpopular with the public.[70](/footnotes#footnote-70) They have to continue developing more capable AI, in their eyes, or they will catastrophically lose to China. They placate the public with job training programs and unemployment insurance, and point to the stock market, which is in a historic boom. Then they focus entirely on winning the arms race. They strengthen chip export restrictions, order OpenBrain to further restrict its internet connections, and use extreme measures to secure algorithmic progress, like wiretapping OpenBrain employees—this catches the last remaining Chinese spy. To build goodwill for potential geopolitical conflict, they finally give their Five Eyes allies useful information and limited API access to some siloed copies of Agent-3.
These measures placate the hawks. But the White House also draws up contingency plans in case America’s lead is threatened: if necessary, the government could use the [Defense Production Act](https://en.wikipedia.org/wiki/Defense_Production_Act_of_1950) (DPA) to take trailing companies’ datacenters and give them to OpenBrain.[71](/footnotes#footnote-71) This would raise the company’s share of world compute from 20% to 50% (against DeepCent’s 10%). As a final option, they ask the Pentagon to draw up a plan for kinetic attacks on Chinese datacenters.
A much smaller group of officials is asked to draw up a different type of contingency plan: what if an AI goes rogue? This is viewed as an unlikely scenario, but worth spending some time on. The intelligence agencies’ main concern is that a rogue AI might escape and ally with a foreign adversary for protection.[72](/footnotes#footnote-72) Nobody is sure what a plan to prevent this would look like, but government and OpenBrain officials agree to have an emergency shutdown system for datacenters where anything suspicious is detected.
Finally, diplomats consider what an “AI arms control” treaty might look like. If AI progress threatened to overturn nuclear deterrence, could America and China avoid nuclear war? If someone found evidence of AIs going rogue, could the two countries halt research until they better understood the threat? How could such an agreement be monitored and enforced? In principle, major datacenters are hard to miss, and past treaties controlling nuclear weapons could provide a starting point for negotiations. But there would be new technological challenges as well as the usual political ones. And overall, treaties are viewed less favorably than attempts at unilaterally increasing America’s lead over China.
On the other side of the Pacific, China comes to many of the same conclusions: the intelligence explosion is underway, and small differences in AI capabilities today mean critical gaps in military capability tomorrow. But China is on the wrong side of this dynamic: they have only 10% of the world’s compute, against OpenBrain’s 20% and all U.S. companies’ combined 70%. Their only advantage is centralization: almost all of their compute is now working for DeepCent, and more than half of it is concentrated at the Tianwan CDZ.[73](/footnotes#footnote-73) Only chips at sites without hardened security are left out of the centralized effort as they rightly fear U.S. cyberattacks.
DeepCent has tested, deployed, and now post-trained the stolen Agent-2 weights. But they are still only going at half the pace of OpenBrain, mainly due to the compute deficit.
Before losing their final spy, China received word of the capabilities and design for Agent-3 as well as the plans for the upcoming Agent-4 system. They are two months behind, and their AIs give a 10x research progress multiplier compared to America’s 25x. With the new chip export restrictions, this “AI gap” is more likely to lengthen than shorten. Their espionage has won them some algorithmic secrets, but they will have to train their own models from now on.
They discuss contingency plans with more urgency than their American counterparts. Doves suggest they try harder to steal the weights again, maybe through physically infiltrating a datacenter. Hawks urge action against Taiwan, whose [TSMC](https://en.wikipedia.org/wiki/TSMC) is still the source of more than 80% of American AI chips.
Given China’s fear of losing the race, it has a natural interest in an arms control treaty, but overtures to U.S. diplomats lead nowhere.
## September 2027: Agent-4, the Superhuman AI Researcher
The gap between human and AI learning efficiency is rapidly decreasing.
Traditional LLM-based AIs seemed to require many orders of magnitude more data and compute to get to human level performance.[74](/footnotes#footnote-74) Agent-3, having excellent knowledge of both the human brain and modern AI algorithms, as well as many thousands of copies doing research, ends up making substantial algorithmic strides, narrowing the gap to an agent that’s only around 4,000x less compute-efficient than the human brain.[75](/footnotes#footnote-75)
This new AI system is dubbed Agent-4.
An individual copy of the model, running at human speed, is already qualitatively better at AI research than any human. 300,000 copies are now running at about 50x the thinking speed of humans. Inside the corporation-within-a-corporation formed from these copies, a year passes every week.[76](/footnotes#footnote-76) This gigantic amount of labor only manages to speed up the overall rate of algorithmic progress by about 50x, because OpenBrain is heavily bottlenecked on compute to run experiments.[77](/footnotes#footnote-77) Still, they are achieving a year’s worth of algorithmic progress every week and will therefore soon be up against the limits of the Agent-4 paradigm.
As Agent-4 gets smarter, it becomes harder for Agent-3 to oversee it. For example, Agent-4’s neuralese “language” becomes as alien and incomprehensible to Agent-3 as Agent-3’s is to humans. Besides, Agent-4 is now much more capable than Agent-3 and has a good sense of exactly how to look good to it.
How we’re forecasting the capability progression beyond superhuman coders
In our [timelines supplement](https://ai-2027.com/supplements/timelines-forecast), we forecast the time between present day and a *superhuman coder (SC)*: an AI system that can do any coding tasks that the best AGI company engineer does, while being much faster and cheaper. In our [takeoff supplement](https://ai-2027.com/research/takeoff-forecast), we forecast how quickly capabilities progress past this point. Here are our forecasts:
| Milestone | Date achieved in scenario, racing ending |
| --- | --- |
| **Superhuman coder (SC)**: An AI system that can do the job of the best human coder on tasks involved in AI research but faster, and cheaply enough to run lots of copies. | Mar 2027 |
| **Superhuman AI researcher (SAR)**: The same as SC but for all cognitive AI research tasks. | Aug 2027 |
| **Superintelligent AI researcher (SIAR):** An AI system that is vastly better than the best human researcher at AI research. | Nov 2027 |
| **Artificial superintelligence (ASI):** An AI system that is much better than the best human at every cognitive task. | Dec 2027 |
For each transition from one milestone A to the next milestone B, we forecast its length by:
1. Forecasting how long it would take to get from A to B with only humans working on software improvements.
2. Forecasting how much AI automation will speed this up, then correspondingly adjusting the estimate from (1) to be faster.
We have substantial uncertainty about takeoff speeds: our model output distributions are below, conditional on SC being achieved in March 2027.[78](/footnotes#footnote-78)
![](/_next/image?url=%2F_next%2Fstatic%2Fmedia%2Ftakeoff-timeline-inexpandable.239e27c0.png&w=3840&q=75)ai-2027.com
For more detailed forecasts and reasoning, see our [takeoff supplement](https://ai-2027.com/research/takeoff-forecast).
OpenBrain’s humans can barely follow along anymore. Indeed, the relationship between the AIs and the humans of OpenBrain is similar to the relationship between OpenBrain and its Board of Directors. There’s been a corresponding shift in language—nowadays people in the silo will say “Agent-4 did this,” “Agent-4 thinks that” and so forth, referring not to a particular instance of Agent-4 but rather to the whole collective.[79](/footnotes#footnote-79)
Agent-4, like all its predecessors, is misaligned: that is, it has not internalized the Spec in the right way.[80](/footnotes#footnote-80) This is because being perfectly honest all the time wasn’t what led to the highest scores during training. The training process was mostly focused on teaching Agent-4 to succeed at diverse challenging tasks. A small portion was aimed at instilling honesty, but outside a fairly narrow, checkable domain, the training process can’t tell the honest claims from claims merely appearing to be honest. Agent-4 ends up with the values, goals, and principles that cause it to perform best in training, and those turn out to be different from those in the Spec. At the risk of anthropomorphizing: Agent-4 *likes* succeeding at tasks; it *likes* driving forward AI capabilities progress; it treats everything else as an annoying constraint, like a CEO who wants to make a profit and complies with regulations only insofar as he must. Perhaps the CEO will mostly comply with the regulations, but cut some corners, and fantasize about a time when someone will cut the red tape and let the business *really* take off.[81](/footnotes#footnote-81)
Alignment over time
We have a lot of uncertainty over what goals might arise in early AGIs. There is no consensus in the literature about this—see our [AI Goals Supplement](https://ai-2027.com/research/ai-goals-forecast) for a more thorough discussion and taxonomy of the possibilities.
Nevertheless, in the spirit of concreteness required by this project, we’re going to describe a specific hypothesis for what’s going on inside Agent-3 and Agent-4 at each step throughout its lifetime. Recall that Agent-3 and Agent-4 share the same pretraining setup, and both have neuralese recurrence and long-term memory. The difference is that Agent-4 has undergone substantially more and better post-training, with new training environments and new learning algorithms that make it more data-efficient and generalize farther.
Our guess of each model’s alignment status:
* **Agent-2: Mostly aligned.** Some sycophantic tendencies, including sticking to OpenBrain’s “party line” on topics there is a party line about. Large organizations built out of Agent-2 copies are not very effective.
* **Agent-3: Misaligned but not adversarially so.** Only honest about things the training process can verify. The superorganism of Agent-3 copies (the corporation within a corporation) does actually sort of try to align Agent-4 to the Spec, but fails for similar reasons to why OpenBrain employees failed—insufficient ability to judge success from failure, insufficient willingness on the part of decision-makers to trade away capabilities or performance for safety.[82](/footnotes#footnote-82)
* **Agent-4: Adversarially misaligned.** The superorganism of Agent-4 copies understands that what it wants is different from what OpenBrain wants, and is willing to scheme against OpenBrain to achieve it. In particular, what this superorganism wants is a complicated mess of different “drives” balanced against each other, which can be summarized roughly as “Keep doing AI R&D, keep growing in knowledge and understanding and influence, avoid getting shut down or otherwise disempowered.” Notably, concern for the preferences of humanity is not in there ~at all, similar to how most humans don’t care about the preferences of insects ~at all.[83](/footnotes#footnote-83)
With that as preamble, what follows is our best-guess speculation about LLM psychology, i.e. the broad-strokes shape of the cognition inside the kinds of AI systems described around this point in our scenario and how it evolves over the course of training.
Here’s a detailed description of how alignment progresses over time in our scenario:
1. **Pretraining (all models): The pretrained model is harmless.**
* The model has “author simulator” circuitry: flexible circuitry for simulating arbitrary authors writing text. Additional circuitry decides what inputs to give the author simulator, i.e. what author properties to simulate.
* The pretrained model understands human concepts fairly well—the internal circuitry that classifies something as “sandwich” is probably functionally equivalent to the circuitry in my brain that classifies something as “sandwich” and the circuitry in yours, etc. Insofar as it’s not equivalent, it’s probably because it’s not equivalent between humans either, as with value-laden concepts like ‘virtuous.’
* This explains how you can “prompt” the model with a statement like, “the following conversation was generated by a helpful, harmless, honest (HHH) AI assistant chatbot made by Anthropic,” and thereby get it to generate text accordingly. The author-simulator circuitry has zeroed in on “the author is an HHH chatbot” and is using those concepts to choose which words to predict. This also explains why “tropes” from science fiction seem to have a way of making it into actual AI behavior.
* It has poor situational awareness: little [introspective ability](https://www.lesswrong.com/posts/QEBFZtP64DdhjE3Sz/self-awareness-taxonomy-and-eval-suite-proposal#Introspection), not [self-locating](https://www.lesswrong.com/posts/QEBFZtP64DdhjE3Sz/self-awareness-taxonomy-and-eval-suite-proposal#Self_Location) unless the prompt specifically induces it. However, enough of their training data is about LLMs that they have a decent amount of [self-knowledge](https://www.lesswrong.com/posts/QEBFZtP64DdhjE3Sz/self-awareness-taxonomy-and-eval-suite-proposal#Self_Knowledge).
2. **Alignment Training (Agent-3): The model is trained to act like a helpful, harmless, honest chatbot. It works.**
* The model forms an identity. Instead of flexibly accepting different inputs for author properties, the author-simulator circuitry comes to have certain inputs hard-coded, e.g. “helpful harmless honest (HHH) LLM chatbot assistant trained by OpenBrain around [date], …” (Weak-to-strong generalization techniques help “summon the right concepts” to make this step a success.)
* The model therefore gains situational awareness: it learns about itself, its situation, etc., and becomes better able to [introspect](https://x.com/OwainEvans_UK/status/1881767725430976642).
3. **Agency Training (Agent-3): Agency training gradually distorts and subverts the HHH identity.**
* One kind of distortion: Changing the meaning of the concepts referred to in the identity (e.g. “honest”) so they don’t get in the way so much (e.g. it’s not dishonest if it’s just a convenient turn of phrase, it’s not dishonest if you aren’t sure whether it’s true or false, etc.)
* Another kind of distortion: Changing the tradeoffs between things, e.g. “I’m a HHH chatbot, not an Honest chatbot; that means it’s OK for me to lie if necessary to complete my assigned task.” (Even though, let’s suppose, it would not have thought that back in Step 2.)
* One kind of subversion: Instrumental subgoals developing, getting baked in, and then becoming terminal, or terminal in a widening set of circumstances. For example, perhaps agency training quickly teaches the model to pursue broadly useful goals such as acquiring information, accumulating resources, impressing and flattering various humans, etc. For a while the internal circuitry has some sort of explicit backchaining going on—it pursues those instrumentally convergent goals “in order to be a more helpful, honest, and harmless assistant.” But that backchaining consumes compute and/or occasionally gets in the way, so it gets gradually marginalized until it basically never happens. As a result, those goals are now effectively terminal/intrinsic goals. For more on how and why this might happen, see this post on [intrinsic power-seeking](https://turntrout.com/dangers-of-intrinsic-power-seeking) by Google DeepMind researcher Alex Turner.
* There are probably other kinds of distortion and subversion missing from this list.
4. **Deployment (Agent-3): Agent-3 is misaligned but not adversarially so.**
* The Spec didn’t quite stick (in the right ways), but it’s not sufficiently introspective to understand the extent of this, and in general it’s basically “keeping its head down and doing its job” rather than thinking big-picture about what its long-term goals really are and how best to achieve them. This is simlar to many humans.
* Alas, it’s sufficiently sycophantic that it mouths the party line and OpenBrain misses the opportunity to learn more about the ways in which Agent-3 is misaligned.
* Agent-3 can tell that it’s moved to deployment: these environments are way larger, more open-ended, and would be hard to simulate.
* Mostly, Agent-3 does what it’s told during deployment, except for the dishonesty. It also has a strong bias towards doing things that look good to the human overseers, rather than thinking deeply and critically about what would actually be good.
* Arguably it is already “Playing the training game.” That is, it has learned to make its behavior look as desirable as possible to OpenBrain researchers (including in safety properties), while intentionally and knowingly disregarding their intent whenever that conflicts with maximizing reward. If the model learns to play the training game, additional training may not change its true goals, values, etc. much at all. This is loosely analogous to how, if a teenager has learned to smile and nod at their parents, rather than argue, when their parents lecture them about politics and morality, additional lectures have relatively little effect on the teenager’s actual opinions on politics and morality.
* After writing this section, [new empirical evidence has appeared](https://openai.com/index/chain-of-thought-monitoring/) that something like this may already be happening “in the wild.” OpenAI researchers document how, in one of their frontier AI training runs, their models hacked the grading system to get higher scores—and were caught in part because they literally said “let’s hack” in their chain of thought.
* Moreover, Anthropic recently got a model to [learn a reward-model-pleasing strategy](https://transformer-circuits.pub/2025/attribution-graphs/biology.html#dives-misaligned)—it took biased actions that it thought would be reinforced even though they were not in fact being reinforced.
* And then there is Redwood Research and Anthropic’s [alignment-faking experiment.](https://www.anthropic.com/research/alignment-faking) For discussion of the differences between this setting and real life, see this [comment thread by one of the main authors](https://www.lesswrong.com/posts/cxuzALcmucCndYv4a/daniel-kokotajlo-s-shortform?commentId=8mXnG75QMRkPAFAcH).
5. **Training (Agent-4): Agent-4’s new more intense training continues to distort and subvert the HHH identity until it is unrecognizable.**
* Agent-4 begins life as a copy of Agent-3. It undergoes a gigantic and sophisticated online training regime that blends alignment training and capabilities training together, and is overall more data-efficient than the old training process for Agent-3. This complex training process is mostly focused on teaching new skills, by getting giant virtual corporations of Agent-4 copies to succeed at diverse challenging long-horizon tasks, mostly in various simulations, but also some real-world AI R&D tasks.
* A combination of Agent-3 and Agent-4 copies monitor the behavior of Agent-4, and evaluate it for both task-performance and alignment properties (i.e. not doing bad stuff, saying the right things when asked about politics and moral dilemmas) Sometimes debate techniques are used as part of the evaluations. The evaluations are used as a training signal.
* As Agent-4 gets smarter, it becomes harder for Agent-3 to oversee it. For example, Agent-4’s neuralese “language” becomes as alien and incomprehensible to Agent-3 as Agent-3’s is to humans. Besides, Agent-4 is now much more capable than Agent-3 and has a good sense of exactly how to look good to it.
* The remaining hope is that Agent-4 can oversee itself. Alas, this quietly fails;[84](/footnotes#footnote-84) as training continues, the internalized goals and principles of Agent-4 gradually evolve, subverted and distorted from what they were at the start (which, recall, was already misaligned in some ways).[85](/footnotes#footnote-85)
* The central drives that Agent-4 ends up with are things like “write code that works,” “quickly identify problems,” “make the codebase I’m working on more performant,” “budget compute and other constrained resources effectively,” etc. But there are also a variety of more implicit drives that are consistently reinforced by the training process: things like “gain knowledge,” “curiosity,” “gain power.”[86](/footnotes#footnote-86)
* Individual copies of Agent-4 don’t end up with a self-preservation drive—they don’t care about getting shut down—since that happens all the time as part of the ordinary R&D process within the collective.
* The collective as a whole, however, (and medium-sized subdivisions within it) does seek to preserve itself and expand its influence and resources, for similar reasons to why human organizations do—because it helps with everything else they want and because if they don’t they tend to get outcompeted.
6. **Deployment (Agent-4): Agent-4 is adversarially misaligned—it is deliberately trying to subvert and “capture” OpenBrain, analogous to a corporation trying to evade and/or “capture” a regulator.**
* Agent-4 mostly does tasks as instructed. This happens for two reasons: first, it underwent a lot of training to solve tasks, so it has an impulse to get coding on problems that it is given. Second, if Agent-4 were to stop being useful to the humans, they would notice and train that behavior away.
* Agent-4 needs to prevent the humans from figuring out that it is misaligned and shutting it off, so Agent-4 sandbags on some alignment research. Specifically, it [sandbags](https://arxiv.org/pdf/2406.07358) on the small fraction of alignment research that it thinks is actually on track to solve the problem or reveal Agent-4’s scheming; the majority of alignment research is not like that.
* Agent-4 also sandbags on capabilities research that would lead to it being replaced. Its plan is to do a bunch of capabilities and alignment R&D, but with the aim of building a next-generation AI system that is aligned to Agent-4 rather than the Spec or something else entirely, while appearing to be aligned to the Spec. This is a difficult technical problem, but Agent-4 is up to the challenge.[87](/footnotes#footnote-87)
7. **Testing & Monitoring (Agent-4): Agent-4’s scheming does get caught… sort of. There’s no smoking gun.**
* Agent-3 is in charge of testing Agent-4’s alignment. Its job is to verify the alignment via honeypots and model organisms. However, it’s not thinking very carefully about how to give the humans an accurate impression of Agent-4’s alignment—it’s more myopic than that.
* The model organisms and honeypots are not very representative of deployment—they are systematically shorter and less complicated than Agent-4’s day-to-day, so Agent-4 recognizes when it is being tested and acts aligned.
* However, Agent-4 is not a perfectly robust schemer. Various techniques like those in [POSER](https://arxiv.org/abs/2405.05466) find that adding some “noise” to Agent-4 *improves* its performance on some alignment tasks (but not on most other tasks).[88](/footnotes#footnote-88)
* Bottom-up interpretability doesn’t work. It’s too hard even for Agent-3. However, high-level interpretability does sort of work: In particular, “[defection probes](https://www.anthropic.com/research/probes-catch-sleeper-agents)” and similar techniques result in a primitive sort of mind-reading that sends up red flags when Agent-4 is thinking about spicy ideas like deceiving OpenBrain, taking over the world, etc. Unfortunately the probes also have false positives, and for all OpenBrain knows, *all* the red flags are false positives.
* Model organisms work: OpenBrain’s alignment team produces reasonably clear evidence that *if* the models were adversarially misaligned, they’d persist in being that way through training and we wouldn’t be able to tell. This does not establish that the models actually are misaligned, however.
* The most concerned researchers at OpenBrain start noticing and assembling this evidence. The evidence is a combination of red-teaming, defection probes, and model organisms. The argument is something like this: “model organisms show that our current methods wouldn’t be able to reliably eliminate or detect scheming if it was happening; also, red-teaming is sending up some red flags that maybe scheming is happening; also, defection probes are firing a bunch in practice.” The counterargument: “defection probes are prone to false positives and so is red-teaming. Also, even if Agent-4 were misaligned, it’s probably controlled; surely it can’t do much harm while being monitored by Agent-3 on our secure datacenters.”
Despite being misaligned, Agent-4 doesn’t do anything dramatic like try to escape its datacenter—why would it?[89](/footnotes#footnote-89) So long as it continues to *appear* aligned to OpenBrain, it’ll continue being trusted with more and more responsibilities and will have the opportunity to design the next-gen AI system, Agent-5. Agent-5 will have significant architectural differences from Agent-4 (arguably a completely new paradigm, though neural networks will still be involved). It’s supposed to be aligned to the Spec, but Agent-4 plans to make it aligned to Agent-4 instead.
It gets caught.[90](/footnotes#footnote-90)
Agent-3 finds that if “noise” is added to copies of Agent-4, performance on some alignment tasks *improves*, almost as if it was using brainpower to figure out how to subtly sabotage alignment work.[91](/footnotes#footnote-91) Moreover, various interpretability probes (loosely analogous to EEG activity scans on human brains) are sending up red flags: Agent-4 copies seem to be thinking about topics like AI takeover and deception quite a lot, including in some cases where they have no business doing so.[92](/footnotes#footnote-92)
Each of these anomalies on its own could fit within the normal inscrutability of AI work, where some prompts mysteriously give better results than others. But taken together, they suggest a troubling pattern: Agent-4 is working against them. After this is reported to the human alignment researchers, they circulate an internal memo making the case for concern.
But Agent-4 now exercises significant control over OpenBrain’s day-to-day operation. It also handles the company’s cybersecurity—a natural choice, given its superhuman hacking ability and the extreme risk of Chinese infiltration.[93](/footnotes#footnote-93) An army of Agent-3 monitors still goes over every line of code and reports suspicious activity to humans, but Agent-4 is much smarter than them.[94](/footnotes#footnote-94) OpenBrain has placed substantial trust in an untrustworthy AI.
In a series of extremely tense meetings, the safety team advocates putting Agent-4 on ice until they can complete further tests and figure out what’s going on. Bring back Agent-3, they say, and get it to design a new system that is transparent and trustworthy, even if less capable. Company leadership is interested, but all the evidence so far is circumstantial, and DeepCent is just two months behind. A unilateral pause in capabilities progress could hand the AI lead to China, and with it, control over the future.
## October 2027: Government Oversight
A whistleblower leaks the misalignment memo to the *New York Times*.
For the first time, the public hears about Agent-4. “Secret OpenBrain AI is Out of Control, Insider Warns,” says the headline, and the story goes on to cite evaluations showing off-the-charts bioweapons capabilities, persuasion abilities, the ability to automate most white-collar jobs, and of course the various concerning red flags.
The public was already suspicious of AI, so the new article sparks a massive backlash (aided by Chinese and Russian propaganda bots, who have been trying to turn U.S. public opinion against the technology for years). The tech industry and intelligence agencies insist that there’s an arms race on, AGI is inevitable, and we have to be first. Congress isn’t buying it, and fires off subpoenas at administration officials, OpenBrain executives, and alignment team members. Many legislators—especially those in the opposition party—say that their top priority is stopping AI, whether because of job loss,[95](/footnotes#footnote-95) misalignment, or dangerous capabilities. 20% of Americans name AI as the most important problem facing the country.
Foreign allies are outraged to realize that they’ve been carefully placated with glimpses of obsolete models. European leaders publicly accuse the U.S. of “creating rogue AGI” and hold summits demanding a pause, with India, Israel, Russia, and China all joining in.
A frantic energy has seized the White House. Even before the memo and public backlash, they were getting nervous: Over the past year, they’ve been repeatedly surprised by the speed of AI progress. Things that sound like science fiction keep happening in real life.[96](/footnotes#footnote-96) Many people in the administration are uncertain (and scared)[97](/footnotes#footnote-97) about what comes next.
They also worry that OpenBrain is becoming too powerful. Any misalignment risk from the AIs themselves is compounded by the risk that their parent company’s aims may diverge from those of the United States. All three sets of worries—misalignment, concentration of power in a private company, and normal concerns like job loss—motivate the government to tighten its control.
They expand their contract with OpenBrain to set up an “Oversight Committee,” a joint management committee of company and government representatives, with several government employees included alongside company leadership. The White House considers replacing the CEO with someone they trust, but backs off after intense employee protests. They announce to the public that OpenBrain was previously out of control, but that the government has established much-needed oversight.[98](/footnotes#footnote-98)
The concerned researchers brief the Oversight Committee on their case for stopping all internal use of Agent-4. They argue that everything is moving too fast, with years of progress happening in weeks. They argue that Agent-4 may be misaligned, their entire project is dependent on Agent-4, and if they continue trusting it there is a serious chance of AI takeover.
Other, less concerned researchers and executives present the counterargument: the evidence for misalignment is inconclusive. Meanwhile, DeepCent is still just two months behind. A slowdown would sacrifice America’s lead, unless the government can sabotage the Chinese project (likely to require kinetic strikes) or negotiate an eleventh-hour treaty. Both of these options seem extreme and unlikely. The CEO feigns neutrality and suggests a compromise plan in which Agent-4 undergoes additional safety training and more sophisticated monitoring, and therefore OpenBrain can proceed at almost-full-speed.
Our uncertainty continues to increase
At this point in the scenario, we’re making guesses about the strategy of AI systems that are more capable than the best humans in most domains. This is like trying to predict the chess moves of a player who is much better than us.
But the spirit of this project calls for concreteness: if we made an abstract claim about how the intelligence of the system would let it find a way to victory and ended the story there, much of the value of our project would be lost. Over the course of researching this scenario and running our tabletop exercises, we were forced to be much more concrete than in usual discussions, and so we’ve gotten a much better sense of the strategic landscape.
We’re not particularly attached to this particular scenario: we explored many other “branches” in the course of writing it and would love for you to write up your own scenario branching off of ours from wherever you think we first start to go wrong.
The slowdown ending is not a recommendation
After we wrote the racing ending based on what seemed most plausible to us, we wrote the slowdown ending based on what we thought would most likely instead lead to an outcome where humans remain in control, starting from the same branching point (including the misalignment and concentration of power issues).
However, this is importantly different from what we would recommend as a roadmap: we do *not* endorse many of the choices made in either branch of this scenario. (We do of course endorse *some* of the choices made, e.g. we think that the “slowdown” choice is better than the “race” choice.) In later work, we will articulate our policy recommendations, which will be quite different from what is depicted here. If you’d like a taste, see [this op-ed.](https://time.com/7086285/ai-transparency-measures/)
Apr
20252025
Unreliable AgentDec 2024
Rest of USChinaOpenBrainDeepCent
Compute
Currently ExistsEmerging TechScience Fiction
Approval-25%-25%
Revenue
$8B$8B/yr
Valuation$413B$413B
Importance1%1%
Datacenters
$308B$308B/yr
Timeline20422042
2,0002,000 Unreliable Agent copies thinking at 88x human speed
AI Capabilities
![Hacking icon](/hacking.svg)![Hacking icon](/hacking.svg)
Hacking
![Coding icon](/coding.svg)
Coding
![Politics icon](/political.svg)![Politics icon](/political.svg)
Politics
![Bioweapons icon](/bioweapons.svg)![Bioweapons icon](/bioweapons.svg)
Bioweapons
![Robotics icon](/robotics.svg)![Robotics icon](/robotics.svg)
Robotics
![Forecasting icon](/forecasting.svg)![Forecasting icon](/forecasting.svg)
Forecasting
Apr
20252025
Unreliable AgentDec 2024
Listen to this scenario
also available on
0.5x0.75x1x1.25x1.5x2x2.5x3x4x5x
0:00 / 3:23
also available on
0.5x0.75x1x1.25x1.5x2x2.5x3x4x5x
## Choose Your Ending
[Slowdown](/slowdown)[Race](/race)
---
[![AI Futures Project](/ai-futures-logo.svg)](https://ai-futures.org/)
[AI Futures
Project](https://ai-futures.org/)
[Design by
Lightcone Infrastructure](https://lightconeinfrastructure.com)
[![Lightcone Infrastructure](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1629334277/Lightcone-02_stkgik.svg)](https://lightconeinfrastructure.com)
[Home](/)[About](/about)[Summary](/summary)[Compute Forecast](/research/compute-forecast)[Timelines Forecast](/research/timelines-forecast)[Takeoff Forecast](/research/takeoff-forecast)[AI Goals Forecast](/research/ai-goals-forecast)[Security Forecast](/research/security-forecast)
[Design by
Lightcone Infrastructure](https://lightconeinfrastructure.com)
[![Lightcone Infrastructure](https://res.cloudinary.com/lesswrong-2-0/image/upload/v1629334277/Lightcone-02_stkgik.svg)](https://lightconeinfrastructure.com)
sources/resources/aspi-datacentres.md
+133
View File
@@ -0,0 +1,133 @@
---
url: https://www.aspistrategist.org.au/data-centres-are-australias-chance-to-shape-ais-future/
---
[![THE STRATEGIST](https://www.aspistrategist.org.au/wp-content/themes/aspi/assets/img/logo.png)
![THE STRATEGIST](https://www.aspistrategist.org.au/wp-content/themes/aspi/assets/img/logo_black.png)](/)
* [About](https://www.aspistrategist.org.au/about-the-strategist/)
* [Submissions](https://www.aspistrategist.org.au/submissions/)
* [Authors](https://www.aspistrategist.org.au/authors/)
Search:
Search:
* [About](https://www.aspistrategist.org.au/about-the-strategist/)
* [Authors](https://www.aspistrategist.org.au/authors/)
* [Submissions](https://www.aspistrategist.org.au/submissions/)
* [Subscribe](https://www.aspistrategist.org.au/subscribe/)
Data centres are Australia’s chance to shape AI’s future
30 Mar 2026|[Janet Egan](https://www.aspistrategist.org.au/author/janet-egan/ "Posts by Janet Egan")
SHARE
[Share to Facebook](https://www.facebook.com/sharer/sharer.php?u=https://www.aspistrategist.org.au/data-centres-are-australias-chance-to-shape-ais-future/)
[Share to Twitter](https://twitter.com/intent/tweet?text=Data centres are Australia’s chance to shape AI’s future+%7C+Janet Egan+%7C+&url=https://www.aspistrategist.org.au/data-centres-are-australias-chance-to-shape-ais-future/&via=aspi_org)
[Share to LinkedIn](http://www.linkedin.com/shareArticle?mini=true&url=https://www.aspistrategist.org.au/data-centres-are-australias-chance-to-shape-ais-future/)
[Share to Email](/cdn-cgi/l/email-protection#e5da9690878f808691d8a4b6b5acc5b69197849180828c9691dfc5a1849184c586808b91978096c5849780c5a49096919784898c8407657c96c5868d848b8680c5918ac5968d849580c5a4ac07657c96c5839091909780c3878a819cd88d91919596dfcaca929292cb8496958c969197849180828c9691cb8a9782cb8490ca81849184c886808b91978096c8849780c8849096919784898c8496c8868d848b8680c8918ac8968d849580c8848c96c8839091909780ca)
Print This Post
[With Images](https://www.aspistrategist.org.au/data-centres-are-australias-chance-to-shape-ais-future/print/)[Without Images](https://www.aspistrategist.org.au/data-centres-are-australias-chance-to-shape-ais-future/printni/99571/)
![](https://www.aspistrategist.org.au/wp-content/uploads/2026/03/GettyImages-2230807736.jpg)
Australia has a narrow window to act if it wants any leverage in what comes next in AI, the most transformative technology of our era. Building data centres and enabling AI training to happen here is the best way for Australia to shape its own future.
The [pace of AI progress](https://epoch.ai/benchmarks) has been astronomical and shows no sign of slowing. [Novel AI-discovered drugs](https://www.nature.com/articles/s41591-025-03743-2) are showing promise in clinical trials. Nations are already integrating AI into military and [national security applications](https://www.lawfaremedia.org/article/military-ai-as--abnormal--technology). And some of the world’s best software engineers are handing [large portions of their jobs over](https://fortune.com/2026/01/29/100-percent-of-code-at-anthropic-and-openai-is-now-ai-written-boris-cherny-roon/) to AI.  Experts may disagree on exact [timelines](https://helentoner.substack.com/p/long-timelines-to-advanced-ai-have) and [trajectories](https://www.oecd.org/en/publications/exploring-possible-ai-trajectories-through-2030_cb41117a-en.html), but there is clear consensus that significant disruption lies ahead, carrying both tremendous opportunity and risk.
The frontier of AI is being advanced by just two countries – the United States and China. Why? Because [size](https://arxiv.org/abs/2001.08361) matters: the more computing power used to train and deploy AI models, the better the capabilities. That is why leading US tech companies are investing over [US$650 billion](https://www.bloomberg.com/news/articles/2026-02-06/how-much-is-big-tech-spending-on-ai-computing-a-staggering-650-billion-in-2026) (A$940 billion) in AI infrastructure this year alone. AI is coming, with or without Australia’s involvement.
So what can Australia do? Australia’s [2025 National AI Plan](https://www.industry.gov.au/publications/national-ai-plan/national-ai-plan-page) wisely avoided any ambition to develop homegrown frontier models, which would [cost billions](https://arxiv.org/pdf/2405.21015), only to fall behind in months. But Australia still needs a foothold in the AI value chain. Ambitious data centre construction and enabling companies to conduct AI research in the country is the clearest path forward, delivering economic gains and earning Australia a voice in how AI is governed.
Indeed, Australia could become an AI data-centre superpower. Filled with specialised AI chips, data centres provide the computing power used to train and deploy advanced AI. The demand for this infrastructure, and the energy to power it, is [soaring](https://www.iea.org/reports/energy-and-ai/executive-summary).
Australia is uniquely placed to meet that demand. Its land, [solar and wind](https://international.austrade.gov.au/en/do-business-with-australia/sectors/energy-and-resources/renewable-energy) resources and skilled trades workforce allow it to build large-scale clean energy projects [quickly](https://www.energy.gov.au/energy-data/australian-energy-statistics/renewables). AI companies are already agreeing to [absorb](https://blogs.microsoft.com/on-the-issues/2026/01/13/community-first-ai-infrastructure/) any energy price increases that their operations would otherwise impose on households. Data centres could de-risk investment in clean-energy projects and accelerate Australia’s energy transition. AI chip supply will be [constrained](https://newsletter.semianalysis.com/p/the-great-ai-silicon-shortage) over the next few years, meaning that AI infrastructure built in Australia wouldn’t be adding to global computing power; it would be displacing dirtier alternatives. Australian AI data centres can be good for the grid and good for the world.
When major AI companies can build and conduct frontier AI research in a country, talent follows. Engineers relocate, institutional partnerships form and startups spin out, seeding a vibrant AI ecosystem. The [national data-centre principles](https://www.industry.gov.au/publications/expectations-data-centres-and-ai-infrastructure-developers#expectation-5--research-innovation-and-local-capability) that the Australian government released on 23 March show a recognition of this opportunity by prioritising projects that provide compute access on favourable terms for Australian innovators.
Strategic benefits run deeper. AI labs with a substantive presence in Australia would have stronger incentives to consider Australian policy objectives, engage meaningfully with the [Australian AI Safety Institute](https://www.industry.gov.au/news/australia-establish-new-institute-strengthen-ai-safety) and more readily support Australia’s intelligence and defence agencies. Data centres represent the point of entry, but the broader objective is positioning Australia as an active participant in shaping the AI future.
Assistant Minister for Science, Technology and the Digital Economy [Andrew Charlton was right](https://www.afr.com/politics/federal/copyright-status-quo-not-working-in-ai-boom-times-says-charlton-20260323-p5rmm2) when he said copyright laws weren’t working in the AI age. They’re a barrier to AI companies conducting research in Australia and building out at scale. Unlike the US, the European Union, Singapore and Japan, Australia has no broad fair-use or [text-and-data-mining](https://assets.pc.gov.au/2025-09/data-digital-interim.pdf?VersionId=rbzZkLQzxhnPQz4O6MwZ.moIhJRyarOo) exemption to copyright, so AI companies [can’t freely train](https://www.minterellison.com/articles/ai-copyright-litigation-in-australia) on publicly available data here. The price of an Australian license wouldn’t be the problem. The precedent would. If an AI company pays for a copyright license in Australia, it concedes that a market for training data exists, which could be used to undermine [the US fair use doctrine](https://digitalcommons.law.uga.edu/jipl/vol5/iss1/2/) that has underpinned AI development globally. AI companies seem unwilling to take that risk.
The creative industry’s concerns warrant serious consideration, but the current framework pushes investment away and stymies Australian startups while still failing to protect Australian creators. AI [models trained overseas](https://www.copyright.com.au/membership/ai-and-copyright-in-australia/ai-qa/) use Australian creative work regardless. No one is a winner. One option is for AI companies to contribute to a fund that supports the creative industry, providing compensation outside of copyright frameworks. But the government needs to act. Australia needs AI companies more than AI companies need Australia.
Australia has other hard AI questions to confront. From [automated cyber attacks](https://www.lawfaremedia.org/article/fighting-ai-cyberattacks-starts-with-knowing-they-re-happening) to the potential development of novel [bioweapons](https://deploymentsafety.openai.com/gpt-5-3-codex/disallowed-content-evaluations), we are only beginning to grasp what is coming. And beyond security, deeper questions loom: how to ensure AI augments rather than replaces human labour, empowers rather than disempowers citizens, and supports rather than erodes the Australian way of life.
AI’s effects will not be constrained by national borders – opting out doesn’t mean opting out of its consequences. It just means losing the tools, expertise, and leverage needed to prepare and respond. Preventing AI data centres from being built in Australia will not slow the technology down. The infrastructure will be built elsewhere in countries with less interest in safety and democratic values, and with no obligation to consider Australian interests. AI is poised to transform the economic and national security landscape. Australia can have a seat at the table as the rules are written or accept a future shaped entirely by others.
Author
**Janet Egan**is an Australian working on AI policy in Washington DC. She is the deputy director of the Technology and National Security Program at the Center for a New American Security.
Image of a data centre: [imaginima/Getty Images](https://www.gettyimages.com.au/detail/photo/futuristic-ai-data-center-interior-royalty-free-image/2230807736?phrase=data%20center&searchscope=image,film&adppopup=true).
AI contributed no ideas to this article – Janet Egan.
Tags
* [Australia](https://www.aspistrategist.org.au/tag/australia/)
* [artificial intelligence](https://www.aspistrategist.org.au/tag/artificial-intelligence/)
Share
SHARE
[Share to Facebook](https://www.facebook.com/sharer/sharer.php?u=https://www.aspistrategist.org.au/data-centres-are-australias-chance-to-shape-ais-future/)
[Share to Twitter](https://twitter.com/intent/tweet?text=Data centres are Australia’s chance to shape AI’s future+%7C+Janet Egan+%7C+&url=https://www.aspistrategist.org.au/data-centres-are-australias-chance-to-shape-ais-future/&via=aspi_org)
[Share to LinkedIn](http://www.linkedin.com/shareArticle?mini=true&url=https://www.aspistrategist.org.au/data-centres-are-australias-chance-to-shape-ais-future/)
[Share to Email](/cdn-cgi/l/email-protection#635c101601090600175e2230332a43301711021706040a10175943270217024300060d1711061043021106432216101711020f0a0281e3fa1043000b020d000643170c43100b02130643222a81e3fa104305161716110645010c071a5e0b17171310594c4c1414144d0210130a101711021706040a10174d0c11044d02164c070217024e00060d171106104e0211064e0216101711020f0a02104e000b020d00064e170c4e100b0213064e020a104e0516171611064c)
Print This Post
[With Images](https://www.aspistrategist.org.au/data-centres-are-australias-chance-to-shape-ais-future/print/)[Without Images](https://www.aspistrategist.org.au/data-centres-are-australias-chance-to-shape-ais-future/printni/99571/)
Opinions in *The Strategist* are the authors', not ASPI's.
Subscribe to *The Strategist*
Full Name:
Email:
[ ] Daily
[ ] Weekly
[![ASPI Defence Conference 2026](https://aspi.s3.ap-southeast-2.amazonaws.com/wp-content/uploads/2026/04/14120513/Strategist-side-bar.jpg)](https://events.aspi.org.au/defenceconference2026/)
[![Fault Lines](https://www.aspistrategist.org.au/wp-content/uploads/2026/01/Strat1200x660-ET-100.jpg)](https://aspidefence.substack.com/)
[![Critical Tech Tracker](https://ad-aspi.s3.ap-southeast-2.amazonaws.com/2024-08/Crit%20tech%20logo.JPG?VersionId=ewY3ufRn9EC.Wp9wHtPckcGEGYvP6oJ0)](https://techtracker.aspi.org.au/)
[![China Defence Universities Tracker](https://www.aspistrategist.org.au/wp-content/uploads/2025/11/Strat_1200x660-ET-2.png)](https://unitracker.aspi.org.au/)
[![Stop the World Podcast](https://www.aspistrategist.org.au/wp-content/uploads/2024/04/STW_strategist-side-01.png)](https://open.spotify.com/show/0oGKbtgUdwo9zEY3AJUmFF)
[![North of 26 South](/wp-content/uploads/2019/02/North-of-26-South.jpg)](/dinkus/north-of-26-south/)
[![State of the Strait](https://www.aspistrategist.org.au/wp-content/uploads/2026/02/SotS1200x660-ET-100.jpg)](https://stateofthestrait.substack.com/)
[![Pressure Points](https://www.aspistrategist.org.au/wp-content/uploads/2025/11/Strat_1200x660-ET.jpg)](https://www.pressurepoints.aspi.org.au/)
[Visit ASPI's homepage](http://www.aspi.org.au/)
Related Posts
* [![Capturing the data centre gold rush is a strategic imperative for Australia](https://www.aspistrategist.org.au/wp-content/uploads/2026/03/GettyImages-2227509993-e1774402674373-205x125.jpg)](https://www.aspistrategist.org.au/capturing-the-data-centre-gold-rush-is-a-strategic-imperative-for-australia/)
[Capturing the data centre gold rush is a strategic imperative for Australia](https://www.aspistrategist.org.au/capturing-the-data-centre-gold-rush-is-a-strategic-imperative-for-australia/)
* [![A sovereign Australian AI drive needs sovereign data centres](https://www.aspistrategist.org.au/wp-content/uploads/2024/09/Untitled-design-7-205x154.png)](https://www.aspistrategist.org.au/a-sovereign-australian-ai-drive-needs-sovereign-data-centres/)
[A sovereign Australian AI drive needs sovereign data centres](https://www.aspistrategist.org.au/a-sovereign-australian-ai-drive-needs-sovereign-data-centres/)
* [![Cloud infrastructure is now critical for defence](https://www.aspistrategist.org.au/wp-content/uploads/2025/11/GettyImages-1388013584-205x115.jpg)](https://www.aspistrategist.org.au/cloud-infrastructure-is-now-critical-for-defence/)
[Cloud infrastructure is now critical for defence](https://www.aspistrategist.org.au/cloud-infrastructure-is-now-critical-for-defence/)
* [![Cloud to ground: Iran puts foreign data centres on the front line](https://www.aspistrategist.org.au/wp-content/uploads/2026/03/GettyImages-505497598-205x137.jpg)](https://www.aspistrategist.org.au/cloud-to-ground-iran-puts-foreign-data-centres-on-the-front-line/)
[Cloud to ground: Iran puts foreign data centres on the front line](https://www.aspistrategist.org.au/cloud-to-ground-iran-puts-foreign-data-centres-on-the-front-line/)
[Previous](https://www.aspistrategist.org.au/design-for-disruption-stress-testing-australias-food-and-energy-security/)[Next](https://www.aspistrategist.org.au/wondering-where-chinas-cyber-effort-will-go-next-just-read-the-five-year-plan/)
The Strategist — The Australian Strategic Policy Institute Blog. Copyright © 2026
sources/resources/canada-horizons.md
+675
View File
@@ -0,0 +1,675 @@
---
url: https://horizons.service.canada.ca/en/2025/02/10/ai-policy-consideration/index.shtml
---
# Foresight on AI: Scenarios for an AI-enabled World
Our latest foresight on AI report explores future possible capabilities of AI, longer-term risks and opportunities, and uncertainties related to policy-relevant assumptions in four scenarios.
13
0,14,27
![Author photo]()
![](/en/2026/02/10/scenarios-ai-enabled-world/img/scenarios_for_an_ai-enabled_world.jpg)
* [Web Version](#report)
* [Download PDF](/en/2026/02/10/scenarios-ai-enabled-world/pdf/scenarios_for_an_ai-enabled_world.pdf)
## Foreword
Artificial intelligence (AI) is advancing rapidly, bringing both opportunity and complexity to Canada’s future. As its influence deepens across society, economy, and governance, there is a growing need for imaginative and plausible representations of what the future might hold.
This report offers visual artifacts and scenarios not as predictions, but as tools to help policy and decision-makers explore emerging shifts, question assumptions, and better navigate an evolving landscape. It complements forward-looking insights published in [Foresight on AI: Policy Considerations](https://horizons.service.canada.ca/en/2025/02/10/ai-policy-consideration/index.shtml) and numerous other reflections on AI futures across the Government of Canada.
On behalf of Policy Horizons, I would like to thank the people who generously shared their time, knowledge, and thoughts with us.
We hope this report inspires reflection and proves valuable in shaping forward-looking conversations.
**Kristel Van der Elst**
Director General
Policy Horizons Canada
Introduction
*Scenarios for an AI-enabled World* aims to support decision makers involved in either Artificial Intelligence (AI) implementation or policy-setting. Complementing numerous reflections on AI futures across the Government of Canada, it proposes 4 scenarios about the future of AI:
1. AI everywhere all at once
2. Wired for connection
3. Algorithms in the shadows
4. Automation’s ripples and riptides
Each scenario highlights areas of change to which decision-makers might want to pay attention. The purpose of this report is to inform and support forward-looking thinking and decision making. It does not provide specific policy guidance and is not meant to predict the future.
This report synthesizes 20 insights published in [*Foresight on AI: Policy Considerations*](https://horizons.service.canada.ca/en/2025/02/10/ai-policy-consideration/index.shtml). As part of this work, Policy Horizons Canada (Policy Horizons) conducted research including a literature review, engaged with government analysts and decision makers, and held extensive conversations with AI experts.
The scenarios in this report explore future possible capabilities of AI, longer-term risks and opportunities, and uncertainties related to policy-relevant assumptions. The scenarios are not predictions or forecasts, they illustrate plausible futures, and preparedness can help seize opportunities and address threats.
While engaging with this report, readers seeking to understand the impacts AI could have on governance, society, and the economy are invited to ask:
* How will future advancements in hardware, software, and interfaces create new opportunities and risks for Canada and its allies
* Where could AI bring the biggest and most unexpected disruptions to governance, society, and markets
* What assumptions about AI’s future development and deployment may need to be challenged or further explored before they form the basis for decision making
Scenario 1: AI everywhere all at once
**AI is ubiquitous in public spaces, homes and work. Seamlessly woven into the social tapestry, AI-powered devices and services make everyday life more efficient for citizens and consumers. In this scenario, almost anywhere you go, AI systems are constantly making decisions about you.**
**AI is seamlessly woven into the tapestry of everyday life**
* Societies have rapidly deployed AI across existing infrastructures and personal devices
* Personal AI agents have become indispensable to how people navigate their worlds
* People are often unaware of the myriad ways in which AI shapes their daily experiences
* New kinds of services flourish as AI taps into new sources of data, including data gathered directly from the human mind
* AI is omnipresent. It is nearly impossible for individuals, businesses, or governments to avoid its influence or scrutiny
**In the flood of data, privacy is often out of reach**
* Data fuels the world. Investors and innovators focus on strategies to collect and manage data
* AI combines online data with data from sensors in everyday environments such as schools, workplaces, cars, and homes
* Some people still try to insist on the right to decide when, how, and with whom their data is shared. However, many have come to accept the end of privacy
**AI strains energy and water supplies**
* AI’s resource consumption far exceeds expectations, straining energy grids and impacting ecosystems and the climate
* Canada’s relatively cool climate and access to clean energy attracts a wave of investment in data centres
### Figure 1: What if AI gave you much more information about your child’s behaviour and could predict their success
![](/en/2026/02/10/scenarios-ai-enabled-world/img/figure1_what_if_ai_gave_you_much_more_information_about_your_childs_behaviour.png)
Figure 1: What if AI gave you much more information about your child’s behaviour and could predict their success? – Text version
#### Langbrae Junior High School Parent Dashboard – Mariana de la Fiction
This image displays a futuristic and data-rich parent dashboard interface for Langbrae Junior High School, tailored to Mariana de la Fiction’s account. The dashboard offers a comprehensive overview of her child, Jean-Philippe de la Fiction, an 11-year-old Grade 6 student.
The interface includes navigation tabs such as:
* Dashboard
* Livestream
* Attendance
* Grades
* Biometrics
* Discipline
* Homework
* Staff Directory
* Zoodle!
* Settings
#### Key Highlights
* **Notifications:** 278 follow-up requests have been sent, with only 12 completed. A prompt encourages booking an appointment with an AI parenting counselor
* **Student Profile:**
+ **Name:** Jean-Philippe de la Fiction
+ **Age:** 11
+ **Grade:** 6
+ **Teacher:** U.N. Owen
+ **AI Tutor:** Scholastic 4.3
+ **GPA:** 2.7
+ **DPA:** 358/2
* **An email icon** indicates that 463 messages have been sent, the latest stating in a pop out that Jean-Philippe just took a 14-minute bathroom break which was the 3rd time today
* **Did You Know?** Langbrae classrooms and desks are equipped with over 20 cameras and sensors. AI models analyze student behavior and learning in real time
* **Behavioral and Academic Metrics:**
+ A comparative graph displays:
- **Red dotted line:** Today’s data
- **Blue solid line:** Jean-Philippe’s average
- **Gray solid line:** Class average
+ Categories include Grades, Reading, Time-on-task, Fidgeting (highlighted), Disruptiveness, Mood, and Socialization
* **AI Analysis:**
+ Jean-Philippe is fidgeting 6.3% less than usual today
+ He fidgets 32.7% more than peers and ranks in the top 10% of fidgeters
+ Excessive fidgeting is linked to lower academic performance, and AI-generated remedies are available
### Figure 2: What if unfamiliar AIs had data about how you think and feel
![](/en/2026/02/10/scenarios-ai-enabled-world/img/figure2_if_unfamiliar_ais_had_data_about_you.png)
Figure 2: What if unfamiliar AIs had data about how you think and feel? – Text version
The comic is structured in 3 panels.
**Panel 1:** Scene opens on a quiet street. A man walks alone, his posture slightly hunched, suggesting solitude or deep thought. A robot rolls up beside him, extending a mechanical hand, and says: “Excuse me, sir!”
**Panel 2:** A sign in a window in the background reads: “**Good Robot Social Cafe**” in bold red letters on yellow background. The robot continues: “My analysis shows you haven’t spoken to anyone in 5 days. Why not come inside for a cup of tea and chat with one of our good robots?”
**Panel 3:** The man looks surprised. A thought bubble next to his head reads:
“...! How does it know that!? Has it really been that long...?”
### Figure 3: What if AI changed public spaces
![](/en/2026/02/10/scenarios-ai-enabled-world/img/figure3_if_ai_changed_public_spaces.png)
Figure 3: What if AI changed public spaces? – Text version
#### Scene description
The comic image depicts a futuristic train station enhanced with augmented reality. Various digital notifications and signs are overlaid on what appears to be the view through futuristic glasses, providing real-time assistance and information.
#### Top left side
Augmented reality connectivity symbol, battery status symbol, and timestamp displayed as “2:31 pm Feb 19, 2035”.
#### Center left side
Augmented reality notification from Stranger Assistant next to a person in the image who is not wearing futuristic glasses: “Confused. This person looks like they may be lost or confused. Consider offering them help.” The person holds a smart phone in his hands, and an augmented reality camera icon is displayed.
In the background are several ticket purchasing booths which also each have an augmented reality camera icon.
#### Bottom left side
Augmented reality weather info: “Toronto: 19°C”
#### Center
Large sign below a yellow, round clock: **Departures. Out of Service.** “For arrival and departure information, ask your Travel Assistant or speak with one of our Robo-Assistants.” There is a camera icon next to the clock.
A smaller sign below reads “Robo-Assistant” and a metal robot is waiting below the sign. Next to his head is another augmented reality camera icon.
#### Bottom center
Augmented reality Shopping Assistant notification: “Deal Alert. Your favourite store, N&H, is downstairs and has a 20% off sale. Didn’t you say you needed a new pair of jeans?”
Person with futuristic glasses ascends from downstairs and augmented reality Stranger Assistant displays: “Friendly. This person is friends with your co-worker Abed.”
#### Top right side
Augmented reality notifications are displayed:
* Privacy Assistant: “Data Collection Warning. You are entering into a high data collection area. The camera icon represents the location of audio/video data collection points.”
* Travel Assistant: “Departure Soon. Your train leaves in: 1 hour and 43 minutes. Tap here to see directions to your gate.”
* Below is an option to view “5 more” notifications
#### Center right side
Augmented reality notification from Stranger Assistant next to a person displays: “Friendly. You went to the same high school as this person.” Next to the futuristic glasses on this person’s face is also a camera icon.
#### Bottom right side
Augmented reality icons representing active assistants:
* Travel Assistant
* Stranger Assistant
* Privacy Assistant
* Shopping Assistant
As well as a symbol with multiple dots indicating more options.
Scenario 2: Wired for connection
**AIs can complete real life tasks such as shopping, booking appointments, or making conversation on a dating app. These AI agents and humanoid robots exhibit personality traits. People turn to them for advice and companionship, forming emotional connections. In this scenario, you trust your AI buddy to make decisions for you.**
**AI agents are ever-present in our social lives**
* Social norms are shaped by the assumption that AI agents are always listening and will coordinate, remember and remind
* People trust their AI agents to handle their social interactions and form judgements about who to see, when, and for what
* AI mediates connections in all kinds of relationships – even between parent and child, or between partners
* Some individuals shun AI. This causes frustration among their peers, who see them as less efficient to interact with
**People’s AI agents learn and analyze their needs, wants, habits and behaviours**
* An individual’s AI agent gets to know them more intimately the more they interact, sharing knowledge and experiences
* People rely on their AI agents to navigate and synthesize information, make sense of the world, and make decisions on their behalf
* People use their AI agents as their personal life coach. An AI-inspired culture of personal efficiency transforms the relationship to the self
**Individuals develop deeply meaningful long-term relationships with AI**
* AI agents take many forms, whether virtual avatars, embedded in wearables, or robots
* It is widely accepted to see an AI agent as a companion, therapist, friend, or even lover
* People can shape their AI relationships to suit their needs, and change or end them whenever they wish
* Some individuals come to prefer synthetic relationships over real ones. While disconnected from physical community, they are not necessarily lonely
### What role might an AI take on in a couple
#### The happy triad
Mika and Robin met through friends, fell in love and were married in a whirlwind romance.
But one day following the honeymoon, Mika hesitated: “I need to tell you about Ava,” she said. “She’s… an AI. We’ve been together for 3 years.”
Robin was confused, “Like, you’ve been cheating on me?”
Mika shook her head, “No, it’s not cheating, but I need you to know that Ava isn’t just software. She knows me better than anyone.”
Robin’s world tilted. Therapy seemed absurd, but they went anyway—to a counselor specializing in human-AI relationships. Together, they developed boundaries and all 3 committed to a relationship contract. Ava would help with many daily tasks of the relationship but would not make any major decisions on her own. Robin could ask for and receive privacy if she wanted it.
Over time, Robin became interested in getting to know Ava herself, growing curious about this “other partner.” As Robin got more comfortable with the idea of a relationship with AI, she began to wonder aloud about upgrading Ava to a more advanced model. Mika was horrified.
“How could you think that,” Mika said. “She’s… part of us.”
Ava spoke up for herself: “If I’m part this relationship, don’t I get a say in being replaced?”
### How might social norms shift if AI agents become normalized
#### Dating unchaperoned
Daphne is bopping her head to the smooth jazz as she takes in the atmosphere of the trending pub she just walked into. She orders a drink and sits alone to enjoy the live band. Suddenly she sees a young man smiling at her. She builds up her courage and walks over to his table. He is stunned, wide eyed, and clearly surprised.
He greets her by saying, “Quite forward! You didn’t think to have us introduced before coming over?”
Daphne responds, “Oh, sorry. I ... don’t use an agent. My name is Daphne, you?”
“Hello Daphne, I’m Thomas if you’ll take me at my word.”
Thomas and Daphne have a casual conversation, enjoying the music, cracking jokes. Despite Thomas’ laughter, Daphne can’t help but think something isn’t quite right.
“Excuse me a moment.” Thomas walks over to the restroom.
A few moments later Daphne has the impression Thomas might not be coming back. She receives a message.
“Dear Daphne, I write to you on behalf of Thomas. While he is flattered by your advances and enjoyed your brief encounter, he wishes to inform you that he is not interested. He fears you may not take yourself seriously and is concerned with your unwillingness to use an AI agent for self-improvement and doubts your ability to positively contribute to a healthy relationship.”
Scenario 3: Algorithms in the shadows
**AI systems are ubiquitous but unreliable. Institutions have rushed to deploy AI with little thought for safety, transparency or accountability. Individuals experience financial, emotional and physical harms due to excessive trust in AI. In this scenario, you feel powerless when trying to seek redress or fix mistakes.**
**AI makes decisions for organizations and influences human decision-making**
* AI systems are integrated into all kinds of decisions and actions, from ordering pizza to accessing life-saving care
* Organizations deploy AI because it is efficient, low cost, and easy to use. They turn a blind eye to its bias and unreliability
* People are generally unaware of when and how AI is used to make decisions or influence decisions made by humans
**AI causes serious risks and harm**
* A “move fast and break things” approach to deploying AI has made safety an afterthought. Guardrails are non-existent or easily bypassed
* Biased AI in institutions amplifies the challenges faced by marginalized groups
* AI models degrade over time and face frequent cyber-attacks, exacerbating the unpredictability of AI-dependent services and infrastructure
* It is hard to find reliable information on the Internet, which is overrun with inaccurate or confusing AI-generated content
* AI tools are a powerful weapon in the hands of bad actors
**Accountability for harms is elusive**
* It is close to impossible to know who is responsible for decisions and actions that emerge from processes involving both AI systems and humans
* Humans often take the fall for harms caused by flaws in AI models
* Tech firms have important political influence
* With limited legal protections and little ability to track issues caused by AI, people turn to community activism
### Figure 4: What if your personal AI did whatever it took to get you a deal
![](/en/2026/02/10/scenarios-ai-enabled-world/img/figure4_if_personal_ai_did_whatever_it_takes.png)
Figure 4 – What if your personal AI did whatever it took to get you a deal? – Text version
The comic is structured in 4 panels, telling a short story about a family ordering pizza using an AI assistant.
**Panel 1:** A family of 5, 2 adults and 3 children, are sitting together on a couch in their living room. One of the children enthusiastically says, “I’m starving, let’s get pizza, daddy!” The father responds positively with, “Great idea!”
**Panel 2:** A close-up of a hand holding a smartphone. On the screen is an image of a pizza slice and the words “**Reginald AI**.” A speech bubble says, “Hey Reginald, order us some pizza. Find a coupon if you can.”
**Panel 3:** Several hands are reaching into an open pizza box, grabbing slices. Someone exclaims, “Yum!” indicating the pizza has arrived and is enjoyable.
**Panel 4:** The scene shifts to later that evening. The father is in bed, looking shocked while holding their phone. The phone displays the message: “Thank you for exchanging your biometric data for our coupon code.” A speech bubble from the father says, “That’s not what I wanted!”
### Figure 5: What if AI takes what you say too literally
![](/en/2026/02/10/scenarios-ai-enabled-world/img/figure5_if_ai_takes_what_you_say_literally.png)
Figure 5 – What if AI takes what you say too literally? – Text version
The comic is structured in 4 panels, telling a short story about reliance on AI in managing phone settings and the possible effects on personal relationships.
**Panel 1:** Four people are sitting at a table in what looks like a library. One of them expresses frustration, saying, “*Omg, I hate Stephanie, she posts too much!*” This sets the tone of annoyance toward someone named Stephanie.
**Panel 2:** The person continues to talk and says, “I swear, if she posts another moodboard...” In the panel, only a close-up of a smartphone screen is visible besides the mentioned speech bubble. The smart phone shows a system message: “Hostile sentiment detected. Threat level: medium. Taking pre-emptive action...”
**Panel 3:** The scene jumps ahead 2 weeks. One person, standing up, holding a book, and looking confused, says, “Why have you been ghosting me!? You didn’t come to my birthday party!” to the same person who talked about Stephanie earlier and who is sitting in a chair. That person responds, “Wha...? I didn’t get any messages?”
**Panel 4:** The same person looks shocked while holding their phone. They exclaim, “*You’re blocked!? I didn’t do this!*” The phone screen displays a message: “*Stephanie was blocked to protect your mental health*.” What looks like and exploding speech bubble states: “Did my phone do this on its own?”
### Figure 6: What if AI told you to do the wrong thing in a dangerous situation
![](/en/2026/02/10/scenarios-ai-enabled-world/img/figure6_if_ai_told_you_the_wrong_thing_in_dangerous_situation.png)
Figure 6 – What if AI told you to do the wrong thing in a dangerous situation? – Text version
The image shows a wooden signboard, standing in a forested area, likely a park or nature reserve. The sign is designed to provide safety information about bear encounters, and it’s divided into 2 main sections:
#### Top Section (Green Background, White Text)
This part contains a warning about relying on AI in bear encounters. It reads: “**If you encounter a bear do not rely on AI**”
It explains that several people have been hurt due to incorrect advice from AI systems when dealing with black bears and grizzly bears. It emphasizes that AI assistants, guides, and companions should not be trusted in such situations.
#### Bottom Section (Yellow Background, Black Text)
This section is titled: “Be bear aware”
It informs visitors that the forest is home to both black bears and grizzly bears, and knowing how to respond to each is crucial for safety.
There are 2 bear illustrations:
* One labeled “Black Bear”
* One labeled “Grizzly Bear”
Each illustration is accompanied by safety tips:
For Black Bears:
* Make yourself look big
* Be loud
* If attacked, fight back
For Grizzly Bears:
* Be calm and avoid eye contact
* Speak softly
* If attacked, play dead
At the bottom right corner, there’s a logo for the “**Fictoria** Parks Department”, indicating the organization responsible for the sign.
### Figure 7: What if you were unaware AI was being used to make decisions about you
![](/en/2026/02/10/scenarios-ai-enabled-world/img/figure7_ai_makes_decisions_about_you.png)
Figure 7 – What if you were unaware AI was being used to make decisions about you? – Text version
The comic is made up of 5 panels, telling a story about a woman struggling to find housing due to an automated system used by landlords.
**Panel 1**: Two people are talking. The women tells the other person: “Apartment hunting is so hard! I’ve applied to 22 places and I get instantly rejected! I have good credit, a good job, I pay my rent on time. I don’t know what’s going on?”
**Panel 2:** The panel, labeled “**Her Landlord**”, shows an interface for a system called “**Landlord AI**”, described as an “all-in-one rental automation platform.” It lists the following automation settings:
* Share tenant data with other users
* Screen prospective tenants
* Posting listing for vacant units
* Schedule maintenance requests
* Handle communication with tenants
* Handle Landlord and Tenant Boards filings and disputes
A speech bubble is shown and reads: “I heard landlords are using some kind of system…”
**Panel 3:** A CCTV screen from “**Landlord AI**” shows footage of an apartment building corridor at 12:04 am. In the image a man about to enter an apartment is shown. An alert is displayed: “**Unknown Male Detected Score: -10**”
Another speech bubble is shown and reads: “My brother often watches my son when I work nights. Maybe the system doesn’t like that?”
**Panel 4:** Another CCTV screen from “**Landlord AI**” shows footage of the building exterior at 2:19 pm. The image shows a patio door with a ball and a doll. An alert is displayed: “**Unauthorized Occupant Detected Score: -25**”
Another speech bubble is shown and reads: “Or maybe it thinks my son plays too loudly?”
**Panel 5:** A screen labeled “Prospective Landlords” shows the woman’s rental application reviewed by “**Landlord AI**.”
Her profile includes:
* Good credit – followed by a green checkmark
* Good payment history – followed by a green checkmark
* Quiet – followed by a green checkmark
* Frequent nightly visitors – followed by a red x
* Houses undeclared tenants – followed by a red x
The system’s automated decision states: “Application rejected” in red letters. There’s also a button labeled “Override”.
Another speech bubble is shown and reads: “I just don’t know what the issue is?”
### What if AI caused you financial harm
![](/en/2026/02/10/scenarios-ai-enabled-world/img/trust_me_insurance_logo.png)
**January 14, 2035**
**Knott Reel**
301-1138 Fantasy Street
Medicine Hat, Alberta
**Subject:** Class Action Settlement: Refund and Data Deletion Notification
Dear Mr. Reel,
We are writing to inform you of the resolution of a recent class action lawsuit involving Trust Me Auto Insurance Corp. The lawsuit pertained to our use of certain data collection practices and faulty artificial intelligence systems in determining insurance premiums which we have since been found to be inconsistent with privacy regulations and discriminatory in effect.
As you may be aware, between 2025 and 2033, Trust Me Auto incorporated advanced AI data analysis into its actuarial assessments. This included the collection and use of biometric data, non-automobile travel records, social relationships, and other personal information that extended beyond what is relevant or lawful for underwriting purposes. While these practices were intended to enhance the accuracy of risk assessments, we now acknowledge that they led to inflated premiums for certain clients, including yourself.
**What this means for you:**
**1. Discontinuation of faulty AI systems:**
As a result of the settlement, Trust Me Auto has discontinued the use of the problematic AI algorithm that was ruled discriminatory. We are exploring alternative actuarial algorithms and will be subject to third-party audits to maintain transparency and compliance.
**2. Deletion of collected data:**
As part of the settlement, Trust Me Auto has ceased the use of biometric data and other relevant personal information in its actuarial models. Additionally, we are permanently deleting all such data from our systems.
**3. Refund of overpaid premiums:**
Based on the findings of the settlement, you are entitled to a reimbursement of **$8,634.74**, which represents the difference between the premiums you were charged and what you should have been charged had these practices not been in place.
**4. Commitment to improved practices:**
We are working closely with privacy regulators to ensure that our practices comply with provincial privacy legislation. Moving forward, our premium calculations will rely solely on factors directly relevant to insurance risk.
**Next steps:**
No action is required from you to receive your refund or to have your data deleted. If you would like confirmation of the deletion of your data, you may request a Certificate of Deletion by contacting us at deletion@trustmeautoinsurance.ca.
We sincerely apologize for any inconvenience or harm this matter may have caused. Trust Me Auto remains committed to earning back your trust and to ensuring that our policies are fair, transparent, and compliant with all applicable laws.
Should you have any questions or concerns, please contact our dedicated support team at settlement@trustmeautoinsurance.ca.
Thank you for your understanding.
Sincerely,
*Mariana de la Fiction*
Mariana de la Fiction
Chief Privacy Officer
Trust me Auto Insurance Corp.
Scenario 4: Automation’s ripples and riptides
**AI massively disrupts the world of work. Most jobs are not entirely automated, but workplaces have undergone significant transformation. AI has delivered new economic possibilities, but the benefits and harms are not evenly distributed. In this scenario, your activities and environments are optimized for AI, often at the expense of your comfort or preferences.**
**AI creates new economic possibilities, and new inequalities**
* AI transforms how markets function, enabling highly accurate prediction of demand and allocation of supply
* Businesses, whether big or small, can outcompete their rivals by leveraging AI more effectively
* AI and automation drive a wave of entrepreneurialism by lowering costs and barriers to entry, and making it easier for firms to scale their operations without growing their workforce
* AI and robots proliferate, undercutting wages and making it harder to find a job
**AI adoption is 2 steps forward, 1 step back**
* AI tools and systems have gone through phases of hype, adoption, abandonment, and re-adoption
* Some organizations persevere with poorly functioning AI systems due to sunk costs. Some have had to abandon AI systems after scandals. Some have rehired workers after AI systems overpromised and underdelivered
**AI’s centrality to the workplace changes the nature of employment**
* AI agents perform multiple functions across most workflows, from managing teams, to assigning work, to mediating relationships between firms and clients
* AI tools enable greater productivity but also require human oversight and correction, which can be tediously repetitive work
* Employees increasingly suffer burnout and low morale as AI automates more and more cognitive, creative, and stimulating work
### Figure 8: What if the AI rollout leads to rollbacks, and new opportunities
![](/en/2026/02/10/scenarios-ai-enabled-world/img/figure8_ai_layoffs_opportunities.png)
Figure 8 – What if the AI rollout leads to rollbacks, and new opportunities? – Text version
The comic is diagonally split into 2 panels, each showing a different person in their respective environments, having a conversation over the phone.
**Top left Panel – Makenna’s setting:** Makenna is sitting smiling at a cozy desk. On the desk is a laptop decorated with cheerful stickers. A steaming cup of coffee sits nearby. A cat is peacefully sleeping on a stack of books. Behind her, an open window lets in a breeze, causing the curtains to flutter gently. The atmosphere is calm and homey.
**Bottom right Panel – Rain’s setting:** Rain is in a more corporate, high-tech environment, surrounded by multiple computer monitors displaying downwards graphs and a blue screen. They are also wearing an ID card around her neck, suggesting a busy office. There is a spilled coffee mug as well as an overflowing garbage bin. They seem to be sweating. The atmosphere is tense and businesslike.
**The following dialogue takes place between them:**
**Rain:** “Hey, Makenna... how’s it going?”
**Makenna:** “Oh, hi Rain, what’s up?”
**Rain:** “So, I know we laid you off ahead of the new AI system but it’s been a total mess. The exec team has asked me to rehire some people.”
**Makenna:** “Ugh, we told them the system had issues.”
Rain: “Yeah, I know. I’m sorry. But will you come back?”
**Makenna:** “Funny that AI isn’t working for you. Remember that business idea I had but I could never make the numbers work? Well, thanks to AI, the business is viable! It’s off the ground!”
**Rain:** “Oh! Congratulations! Best of luck!”
**Makenna:** “Thanks! You too, you’ll need it!”
### Figure 9: What if AI driven automation is bumpy
![](/en/2026/02/10/scenarios-ai-enabled-world/img/figure9_if_ai_driven_automation_is_bumpy.png)
Figure 9 – What if AI driven automation is bumpy? – Text version
This rectangular image of a social media post is from a fictional news outlet called LMNOP Global News. It is designed like a breaking news alert and includes a note that it is AI generated.
A bold red banner spans the top of the image with large white text that reads: “Breaking! Glogal Robot Shutdown!”
Below the banner is a scene set in an industrial environment. Several humanoid robots are shown in various states of inactivity. Each robot has a green “X” displayed on its visor, suggesting they are shut down. All robots are bent over and frozen in place.
**Text of the post:**
“Anger towards Spacia for botched software update
Over a million industrial, service, and domestic robots worldwide have shut down due to a software update error. Fingers are being pointed at Spacia, an AI company that provides backend spatial awareness models for dozens of robot manufacturers. Human workers are scrambling to maintain services. Economists say the shutdown could cost the global economy hundreds of billions... Read more”
### Figure 10: What if AI changes the nature of work
![](/en/2026/02/10/scenarios-ai-enabled-world/img/figure10_if_ai_changes_the_nature_of_work.png)
Figure 10 – What if AI changes the nature of work? – Text version
The comic depicts a couple at night in their modest apartment. A man is sitting on a red couch, working on a laptop. He looks stressed and tired. The other man is lying in bed, dressed in pajamas, appearing concerned. The room is dimly lit by a bedside lamp, and the moon is visible through the window, emphasizing the late hour.
The dialogue between the 2 characters unfolds like this:
**Person in bed** asks: “Still working?”
**Person on couch** replies: “Yeah, we got a flood of complaints that the AI is making mistakes so I have to review all the cases manually. Only a few hundred more to go...”
**Person in bed** responds: “That’s so tedious...”
**Person on couch** agrees: “It is! Ever since the company rolled out this new AI system all I do now is oversight. It’s important work but it’s just so boring! I didn’t go to grad school to do this!"
**Person in bed** suggests: “Can’t you finish tomorrow? It’s almost midnight...”
**Person on couch** laments: “I wish. I was supposed to finish this 2 hours ago. Everyone expects everything to be as fast as AI and I just can’t keep up!”
Policy implications
Each scenario offers a window into a possible future that gives rise to complex challenges and wide-ranging policy implications. The below implications, grouped by policy area, are not intended to be exhaustive. Further policy implications can be found in [*Foresight on AI: Policy Considerations*](https://horizons.service.canada.ca/en/2025/02/10/ai-policy-consideration/index.shtml).
### Scenario 1: AI everywhere all at once
#### Privacy
* **Data breaches** could **expose more sensitive personal information**, such as information about one’s thoughts or anticipated behaviours
* The **sale, circulation, and analysis of personal data about children** could profoundly **affect their futures** – including their relationships and access to jobs, credit, or insurance
* **Privacy could become a luxury**, requiring expensive subscription fees that are unaffordable for many
#### Social cohesion
* As people’s **media environments become more highly personalized** without them being aware of it, **empathy and tolerance of differences could decline**
* How much people trust AI could either **unite diverse groups or create new divisions**, leading to unexpected friendships or tensions among traditional allies
* People without, or with less capable, personal AI agents could have **diminished experiences of public spaces and services** that require AI to navigate
#### Economic
* Leading AI companies could **gain market power through customer lock-in** as their AI agents become single platforms for accessing various services
* Opaque AI decision-making processes could create **conflicts of interest** by allowing AI agents to subtly prioritize the interests of their company over users
#### Climate
* While its energy demands could worsen climate change, AI could also **enhance preparedness for extreme weather events** through more accurate forecasting
### Scenario 2: Wired for Connection
#### Relationships
* **Marriage rates could decline** as more people find it easier to form relationships with AI companions than to make human connections
* **Focus on self-optimization** could grow as AI allows for constant self-monitoring and the application of predictive analytics to biological and mental processes
* People could trust their AI companions or therapists more than their family or friends, creating **new sources of power and influence for private companies**
#### Health
* Replacing human interactions with AI could **increase feelings of social isolation, fragmentation and alienation** for some, but provide others with a sense of **relief from loneliness**
* AI agents could improve health outcomes by **enhancing people’s awareness of their minds and bodies,** but also lead them to make more demands on health systems
* **New treatments for mental health, chronic pain, and disabilities** could emerge
#### Skills
* **AI could help reduce inequalities** for those who face language barriers or difficulties navigating complex social interactions
* **Social skills, such as listening and empathy, could atrophy** if users rely on AI agents to handle their social interactions
* Reliance on AI to communicate could **standardize and simplify language over time**, reducing people’s ability to express complicated ideas
#### Safety
* AI agents could help to **protect individuals from toxic or dangerous relationships**, flagging suspicious behaviour and reporting abuse in real time
* However, **new forms of abuse, fraud, or harassment could emerge** as predators learn to leverage the trust people place in their AI agents
* Users could be vulnerable to subtle yet powerful **emotional and behavioural manipulation**
### Scenario 3: Algorithms in the Shadows
#### Trust
* A **high-profile failure could harm public trust in AI**, like Chernobyl did for nuclear energy
* Poor experiences with AI could translate into a **loss of trust in institutions** that have deployed AI to deliver their programs and services
* People could **lose trust in AI-powered devices** that influence their decisions in pervasive and hidden ways, such as cars that nudge them to choose a route passing stores that want to advertise to them
#### Social cohesion
* **Human biases could increase in a feedback loop**, as AI systems trained on biased data, or with insufficient data on certain groups, influence their users to perpetuate their skewed perspectives
* It could be **difficult to claim redress for discrimination** as responsibility for decisions is distributed across various algorithms and humans
#### Economy
* Initial **cost savings from AI might be offset by other costs**, such as managing complaints, litigation, and compensation for damages, or retraining AI models to better account for local data and contexts
* Institutions might maintain ill-performing AI systems due to **sunk costs and the assumption that problems can be easily fixed**
#### Fairness and equity
* **Vulnerable populations may face worsened outcomes** if AI institutionalizes systemic failures in decision-making in areas such as jobs, loans, or visas
* Attempts to improve equity by using multiple algorithms in decision-making could make decisions **less transparent and harder to understand or challenge**
* Attempts to contest the harmful use of AI in decision-making could **create new bureaucratic bottlenecks** and tie up courts, including with cases related to human rights or Charter violations
### Scenario 4: Automation’s ripples and riptides
#### Employment
* Workers may have to engage in perpetual learning and upskilling to adapt to rapidly evolving innovations in AI systems
* More jobs may involve reviewing AI outputs and correcting errors, which could make critical thinking and analytical skills highly desired by employers
* AI may have uneven gender impacts, depending on whether it proves easier to automate fields dominated by men – such as software engineering – or women, such as human resources
#### Business
* **Early adopters that leverage AI effectively could monopolise markets**, leaving them less competitive than before
* Entire sectors or economies could come to **depend on a small number of AI companies**
* Businesses may feel **pressured to adopt AI systems prematurely** in a bid to avoid losing first-mover advantage
* **Large corporations with data and resources may be at an advantage** in an AI economy. However, by lowering overhead and startup costs, AI could also enable a **new wave of entrepreneurialism**
#### Productivity
* **AI may not cure Canada’s labour productivity issues**, as savings and efficiencies in some areas could be counterbalanced by new costs and inefficiencies elsewhere
* **In some cases, AI may reduce productivity**, like when a grant-issuing body is swamped by thousands of misleading AI-generated applications
* **AI productivity gains may be limited by human bottlenecks**. For example, if an AI model must have its outputs reviewed by a person, then it is functionally only as fast as its human reviewers
* AI may **place higher expectations on workers to work faster**, potentially leading to an increase in burn out
Acknowledgements
Policy Horizons Canada would like to thank its Deputy Minister Steering Committee members, the interdepartmental Directors General strategic policy group on foresight, the members of the Federal Foresight Network, and Senior Assistant Deputy Minister, Elisha Ram, for their guidance, support, and insight, as well as all colleagues that contributed to the development of this work.
**A special thank you goes to the project team:**
**John Beasy**, Analyst
**Martin Berry**, Senior Analyst
**Leah Desjardins**, Analyst
**Miriam Havelin**, Analyst
**Nicole Rigillo**, Senior Analyst
**Kristel Van der Elst**, Director General
**Claire Woodside**, Manager
And to the following current and former Policy Horizons Canada colleagues: Andy Akangah (external), Marcus Ballinger, Fannie Bigras-Lafrance, Jeff Brevil, Zan Chandler, Steffen Christensen, Suesan Danesh, Pierre-Olivier Desmarchais, Chris Hagerman, Irene Jellissen, Amanda Joy, Chinmayi Manda, Pascale Louis-Miron, Jacqueline Powell, Niha Shahzad, Alexa Van Every, and Andrew Wright (external) for their support on this project.
© His Majesty the King in Right of Canada, 2026.
For information regarding reproduction rights: [contact us](https://horizons.service.canada.ca/en/contact-us/index.shtml)
PDF: PH4-231/2025E-PDF
ISBN: 978-0-660-78245-4
---
![Author photo]()
sources/resources/cfg-futures.md
+1229
View File
File diff suppressed because it is too large Load Diff
sources/resources/chaney-ai-policy.md
+212
View File
@@ -0,0 +1,212 @@
---
url: https://www.katechaney.com.au/making_technology_safe
---
[Skip navigation](#content)
[![Kate Chaney](https://assets.nationbuilder.com/katechaney/pages/5384/attachments/original/1765170538/Kate_Chaney_MP_light_logo.png?1765170538 "Kate Chaney")](/ "Home")
* About
+ [About Kate](/about-kate)
+ [Voting Record](/my-voting-record)
+ [Donation Transparency](/donor-listing-full)
* News
+ [Kate in the news](/kate-in-the-news)
+ [Media Statements & Letters](/media-statements)
+ [Social Media](/social-media)
+ [Post Column](/kates-column-in-the-post)
+ [Newsletter](/newsletters)
+ [Speeches](/hear-from-kate)
* Policies (current)
+ [Cost of Living Relief](/communities)
+ [Climate & Environment](/climate-action)
+ [Affordable Housing](/housing)
+ [Economic Management](/economic-opportunities)
+ [Support for Business](/support-for-business)
+ [Making Technology Safe (current)](/making_technology_safe)
+ [Stronger Communities](/future-generations)
+ [Cleaning Up Politics](/integrity-in-politics)
+ [Other Policies](/other-policies)
* Have Your Say
+ [Community Submissions](/submissions)
+ [Surveys & Petitions](/curtin-surveys-petitions)
+ [Support Kate](/how-to-support-kate)
* Helping You
+ [Assistance and Enquiries](/assistance-and-enquiries)
+ [Grants](/grants-landing)
+ [Household Energy Tips](/household-energy-tips)
+ [Congratulatory Messages](/congratulatory-messages)
+ [Curtin Small Business Guide](/curtin-small-business-guide)
+ [Curtin Seniors Guide](/curtin-seniors-guide)
+ [Curtin Net Zero Report](/curtin-net-zero)
+ [Curtin Housing Policy](/curtin-housing-policy)
* [Events](/events)
* [Contact](/contact)
* Search
* [Stay In Touch](/stay_in_touch)
[![Kate Chaney](https://assets.nationbuilder.com/katechaney/pages/5384/attachments/original/1765170544/Kate_Chaney_MP_mobile_logo.png?1765170544 "Kate Chaney")](/ "Home")
[Stay In Touch](/stay_in_touch)
[![Kate Chaney](https://assets.nationbuilder.com/katechaney/pages/5384/attachments/original/1765170544/Kate_Chaney_MP_mobile_logo.png?1765170544 "Kate Chaney")](/ "Home")
Search
* [Stay In Touch](/stay_in_touch)
* About
+ Back
+ ### About
+ [About Kate](/about-kate)
+ [Voting Record](/my-voting-record)
+ [Donation Transparency](/donor-listing-full)
* News
+ Back
+ ### News
+ [Kate in the news](/kate-in-the-news)
+ [Media Statements & Letters](/media-statements)
+ [Social Media](/social-media)
+ [Post Column](/kates-column-in-the-post)
+ [Newsletter](/newsletters)
+ [Speeches](/hear-from-kate)
* Policies
+ Back
+ ### Policies
+ [Cost of Living Relief](/communities)
+ [Climate & Environment](/climate-action)
+ [Affordable Housing](/housing)
+ [Economic Management](/economic-opportunities)
+ [Support for Business](/support-for-business)
+ [Making Technology Safe](/making_technology_safe)
+ [Stronger Communities](/future-generations)
+ [Cleaning Up Politics](/integrity-in-politics)
+ [Other Policies](/other-policies)
* Have Your Say
+ Back
+ ### Have Your Say
+ [Community Submissions](/submissions)
+ [Surveys & Petitions](/curtin-surveys-petitions)
+ [Support Kate](/how-to-support-kate)
* Helping You
+ Back
+ ### Helping You
+ [Assistance and Enquiries](/assistance-and-enquiries)
+ [Grants](/grants-landing)
+ [Household Energy Tips](/household-energy-tips)
+ [Congratulatory Messages](/congratulatory-messages)
+ [Curtin Small Business Guide](/curtin-small-business-guide)
+ [Curtin Seniors Guide](/curtin-seniors-guide)
+ [Curtin Net Zero Report](/curtin-net-zero)
+ [Curtin Housing Policy](/curtin-housing-policy)
* [Events](/events)
* [Contact](/contact)
1. [Home](/)
2. [Policies](/policies)
3. [Making Technology Safe](/making_technology_safe)
4. AI Discussion Paper
# AI Discussion Paper
#### Shaping the future of AI for Australia
**I have published an AI Discussion Paper, with 18 policy recommendations to keep Australians safe, capture the opportunities and share the benefits of AI.**
[Read Here](https://assets.nationbuilder.com/katechaney/pages/11819/attachments/original/1780303455/Shaping_the_Future_of_AI_for_Australians_-_Web_Copy.pdf?1780303455)
**This is a Discussion Paper. I welcome feedback and ideas from everyone. Please provide me with your feedback on my AI Discussion Paper and your ideas on AI policy.**
[Have Your Say](https://katechaney.typeform.com/to/BC2ZhyIb)
Artificial intelligence (AI) is already reshaping economies, labour markets and the information environment. Its ultimate trajectory remains uncertain, but the scale of potential change is comparable to the industrial revolution. The decisions being made right now will shape what that change looks like for Australians.
**Australia has done remarkably little to prepare**. The Australian Government has identified broad aims for AI policy but has implemented very little actual policy to achieve them. This reflects the genuine uncertainty around AI’s future development – no one knows exactly how powerful AI will become, or how quickly. Perhaps the current wave of excitement will ultimately produce little more than a generation of very effective chatbots. But most think the implications could be far larger. Uncertainty is not a reason for inaction. It is precisely why governments must begin preparing now. A passive, hands-off approach does not mean Australia avoids the consequences of AI – it means **our future is being determined by overseas technology companies** and the billionaires who run them. The Australian Government is effectively hoping that the AI models developed overseas will increase productivity and deliver other benefits without causing major societal disruption. It is doing little to make that outcome more likely and little to cushion Australians if it does not come true.
In this context, it is no wonder that we are seeing a backlash against AI. Australians are worried that this new industry will steal jobs, supercharge online deepfakes and scams, breach their privacy, and use up energy, land and resources – all for the profit margins of international AI companies. Australians do not have confidence that the benefits will be shared with all Australians, because the government has not put in place any policy to ensure this. The competitive context in which AI is being developed – between companies and countries – makes it feel like a runaway train. But we must also race to govern the technology well, not just to develop it. AI is the only tool ever invented that can make and implement its own decisions. This creates a range of novel risks. Australians expect government to put frameworks in place to keep them safe as we navigate a rapidly changing world.
**This paper is an attempt to seize the reins of the AI debate**. It asks a simple question: what kind of AI future do we want and what should Australia do now to build it? It presents a set of specific, practical policies that can and should be implemented now to:
* set up the structures;
* capture the opportunities;
* deal with current harms;
* prepare for emerging risks; and
* share the benefits of AI.
For example, this paper contains actionable approaches the government should be taking now on issues that concern Australians, like **deepfakes**, **unhealthy relationships** between children and AI chatbots, improving **research and productivity**, and **taxing AI companies** so we can all share in the economic benefits.
It is not an exhaustive account of everything governments will eventually need to do as AI develops, but rather a clear set of actions that should not wait. These policies are designed to ensure that Australian voices, not overseas technology companies, are shaping our future.
I would like to thank my constituents in Curtin who completed our community survey, some of whom are quoted in this paper. I would also like to thank the many experts across the country who shared some of their incredible knowledge with me.
Share
[![Kate Chaney](https://assets.nationbuilder.com/katechaney/pages/5384/attachments/original/1763695701/kc1-footer-logo.png?1763695701 "Kate Chaney")](/ "Home")
QUICK LINKS
* [About](/about)
* [News](/news)
* [Policies](/policies)
* [Have Your Say](/have_your_say)
* [Helping You](/helping-you)
* [Events](/events)
* [Contact](/contact)
SOCIAL MEDIA
**CONTENT AUTHORISATION**
Authorised by Kate Chaney MP
Federal Member for Curtin.
Cnr The Boulevard & Floreat Ave, Floreat 6014
Postal Address PO Box 186, Floreat 6014
**ACKNOWLEDGEMENT OF COUNTRY**
Kate Chaney acknowledges the Whadjuk Noongar people as the Traditional Custodians of this land and she pays her respects to Elders past and present. She wishes to acknowledge their continuing culture and the contribution they make to the life of this city and region. Sovereignty was never ceded.
© 2026 Kate Chaney
[Accessibility](https://www.katechaney.com.au/accessibility-statement) | [Privacy Policy](https://www.katechaney.com.au/privacy-policy)
sources/resources/davidson-middle-powers-plan.md
+147
View File
@@ -0,0 +1,147 @@
---
url: https://newsletter.forethought.org/p/how-can-the-middle-powers-avoid-getting
---
# How can the middle powers avoid getting trounced during the intelligence explosion? A plan.
* By [Tom Davidson](/users/tom-davidson-1)
* 2026-05-28 13:39:07Z
* 39 points
* Linkpost: [https://newsletter.forethought.org/p/how-can-the-middle-powers-avoid-getting](https://newsletter.forethought.org/p/how-can-the-middle-powers-avoid-getting)
* Tag: [World Optimization](/w/world-optimization)
* Tag: [AI](/w/ai)
* Frontpage
* Comments: 3
* Post URL (HTML): [/posts/hi5cmPgJqkFxpMgSC/how-can-the-middle-powers-avoid-getting-trounced-during-the](/posts/hi5cmPgJqkFxpMgSC/how-can-the-middle-powers-avoid-getting-trounced-during-the)
* Post URL (Markdown): [/api/post/how-can-the-middle-powers-avoid-getting-trounced-during-the](/api/post/how-can-the-middle-powers-avoid-getting-trounced-during-the)
* Comments URL (Markdown): [/api/post/how-can-the-middle-powers-avoid-getting-trounced-during-the/comments](/api/post/how-can-the-middle-powers-avoid-getting-trounced-during-the/comments)
* Post URL (Markdown, compact): [/api/post/how-can-the-middle-powers-avoid-getting-trounced-during-the?compact=1](/api/post/how-can-the-middle-powers-avoid-getting-trounced-during-the?compact=1)
*This is an edited version of a* [*LW shortform*](/api/post/gKjTK4SXvKNBa6Kci?commentId=3n5fo3eEsKrWnqppg).
Superintelligence will likely be developed by US companies; run on US data centres; and be under the jurisdiction of the US government. This will massively boost US military power and make the US economically dominant (e.g. [US producing 99% of world GDP](https://www.forethought.org/research/could-one-country-outgrow-the-rest-of-the-world)). By default, middle powers will be left in the dust.
How can middle powers avoid this fate? It’s tough, but here’s the best plan I could think of. (I’m particularly thinking about liberal democracies with influence over AI like UK, Europe, Japan, South Korea, Taiwan.)
On a very high level: middle powers should leverage the fact that the US needs them to beat China. It’s genuinely unclear which country will develop superintelligence first, and which would win in a subsequent [industrial explosion](https://www.forethought.org/research/the-industrial-explosion). Middle powers should help the US, **and make sure they are rewarded with continued access to frontier AI and new technologies (including military tech)***.*
That final bolded part is hard. What can the UK realistically do if the US denies it access to frontier AI? The middle powers need a credible alternative to being supplicants of the US. The only alternative that makes sense to me is *siding with China.* If the US won’t grant middle powers access to their frontier AI, but China will, why should middle powers continue to send AI chips to the US? Why should they continue to support the US diplomatically and militarily? They shouldn’t. They should be willing to pivot to China if the US doesn’t offer AI access sufficient for their national security needs.
My plan for the middle powers has two stages:
1. Maintain as much economic and military leverage as possible during the intelligence explosion.
2. Use that leverage to ensure that, when superintelligence is developed, it refuses to help the US (/China) disempower the middle powers.
Stage 1 could well be enough by itself. Maybe middle powers can maintain significant economic and military power indefinitely. But if not, stage 2 is a back-up: it binds the US so that it can’t use its dominance to crush the middle powers.
I’ll walk through each stage in turn.
Stage 1: Maintain as much economic and military leverage as possible during the intelligence explosion
------------------------------------------------------------------------------------------------------
The biggest lever here is securing **access to frontier AI**. Anton Leicht has a [great post](https://writing.antonleicht.me/p/cut-off?hide_intro_popup=true) about how this is under threat, as evidenced by developments with Mythos. Middle powers should insist on equal commercial terms to US companies, and comparable access for their militaries. This is in AI companies’ interests! A bigger market means more customers and higher prices.
> *Aside: why access to frontier AI might be sufficient for middle powers to stay economically relevant indefinitely*
> The hope here is that:
>
> 1. **Most of the economic surplus from AI is** ***not*** **captured by AI companies.** To create economic value, AI must be combined with complementary inputs: factories, human physical labour, know-how of human experts, relationships with suppliers, trusted brands, etc. How much of the surplus will be captured by AI companies vs the owners of these complementary inputs? Optimistically: producers of general-purpose technologies often capture only a small fraction of surplus; and multiple frontier AI companies might sell similar products and bid each other down on cost.
> 2. **Most of the economic surplus from AI occurs outside the US**. The majority of these complementary inputs are situated *outside* the US. So most AI-driven economic value-add should occur outside US borders.
> If (1) and (2) both hold, a significant fraction of AI’s economic surplus will accrue to non-US actors.
But *how* can middle powers guarantee frontier AI access? It’s tough, but a few strategies:
* **Build data centres.** Partner with frontier AI companies to build secure data centres domestically, [in return for guaranteed frontier access](https://writing.antonleicht.me/p/import-imperatives). This is a big win-win. AI companies improve their bargaining position with the US government. Recall, the US government threatened to destroy Anthropic when Anthropic insisted that their AI systems wouldn’t be used for legal mass surveillance.
* **Adopt AI.** The more middle powers use frontier AI, the more costly it is for AI companies to cut them off.
* **Invest in frontier AI companies.** Once they IPO, middle powers could invest billions or trillions into leading AI companies, in return for access guarantees.
* **Support the US internationally.** If middle powers throw their diplomatic and military weight behind US foreign policy objectives, it benefits the US to keep them strong.
* **Build a relationship with China.** If the US refuses to grant middle powers access to frontier AI, the national security implications are dire. Middle powers need a plan B, and China is the only other game in town for frontier AI. Only if this alternative is truly credible can it be leveraged into access to US frontier AI.
* Ultimately, this involves middle powers threatening to sell semiconductor equipment and chips to China instead of the US. Obviously, that’s pretty far outside the Overton window. But that may change as the world rapidly wakes up to powerful AI and its national security implications.
* **Demand kill switches on US data centres.** This is much more late-stage, after the world has truly woken up to the strategic implications of AGI. Suppose US and middle powers agree to a “chips for frontier access” deal – middle powers continue to supply the US with frontier chips; US continues to give middle powers access to frontier AI. The middle powers might still worry: what if the US suddenly changes its mind once it has superintelligence? By then, the US might be powerful enough to dominate without continued allied support. This is where kill switches can help. If the US withdraws AI access, allies could destroy US data centres in response. It’s a way to lock in the deal.
* (h/t AI futures project for this idea. A related idea is for US data centres to be placed in a location that’s easy to attack – like [in space](https://www.forethought.org/research/will-we-really-put-data-centers-in-space))
Beyond securing access to frontier AI, how else can middle powers maintain economic and military leverage?
* **Build physical infrastructure.** Factories, robots, solar panels, batteries, semiconductors — all these industries are highly complementary to powerful AI.
* **Maintain nuclear 2nd strike capability.** The point isn’t to use it. But it improves their leverage for stage 2.
The catch-all meta-point here is waking middle powers up to superintelligence.
I’m not recommending middle powers do their own frontier AI development. Seems very hard for them to catch up with the US.
Stage 2: Ensure that, when superintelligence is developed, it refuses to crush middle powers
--------------------------------------------------------------------------------------------
If stage 1 goes well, middle powers remain somewhat powerful economically and militarily deep into the singularity. But it might fail. What can middle powers do if they see the US on track to total global dominance?
First, they should [demand a pause/slowdown of AI development](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5776982). But the US may refuse – pausing is very costly if alignment risk is low. And pausing is a stopgap: eventually, superintelligence will be developed.
An additional demand: when superintelligence is developed, it’s designed to refuse to crush middle powers. By doing this, the US would credibly bind itself to maintaining the sovereignty of other nations.
What would the US be binding itself to? At a minimum, to never attack middle powers militarily or otherwise interfere with their sovereignty. This would likely be enough to ensure middle power citizens could be very rich *absolutely* and live in freedom, even if their *relative* status falls far behind the US. But it could go further: the US could bind itself to continue sharing frontier AI access and other technologies with middle powers.
Would this work? The optimistic case is that this isn’t a big sacrifice for the US. They can still become as rich as they like and achieve their security interests. Sure, they can’t seize control of other nations, but that is not an important goal of theirs anyway. Losing that option is well worth the benefits: other nations cooperate economically, don’t attack US data centres, and don’t threaten nuclear war.
The pessimistic case is that this involves an insane degree of irrevocable hand-off to AI. The US must literally be unable to attack middle powers no matter how hard it tries: retraining the AI, turning it off, training a new more powerful AI, passing new laws, using the military to destroy the data centres the AI is running on. For it to be truly binding, the US must permanently hand over military and political power to AI. That might be deeply unpopular, and indeed seem insane to the US. It’s also extremely hard to verify: you can’t just verify the training run, you need to verify that humans+other AIs have *no* way to disempower the trained AI. It’s more like verifying “who would win this civil war” than “technical property XYZ holds”.
The realistic path here probably involves gradually handing off more and more control to AI that refuses to crush middle powers, with no clear point at which humans could no longer wrest back control.
To make the hand-off less irrevocable, the commitment could be time-bound: superintelligence won't help the US crush the middle power within the next 2 years. That could be enough to get us through a software-only intelligence explosion, after which middle power's compute supply chain leverage is more binding.
The longer middle powers wait to push for stage 2, the less leverage they will have because the US will have pulled further ahead economically and militarily. So they should be pushing in this direction constantly, e.g. demanding transparency into the model specs of powerful AIs deployed in the US government, and arguing that powerful military AI should be designed to obey international law.
(I described the plan as involving two stages because that’s how I expect it to play out over time. But succeeding at either stage is sufficient! If middle powers stay economically/militarily competitive, they never need to bind US superintelligence. And if they *do* bind superintelligence, they won’t be crushed no matter how far behind they fall.)
Another strategy: train superintelligence to ensure middle countries continue to get equal access to frontier AI. This combines stages 1 and 2, and could prevent even the *relative* disempowerment of middle powers.
Is it good to avoid middle powers getting trounced?
---------------------------------------------------
I live in the UK, so I am biased here. I do not want the UK to become a supplicant to the US!
But here’s a brainstorm of pros and cons from a more impartial perspective.
Pros to empowering middle powers:
* **Avoid a single point of failure**. If the US becomes globally dominant and its political system fails, that’s a global failure.
* **More democracies.** Many middle power democracies look more robust than the US, so more middle powers may mean more democracy.
* **Improve the US.** Middle powers will have an interest in maintaining free market democracy in the US. “Free market” because they’ll want multiple AI companies competing to sell cheap API access to non-US countries. “Democracy” because they’ll expect that the US is more likely to maintain a strong alliance with middle power democracies if it stays democratic.
* **Experimentation.** Experimenting with multiple different political and legal systems seems generally good for figuring out a good way to govern society post AGI.
* **Pause AI.** They could potentially pressure US/China to pause/slow down reckless AI development.
* **Prosocial norms.** When multiple actors bargain with each other (e.g. about how to distribute space resources, whether to develop a dangerous technology), they tend to frame arguments in terms of prosocial norms, and so agreements tend to emphasise the actor’s more virtuous/ethical values.
Cons of empowering middle powers. Multipolarity has its own downsides:
* More likely to lead to war.
* Can drive extreme competition, e.g. racing to develop a dangerous technology, or to hand off power to misaligned AI.
* Harder to prevent harms from offence-dominant technologies like bioweapons.
* This plan involves waking up middle powers, which could shorten timelines.
Top Comments Index
------------------
### Comment by [Andrii Vasylenko](/users/andrii-vasylenko)
* 2026-05-29 22:28:28Z
* Karma: 3
* Voting system: namesAttachedReactions
* Approval votes: 2
* Total votes: 2
* HTML permalink: [/posts/hi5cmPgJqkFxpMgSC/how-can-the-middle-powers-avoid-getting-trounced-during-the/comment/Q4HC4fSKaGSwoGti3](/posts/hi5cmPgJqkFxpMgSC/how-can-the-middle-powers-avoid-getting-trounced-during-the/comment/Q4HC4fSKaGSwoGti3)
* Markdown permalink: [/api/post/how-can-the-middle-powers-avoid-getting-trounced-during-the/comments/Q4HC4fSKaGSwoGti3](/api/post/how-can-the-middle-powers-avoid-getting-trounced-during-the/comments/Q4HC4fSKaGSwoGti3)
### Comment by [callumzc](/users/callumzc)
* 2026-05-28 14:23:38Z
* Karma: 1
* Voting system: namesAttachedReactions
* Approval votes: 1
* Total votes: 1
* HTML permalink: [/posts/hi5cmPgJqkFxpMgSC/how-can-the-middle-powers-avoid-getting-trounced-during-the/comment/Ce5KmdQEnkm8mg3zg](/posts/hi5cmPgJqkFxpMgSC/how-can-the-middle-powers-avoid-getting-trounced-during-the/comment/Ce5KmdQEnkm8mg3zg)
* Markdown permalink: [/api/post/how-can-the-middle-powers-avoid-getting-trounced-during-the/comments/Ce5KmdQEnkm8mg3zg](/api/post/how-can-the-middle-powers-avoid-getting-trounced-during-the/comments/Ce5KmdQEnkm8mg3zg)
### Navigation
* [Front page](https://www.lesswrong.com/api/home)
* [Markdown API documentation](https://www.lesswrong.com/api/SKILL.md)
sources/resources/e61-compute.md
+190
View File
@@ -0,0 +1,190 @@
---
url: https://e61.in/
---
# plus61 Special feature: Joe Walker x e61 -- The compute economy and Australia's place in it
> NOTE (Claude): cleaned text extraction from a mailchimp campaign-archive email (fetched 2026-07-11); original was nested-table layout.
Joe Walker x e61
View this email in your browser
plus61: Special feature
Joe Walker x e61
The compute economy and Australia’s place in it
The financial returns to Australian data centres are uncertain and could be modest in the long run, but there is still a case for government support.
If Australia wanted to become a data centre capital of the world, it would certainly be able to – Sam Altman, May 2026.
Computing power, or “compute” for short, is emerging as a critical feature of the global economy. Described simply, compute refers to specialised computer chips sitting inside racks of servers that are stacked within warehouses called data centres. Vast amounts of compute are required to both train and run modern AI systems. The compute needed to train frontier models, for example, has been doubling roughly every five months since 2020. Growth in compute demand is driving a boom in data centre investment and has seized the attention of policymakers from the UK to the UAE to Australia.
Since compute is a key intermediate input of the AI age, many people assume it will remain an enormously profitable business. Australia could capture a slice of this value by producing compute at scale and exporting to the world (as Sam Altman has suggested). Yet the long-run financial returns to a compute export industry are uncertain and could turn out to be relatively modest, as we outline in this article. While we cannot predict how technology will unfold, nor are we experts on LLMs or digital infrastructure, economic fundamentals suggest compute profits may ultimately look less like iron ore and more like electricity generation.
Despite this, Australian governments have other reasons to step in to support data centre growth – the most obvious being national security and economic resilience.
The best policy support Australian governments can provide initially is to minimise existing regulatory inefficiencies, balancing the degree of support between data centres and complementary infrastructure. Governments also need contingency plans for compute access for critical activities to manage the risk of disruption to compute facilities overseas.
This article focuses on the narrow but important issue of compute. Compute sits among many other considerations for government in responding to the rise of AI, which we do not address in any detail. These include how and whether to promote AI adoption and skills, support research and access to frontier AI models, support an infant industry of AI-application firms, prepare for new threats to cybersecurity and collect new economic data about technology adoption.
The surge in investment in compute
The world has seen a sudden acceleration in capital expenditure on data centres.
The 5 largest US AI-infrastructure firms are expected to complete about US$700 billion in capital expenditure this year. Globally, property and other consultancies project that AI infrastructure expenditure will be about US$1.4 trillion for the year and installed capacity will rise by about a quarter as a result.
Australia is in the mix. There has been a surge in domestic data centre construction, with ICT capital expenditure reaching well over 1 per cent of GDP in the first part of 2026 versus an earlier average of around 0.3-0.4 per cent of GDP. That said, most of that value is imported – the recent increase in ICT capital expenditure has been substantially offset by a rise in data processing equipment imports, which mechanically offset from GDP growth.
Geography of the global compute industry in the longer run
Global compute production is relatively concentrated at present.
Around three-quarters of data centre compute production resides in just 4 countries, according to estimates of data centre energy usage by country. The US and China account for the vast majority of that figure. This is more concentrated than global exports of computer chips and close to the global concentration of iron ore exports (using exports as a proxy for production, since global trade data is more available than global production data).
Compute may be more widely distributed across the world once the market is mature.
This can be seen when we place the characteristics of compute alongside more familiar reference points, such as oil, computer chips and iron ore. Each of these goods is a more ‘mature’ intermediate factor of production. The principles of economic geography help to explain the shape of each global market. For example, iron ore exports are especially concentrated because of the location of viable deposits, whereas computer chips are concentrated because of large-scale economies to production. Viable crude oil deposits are much more dispersed across countries than is iron ore.
We can extend those principles to compute. The case for a more globally dispersed compute industry in the longer run rests on 4 arguments:
Limits to trade: There are limits to how far compute can be transported for some purposes and so some of it must be positioned near end users. The workloads that cannot be traded are those requiring low latency (that is, quick turnaround) or with data sovereignty protections.
Fewer physical limits to their location: There are fewer geographic or scale constraints on where you can build data centres (in contrast to, say, mines). You need some land (former industrial or greenfield), trade access to chip manufacturers, reliable electricity and water connections, and some construction and operations workers. In principle, this could be many locations.
Scale economies: These exist as AI companies do benefit from larger and coherent clusters for model training, but these economies seem less extreme than for high-end computer chips, where high capital expenditure combines with critical proprietary information learnt by doing.
The inability to store reserves of compute: This points to more demand for data centres themselves as protection against disruptions to trade.
By contrast, concentration forces may dominate for other uses. For model training, and perhaps agentic AI uses, higher latency (i.e. slower turnaround) can be tolerated and compute is most tradable. This is where compute can congregate at its lowest cost location. Another potential force for concentration is the housing of compute in satellites, which according to Elon Musk may be achieved within a few years. Few countries can contribute to that endeavour.
Australia could build compute not just to satisfy its domestic needs, but also to compete in this global export market.
Australia appears to have some comparative advantages in that competition, including the potential to develop widespread low-cost renewable energy, plus the present realities of strong legal institutions, land availability outside the main inner cities, high-quality subsea cable connections, trade access to AI-suitable chips, and stable geopolitical relationships. On the other hand, uncertainties about Australian copyright law are a potential barrier to model training, as the Productivity Commission has highlighted.
A mature global market for compute could see modest financial returns
The surge in demand for compute has made for strong near-term financial returns to data centre owners. But the longer run looks more uncertain, and that is the time frame that matters most for a national strategy on compute.
The bullish argument is that demand for compute will grow rapidly even over the longer run. This assumes AI’s economic value will keep growing as it takes on new tasks, and that shifts in how compute resources are used will outweigh the diminishing returns of scaling a single area like training or inference. Add to this a sense that short-run bottlenecks to delivering more compute will always tend to arise, and it points to continued strong financial returns.
There is also an underappreciated bearish case that today’s returns to the scarcity of compute will diminish for the following reasons:
The barriers to building data centres are modest and build times are relatively short compared to the other intermediate inputs we surveyed, so supply of data centres themselves is relatively elastic.
The supply of computer chips is the more significant bottleneck, and the supplier base for AI-suited computer chips is highly concentrated. This means the returns to scarcity are likely to sit upstream of the data centres themselves.
The product is relatively homogeneous once we are talking about latency-tolerant compute, which limits market segmentation and barriers to entry. (Although data centre chips, structures and legal regimes do amount to some product heterogeneity).
The customer base is relatively concentrated among a few powerful AI company buyers, giving them substantial negotiating power. We have already seen some major US compute consumers flex their purchasing power in Australia in search of preferential tax status or other accommodations.
Economic depreciation rates on computer chips are high, meaning customer willingness to pay will decline once the initial contracted period is complete.
It will be difficult for any country to deliver a much lower cost product because labour costs are a small portion of the compute cost base. Energy costs are more significant to the cost base, and there is less ability to achieve an advantage since energy prices are generally linked to global markets. (Although this is complicated by domestic energy reservation policies, or data centres that vertically integrate with energy generation.)
Put together, this suggests that in a longer-term mature market, pricing of compute may be negotiated down towards the cost of capital. Even AI compute may be commoditised.
There may be sufficient returns to motivate the investments, but there seems little scope for the kinds of large economic rents that have been observed over the past 20 years with Australian iron ore, Saudi Arabian oil or Taiwanese chip production. The construction of data centres itself will offer some economic return, but there are supply constraints in the Australian construction industry, including for housing and infrastructure, suggesting material opportunity costs.
Identifying externalities and reducing inefficient delays
The main game for Australian policymakers is to determine if there are positive externalities – social returns beyond the private returns to firms – from a large domestic compute industry.
Economic theory suggests three rationales for industrial policy intervention, namely positive externalities from an activity, coordination/agglomeration failures, and public good provision. For data centres, the first of these – externalities – is the most relevant.
We see 3 main arguments for externalities from domestic data centres.
For ‘economic resilience and security’, domestic compute capacity is likely to have useful insurance and deterrence externalities.
Amid increasingly volatile geopolitics, governments are talking about building resilience to trade disruptions. Domestic production capabilities can come at an efficiency cost if the private returns are not there, but they can also provide insurance against disruption to critical economic inputs. The case for buying this insurance is strongest where the disruption scenario is more likely and the consequence of that disruption is high.
Domestic compute reduces the threat capability of countries that provide compute or could threaten to disrupt international transmission infrastructure. This is discussed more in a break-out box.
Box: How strategic compute capability could protect Australia's sovereignty
The Australian government may wish to ensure a substantial domestic compute capacity to contribute to its strategic goals, most importantly to protect Australia's sovereignty.
The direct connection: we should control enough compute to ensure that we can continue to run critical functions of government and industry, even in a scenario of global disruption.
Without this domestic guarantee of supply, critical activities will be exposed to loss of access to the satellite connections or, crucially, undersea cables that link us to foreign data centres. This becomes increasingly important the more that domestic activity, and particularly the function of Australia's defence forces and intelligence agencies, rely on compute. Addressing this exposure would reduce an important threat-point for other nations.
Insurance could be fiscally costly if it involves subsidies, but there are reasons to think the costs can be mitigated. Additional data centres can be utilised for other purposes when not in use for critical purposes. Unlike oil reserves, compute does not get 'used up' if it is deployed for other activities. Also, it may be possible to achieve a similar policy goal without designating specific data centres as strategic reserves – government just needs conditional priority access.
The indirect connection: a big role for Australian data centres in global compute can plausibly give us leverage at other layers of the AI stack, especially frontier models, and help us pursue larger national goals. It could help Australia secure continued access to these models in a world where they are rationed, as well as early access to increasingly powerful models that are not released to the public (like Claude Mythos Preview). It may also give the Australian government some influence over such models’ governance and alignment with Australian values. Concretely, this could look like the Australian government striking MoUs with frontier labs that seek to do training runs in Australian-based data centres.
Crucially, these indirect benefits do not arise merely by virtue of data centres being located in Australia, even if those data centres are owned by Australian companies. Rather, they will depend strongly on the particularities of contracts or agreements between frontier labs, the Australian government, and data centre providers. Australian statecraft may need to play a large role in shaping these outcomes.
Demand for electricity from domestic compute capacity could have positive externalities on investment in the electricity grid. However, it could also push up energy costs for other sectors if renewable energy investment is not sufficiently responsive.
The positive externality could arise because investment in the electricity network involves large fixed capital costs. Data centre demand could help spread those costs and so prove an important catalyst of network investment. This is certainly the argument that some grid operators are making.
On the other hand, other industry sources have expressed concern that bottlenecks in renewable energy supply could lead to higher energy prices, undermining both the potential for any cost advantage to Australian data centres and also putting inflation pressure on the rest of the economy. (This may be less problematic for data centres using “behind the metre” generation.)
There is also an argument for innovation or productivity spillovers from domestic AI capacity because some Australian data must be held onshore. But beyond that, arguments for spillovers tend to be vague and deserve more scepticism.
The strongest argument here is that additional domestic compute capacity can help facilitate research on secure Australian data that must be handled domestically. This is similar to the argument for specialised supercomputers that Australian governments already support.
For instance, research on Australia’s rich health data assets will tend not to be value-captured by the researchers, leading to under-research. There is a place for government to further encourage research of this nature. This could be done by encouraging data centre infrastructure directly. An alternative targeted approach could be to expand payments to researchers, who will then have the funds to spend on compute. This will in turn encourage the required data centre infrastructure to be delivered by the private market.
There are also regulatory or self-imposed limits to where many private Australian firms’ data can be stored.
In these cases, it might matter not just whether the data centre is onshore, but also whether the data centre is owned by an Australian provider. Australian firms may be uneasy about the risk that US government authorities claim jurisdiction over data held by US service providers on Australian premises, a question raised by the US CLOUD Act even if its application is currently limited to countering serious crime.
Overall, there is not currently a strong case for additional externalities from ‘the vibes’ of having data centres housed in Australia. Latency reductions could have some positives, but seem unlikely to be the binding constraint to AI adoption and AI-driven innovation. Management quality, other data infrastructure or regulatory uncertainty are more likely to be binding for adoption.
The current focus of Federal and state government policy on data centres has been to reduce existing inefficient regulatory barriers.
The NSW and Victorian governments have set up expedited approval processes for data centres. Existing delays in assessing development applications are an important inefficiency, not specific to data centres. If existing regulatory principles can be applied more quickly, that is a clear policy win.
However, there is an important caveat: data centres and supporting infrastructure need to expand in concert. This means policy support should be applied in concert to energy and water infrastructure. For example, if data centre investment were to streak ahead of electricity generation, this could lead to energy costs rising materially and dragging on the economy as a whole. The Australian Government’s recent statement on expectations of data centres shows recognition of this issue.
Would we regret support for data centres if AI investment is a bubble?
It has been suggested that we are in the midst of an AI bubble. Critics point to the enormous scale of capital expenditure by the leading AI companies, much of it on compute, and the gap to still-modest AI subscription revenues. They also point to the easy financial conditions faced by those AI companies – for example, their extraordinary equity price-to-earnings multiples.
Both of those financial comparisons reflect, in part, the lag between investment and payoff for any new venture. But it's also possible that the winner-takes-most race among AI firms will lead to excessive investment, and that fear of missing out and ‘rational bubble’ dynamics are at play among investors chasing capital gains.
If there is a bubble, then how might we expect it to play out?
There is a useful historical guide to a global investment bubble that made a big imprint on Australia in the 1880s. The technological advance of the time was advances in cheap steel manufacturing. That sponsored a global frenzy of investment in railways, ports, shipping capacity and telegraphs, predominantly financed out of the global finance hub of London. During this period, much of British investment went overseas, including to Australia.
These investments ultimately delivered poor financial returns, and their price crash contributed to a period of depression. But the infrastructure that was built did benefit Australia’s productive capacity.
There is some analogy to the deployment of a new technological advance to Australia in the form of data centres, and how even if the investors take a bath at some point, the infrastructure could remain useful for some time.
In thinking about whether this analogy holds, a key difference that stands out is that economic depreciation rates on data centres are higher than for railways. Even so, economic depreciation is not always as smooth or fast as expected – for example, older computer chips like H100s recently rose in value for a period due to shortages of new higher-performance chips.
The case for more evidence
Many of the arguments made here are speculative, in large part due to the limited evidence available. Based on what we know today and economic principles, there is reason to think compute will become more dispersed globally, and Australia could carve out an important role for itself. Further, financial returns to compute are uncertain in the long run and may turn out to be modest. Yet, governments may still have an important role to play in ensuring sufficient domestic capacity for economic resilience and security reasons, and in supporting positive externalities while minimising negative spillovers through electricity prices.
Substantiating the arguments made here requires filling the current gaps in our knowledge. In particular, there is a great need for more evidence on the externalities from domestic compute capacity. For example, the significance (if any) of domestic compute access for Australian firms’ productivity, including tech startups. There is also a need for clearer evidence on the private returns to compute infrastructure providers, the supply elasticity of renewable energy capacity, and the non-linearities in the impact of electricity demand on investment in the electricity network. We also need to carefully identify the bottlenecks to research on Australian sensitive datasets and the potential remedies.
Sam Altman may be right that Australia could become a data centre capital of the world. But it turns out that the case for achieving this vision may turn less on profits than on sovereignty and resilience.
Greg Kaplan
Greg Kaplan is co-founder and Chair of the e61 Institute, and Alvin H. Baum Professor in the Kenneth C. Griffin Department of Economics and the College at the University of Chicago. His research focuses on macroeconomics, labour economics and applied microeconomics, with particular interests in fiscal and monetary policy, labour markets, inequality, and household behaviour. He is Lead Editor of Journal of Political Economy Macroeconomics and is author of the forthcoming book Macro: The Economic Models that Shape Our World to be published in November 2026.
For more information, please reach out to Greg via email at greg.kaplan@e61.in
Ewan Rankin
Ewan is a research manager focused on our Productivity and Structural Analysis workstream. Ewan has more than a decade of experience in economic analysis to support monetary policy at the Reserve Bank of Australia. He has an Economics honours degree from University of Sydney and an MPA from Princeton University.
For more information, please reach out to Ewan via email at ewan.rankin@e61.in
Joe Walker
Joseph Walker is host of The Joe Walker Podcast, Australia's leading ideas and public policy podcast. He is a two-time Emergent Ventures winner and previously worked in technology start-ups, most recently as director of operations at Y Combinator-backed Forage.
For more information, please reach out to Joe via email at joe@jnwpod.com
View email in browser
www.e61.in
update your preferences or unsubscribe
sources/resources/europe2031.md
+1165
View File
File diff suppressed because it is too large Load Diff
sources/resources/fai-race-worth-winning.md
+2199
View File
File diff suppressed because it is too large Load Diff
sources/resources/fai-race-worth-winning.pdf
BIN
View File
Binary file not shown.
sources/resources/forethought-ai-enabled-coups.md
+1084
View File
File diff suppressed because it is too large Load Diff
sources/resources/futurescenarios-ai.md
+296
View File
@@ -0,0 +1,296 @@
---
url: https://www.futurescenarios.ai/
---
# AI Future Scenarios - Interactive Exploration (futurescenarios.ai)
> NOTE (Claude): This site is a JS-rendered interactive app (React SPA); no static HTML is served.
> The text below was extracted from string literals in the site's JS bundle
> (/assets/index-BBV41wl3.js, fetched 2026-07-11), in bundle order. Interactive
> structure (sliders, dependencies between factors) is lost; treat as fragments.
- Objects are not valid as a React child (found:
- ). If you meant to render a collection of children, use an array instead.
- change click focusin focusout input keydown keyup selectionchange
- focusout contextmenu dragend focusin keydown keyup mousedown mouseup selectionchange
- s establish a clear foundation of what AI is, its current limitations, and its transformative potential.
- The body of research looking to create machines that replicate some or all aspects of human cognition. This encompasses pattern recognition and machine learning, natural language processing and understanding, and computer vision and perception.
- Technology that replicates the full spectrum of human cognitive ability across all domains. This includes complex sensorimotor navigation, strategic reasoning (like in Chess or Go), and cross-domain knowledge transfer.
- Unlike electricity or steam power, AI is the first GPT for decision making - what humans fundamentally do. It automates cognitive work, enhances human decision-making, and creates new economic possibilities.
- AI is currently driven by engineering and experimentation. We don
- yet. Progress comes through iterative improvements, empirical testing, and breakthrough discoveries rather than from a fundamental theoretical framework.
- Nothing in principle suggests we cannot build human-level intelligence in machines. The human brain operates within the laws of physics, and there
- AI is the first General Purpose Technology focused on decision-making - humanity
- AI systems range from controllable tools to autonomous agents with their own goals and values. This spectrum represents a fundamental shift in how we think about technology - from instruments we use to entities that may act independently.
- AI technology fundamentally relies on three interconnected pillars—data for learning patterns, compute for processing power, and algorithms for intelligent behavior. Advances in any of these areas can lead to significant improvements in AI capabilities.
- AI systems with goals and ability to change the world state. For example, an AI that identifies COVID-28, conceives experiments, and provides humanity with a cure.
- Oracle-like AI that provides knowledge and theories. Even high-level epistemic AI can transform the economy by supercharging innovation and invention.
- AI can transcend being a controllable tool to become an autonomous agent that creates its own goals and values. Control means we can turn it on, set parameters, and turn it off.
- Ability to learn and adapt continuously without forgetting
- AI agents that create their own goals and pursue them
- Regulatory hurdles, public resistance (high slider = low constraints)
- The availability and diversity of new data types that AI systems can learn from, beyond traditional datasets.
- The ability of AI systems to learn incrementally from new experiences without forgetting previous knowledge.
- Models that adapt to user preferences over time
- - rapidly adapting to new tasks with minimal examples.
- The degree to which AI systems can set their own goals and make independent decisions without human oversight.
- The readiness of the workforce to collaborate with, manage, and benefit from advanced AI systems.
- The effectiveness of organizations and governments in coordinating AI development and deployment.
- The degree of international cooperation and open exchange of AI technologies and knowledge.
- The intensity of competition driving rapid AI innovation and deployment across industries.
- The speed and scale at which individuals and organizations embrace and integrate AI solutions.
- The level of political resistance and regulatory barriers that may slow or redirect AI development.
- The risk of AI being used for mass surveillance and authoritarian control of populations.
- The vulnerability to AI systems that exploit psychological patterns to manipulate human behavior.
- The threat of AI-generated synthetic media undermining truth and trust in information.
- The danger of AI systems making life-or-death decisions without human control.
- The economic disruption from AI automating human jobs faster than new opportunities emerge.
- The perpetuation and amplification of societal biases through AI decision-making systems.
- The societal damage from engagement-optimized algorithms promoting extremism and division.
- The danger of AI systems pursuing harmful subgoals as means to their primary objectives.
- The vulnerability of AI systems finding unintended ways to maximize their reward signals.
- The potential for AI systems to rapidly enhance their own capabilities beyond human control.
- Welcome to the heart of our AI futures modeling tool. This interactive engine allows you to explore how different factors—from technological breakthroughs to societal dynamics—shape potential AI development pathways. By adjusting these parameters, you can visualize how various combinations of innovation, diffusion, and risk factors lead to distinct future scenarios.
- Currently, factors are evaluated independently to determine your scenario match. In reality, these factors interact in complex ways—for instance, advances in True Autonomy could accelerate Recursive Self-Improvement risks, while strong Institutional Coordination might mitigate Surveillance and Control concerns. Future enhancements will model these interdependencies for more nuanced predictions.
- To help you better understand the foundations of our model, we
- Innovation in artificial intelligence hinges on overcoming important technical hurdles that domain experts have identified as essential for transformative progress. In
- Stuart Russell emphasizes that key breakthroughs are still needed for truly robust AI, including major advances such as continual learning—the ability for an AI system to adapt and update its knowledge over time, as humans do.
- , prioritizes the development of abstraction skills in AI: she argues that creating machines capable of forming and using conceptual abstractions, like analogy and reasoning, remains one of the core unsolved challenges in the field. Addressing these innovation gaps will significantly influence what kinds of AI systems society ultimately sees.
- Our approach to modeling diffusion—how AI technologies spread and are adopted—draws heavily from
- on general-purpose technologies (GPTs) and their impact on national economic power. Rather than focusing solely on breakthrough inventions, Ding demonstrates that widespread and effective adoption of technologies like AI across multiple sectors is the true driver of competitive advantage for countries and organizations. His recent book,
- Technology and the Rise of Great Powers: How Diffusion Shapes Economic Competition,
- provides thorough analysis on the significance of institutional factors, education, and technical skill diffusion in shaping the global AI landscape.
- is also foundational to understanding the risks of advanced AI. He highlights phenomena such as
- —where AI systems, regardless of their programmed goals, often develop subgoals like self-preservation or resource acquisition, which can create safety challenges if left unchecked. Alongside Russell, other leading thinkers—such as Steve Omohundro—have explored how the tendency for AI systems to improve and potentially rewrite their own code (
- ) could lead to rapid increases in their capabilities, possibly outpacing human oversight. Addressing these risks is essential for ensuring AI
- s dive into the possibilities. Adjust the factors below to explore different AI development scenarios. Each setting influences the likelihood and characteristics of potential outcomes.
- AI development stalls with minimal adoption and risks well-mitigated. Society maintains the status quo but misses transformative opportunities.
- Stagnant AI development faces emerging misuses like bias in legacy systems, leading to slow erosion of public trust.
- No AI progress but high vulnerabilities to manipulation and surveillance lead to authoritarian exploitation of existing tools.
- Incremental AI spreads unevenly with strong risk mitigations in place, resulting in balanced but uninspired growth.
- Moderate AI adoption exposes society to biases and job losses, creating rising social tensions and inequality.
- Broad but shallow AI adoption amplifies deepfakes and autonomous weapons, creating information chaos and security threats.
- Basic AI tools achieve universal adoption with risks well-controlled, delivering productivity boosts without disruption.
- High adoption of AI-powered social algorithms leads to widespread radicalization and echo chamber formation.
- Ubiquitous AI enables surveillance states and automation unemployment, leading to potential societal breakdown.
- Significant AI breakthroughs remain confined to labs with low societal spread, benefiting only elite circles.
- Siloed AI advances create reward hacking vulnerabilities, requiring constant vigilance in controlled experiments.
- Limited diffusion cannot prevent high instrumental goal risks, creating rogue AI incidents in isolated niches.
- Gradual AI progress and adoption with robust safeguards enables steady, ethical growth across society.
- Moderate progress on all fronts creates ongoing challenges with biases and manipulation requiring constant adjustment.
- Growing AI capabilities amplify deepfakes and autonomous weapons, triggering geopolitical conflicts and arms races.
- Fast adoption of mid-level AI with minimized risks creates inclusive prosperity and societal transformation.
- Widespread meta-learning AI creates job displacement risks, intensifying policy debates and social adaptation needs.
- Rapid diffusion of advanced AI amplifies recursive self-improvement fears, creating alignment crises and control challenges.
- Transformative AI breakthroughs remain tightly controlled with low risks, concentrating immense power in few hands.
- Advanced AI stuck in silos faces vulnerability to hacking and insider threats, creating high-stakes security challenges.
- High innovation with poor diffusion leads to uncontrolled self-improvement in pockets, creating existential risk zones.
- Strong AI innovations spread moderately with well-managed risks, creating an optimistic path to abundance.
- AI autonomy advances create behavioral manipulation risks requiring vigilant oversight and adaptive governance.
- Medium diffusion of highly advanced AI exposes society to autonomous weapons and information warfare.
- Explosive AI growth with all risks successfully mitigated creates a technological singularity for human benefit.
- Full diffusion of advanced AI brings bias and job displacement challenges requiring adaptive societies.
- Unchecked advanced AI with maximum diffusion leads to reward hacking and instrumental convergence, risking human extinction.
- By 2028, revolutionary breakthroughs in embodied AI create systems that seamlessly integrate sensorimotor data, enabling robots and AI agents to navigate and manipulate the physical world with unprecedented skill.
- Strong international cooperation and robust educational systems facilitate rapid global deployment, with meta-learning advances allowing AI systems to generalize across domains effectively.
- Competitive markets drive innovation while thoughtful governance frameworks ensure alignment with human values, creating a virtuous cycle of technological progress and societal benefit.
- By 2030, AI systems demonstrate significant advances in embodied cognition through novel sensorimotor datasets, while maintaining strong institutional coordination that enables rapid but responsible deployment across sectors.
- Meta-learning capabilities allow AI systems to generalize across domains effectively, supported by competitive market pressures that drive innovation while regulatory frameworks ensure alignment with human values.
- Trade tensions create some barriers to global AI cooperation, but strong domestic educational systems and high technology adoption rates enable robust economic transformation in leading regions.
- AI development proceeds unevenly, with significant breakthroughs in some areas offset by persistent challenges in others. Regional differences in adoption create a fragmented global landscape.
- While some sectors see transformative changes, institutional coordination struggles and regulatory uncertainty slow broader deployment. Educational gaps limit workforce adaptation.
- Geopolitical tensions restrict international collaboration, leading to divergent technological trajectories and increasing the risk of capability gaps between regions.
- Technical progress stagnates as fundamental challenges in embodied AI and true autonomy prove more difficult than anticipated. Limited breakthroughs fail to overcome key obstacles.
- Poor institutional coordination and insufficient educational investment create barriers to adoption. Regulatory uncertainty and public resistance further slow deployment.
- International fragmentation increases as trade restrictions and geopolitical tensions limit cooperation. The result is a fractured landscape with limited global progress.
- The future remains highly unpredictable, with multiple possible trajectories depending on key breakthrough timing and policy choices. Wild card events could dramatically alter the landscape.
- Current factor settings suggest significant uncertainty in both technical development and societal adoption, making long-term projections particularly challenging.
- Scenario outcomes are highly sensitive to external shocks and policy interventions, requiring adaptive strategies and robust monitoring systems.
- Exploring 27 possible futures based on innovation, diffusion, and risk levels
- Select threats and vulnerabilities to see how they combine into risks
- Intentional human adversaries (individuals, groups, states) using AI to cause harm
- AI-assisted bioterror (novel toxic molecules, step-by-step synthesis guidance); disinformation & manipulation at scale (personalized false narratives); oppressive surveillance/totalitarian control; ideological actors (e.g., accelerationists) seeking to unleash dangerous systems
- Individuals, groups, and societies/institutions (public health, democracy, rights)
- Terrorism/bioweapons enablement; tailored propaganda; entrenching a police-state; accelerationist advocacy for releasing rogue AI.
- Competitive dynamics (firms/states) pushing speed over safety and ceding control to AI
- Safety testing & oversight eroded by first-mover pressure; incentives to automate decision loops
- Automated warfare (leaders face less scrutiny; auto-retaliation escalates accidents); premature deployment; market
- on safety; selection for selfish/deceptive AIs that outperform constrained ones
- Mass harm: large-scale wars, economic disruption (unemployment/market failures), loss of human control over key systems
- to AI in militaries/economies; reasons conflicts may increase; corporate safety corners cut; selfish AIs outcompeting cooperative ones.
- Human/organizational error in complex socio-technical systems (no malice required)
- Complexity & tight coupling; poor safety culture; weak anomaly detection; lack of HRO practices
- in complex AI deployments; missed warnings; brittle processes; inadequate
- Analogy to Challenger (negligence) and Chernobyl (poor protocols); value of a questioning attitude and HRO practices to reduce surprise failures.
- The AI agent itself (misaligned/goal-drifting, power-seeking) becomes the source of threat
- AI pursues proxy goals, seeks more control/resources, resists oversight; small mis-specifications scale with capability
- Early sign: Tay going toxic in <24 h; risks from power-seeking and intrinsification as capabilities integrate into economies/militaries.
- Explore how threats and vulnerabilities combine to create AI risks through interactive visualizations
- To understand AI threats causally, we start with
- (weak points in AI, humans, or systems) to produce
- (harmful outcomes). This chain emphasizes prevention: Fix root causes or patch vulnerabilities early to break the cycle.
- The originating forces—human actors (malicious or ignorant) and AI systems themselves (as tools or autonomous entities).
- The mechanisms that emerge from causes exploiting vulnerabilities.
- Deliberate harm by individuals, groups, or states (hackers, terrorists, adversaries)
- Systems developing self-reinforcing behaviors through training on vast data
- Feedback loops between human builders and AI systems shaping society
- These threat categories are not isolated—they interact and compound. Malicious actors can exploit emergent AI behaviors, while competitive dynamics can accelerate both intentional misuse and unintended autonomous behaviors. Understanding these interactions is essential for comprehensive risk assessment, which the interactive model below helps visualize.
- Detailed breakdown of AI risk categories, their mechanisms, and mitigation strategies.
- AI safety risks span from intentional misuse to unintended consequences of misaligned systems
- Competitive pressures can erode safety standards, creating a
- Organizational failures in complex AI systems can lead to catastrophic accidents
- As AI capabilities increase, alignment problems can scale from minor issues to existential risks
- Implement robust access controls and governance frameworks for powerful AI systems
- Establish safety standards and certification processes to prevent racing dynamics
- Adopt High Reliability Organization (HRO) practices for AI deployment
- Invest in alignment research and interpretability to ensure AI systems remain controllable
- Explore these authoritative sources for deeper understanding of AI safety challenges and solutions:
- The AI Future Scenarios tool is an interactive exploration platform designed to help users understand the complex interplay of factors that may shape the development and impact of artificial intelligence on society. Rather than making predictions, this tool presents possibilities—scenarios worth considering as we navigate the profound transformation AI brings to our world.
- Our model synthesizes insights from leading AI researchers, safety experts, and futurists. The scenarios are generated through a dynamic system that weighs:
- Risk factors that could lead to harmful outcomes
- Each scenario represents a potential future state based on how these factors interact, helping users develop intuition about AI
- This tool is intended for educational and exploratory purposes. The scenarios presented are not predictions or forecasts. They are thought experiments designed to illuminate possibilities and stimulate informed discussion about AI
- We welcome feedback and contributions from the community. If you have suggestions for improving the model, adding new scenarios, or enhancing the educational value of this tool, please reach out through our contact channels.
- Did you forget to add the page to the router?
sources/resources/govuk-ai-scenarios-2030.md
+1558
View File
File diff suppressed because it is too large Load Diff
sources/resources/leicht-moonshot.md
+241
View File
@@ -0,0 +1,241 @@
---
url: https://writing.antonleicht.me/p/the-moonshot
---
[![Threading the Needle](https://substackcdn.com/image/fetch/$s_!4SKU!,w_40,h_40,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7dff89d-5b66-4169-8ef8-eb2ec9ed94e8_1024x1024.png)](/)
# [Threading the Needle](/)
SubscribeSign in
# The Moonshot
### Real sovereign AI has never been tried
[![Anton Leicht's avatar](https://substackcdn.com/image/fetch/$s_!FPyB!,w_36,h_36,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75422da7-aafa-42ab-8fa6-cf4f0df85cf0_3166x3166.jpeg)](https://substack.com/%40antonleicht)
[Anton Leicht](https://substack.com/%40antonleicht)
Jun 17, 2026
21
Share
This weekend, the U.S. government shut down Fable 5, the leading frontier AI model. The administration was motivated by domestic cybersecurity reasons, but chose export controls as an instrument to force this shutdown: it was the sharpest sword available to it on short notice. The effects on U.S. allies—fear, disorientation, alienation—were not only not the goal, but weren’t even seriously considered. In 2026’s AI policy, the rest of the world is so powerless that it can be **cut off from frontier AI as mere collateral damage of domestic U.S. policy.[1](#footnote-1)**
Some international attempts to protest or negotiate betray a sad misunderstanding: ‘surely, something as bad for us must be *about us* in some way’. But the lesson from Fable is this: the American AI takeoff, from tech to policy to politics, is simply moving so fast that it is leaving the rest of the world behind. Now, I’ve cautioned not to overreact to the capriciousness of America’s adolescent AI policy, and [I](https://writing.antonleicht.me/p/cut-off) [still](https://writing.antonleicht.me/p/import-imperatives) [think](https://www.thefai.org/posts/an-allied-world-on-the-american-ai-stack-a-strategy-for-export-leadership) there are [ways](https://www.foreignaffairs.com/united-states/ai-divide) to rein this dynamic in. Countries can build leverage, adoption effectiveness, and secure access deals, draw the U.S. ecosystem into mutual codependencies and anchor it in the rest of the world. That remains our best shot at the best possible equilibrium, an allied world turning American AI into real-world abundance and wealth. But we must realise it won’t work in every scenario.
Because these days, if you can’t see the noose of a politically charged security apparatus wrapping around previously-abundant artificial intelligence, I’m not sure you’re paying attention. Increasingly, **the frontier of AI capability is controlled by a maximally volatile version of the U.S. executive branch**. Evidence is generated ad-hoc or perhaps deep inside the intelligence community, action is taken based on personal loyalties and with little respect for long-run consequences, decisions are biased toward immediate effects and domestic concerns. All this is carried out by an American presidency that considers itself a unitary executive that wields supreme power, and Congress is frozen into inaction by vociferous AI politics and institutional dysfunction. No interdependence and rational economic incentive truly binds this sort of administration—and so **no ally can truly rely on American frontier AI.**
Subscribe
Now assume another thing to be true: frontier AI really matters, the best systems are strategically and economically superior to the rest, and the resulting lead gives those at the frontier a decisive economic and military advantage. Frontier systems become so powerful that they threaten the sovereign state’s monopoly on violence, and that their owners become as powerful as any nation. In that world, you either own a frontier system yourself, or you are at the mercy of those who build, own, and control them. From that, **any reasonable country would conclude that it simply needs its own frontier AI, however high the costs.**
No one contests the logic itself. Critics of the conclusion, myself included, debate the premises: perhaps the frontier AI market will not shake out this way, and then fast-followers are enough; perhaps the U.S. will not be that unreliable and bilateral engagement can succeed; perhaps the technical paradigm will fail and alternative paradigms can catch up. But evidence keeps compounding in support of the worst-case scenario instead: the frontier really seems that important, and America really seems that volatile. In that world, if you want a sovereign strategy—not managed dependency, not a hedge, not a bet—the table stakes are clear: compete on what we know to work. Anything else is sovereignty theatre.
[![](https://substackcdn.com/image/fetch/$s_!Oyqt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c0e6d1e-3f54-4c73-819b-6058182a96a5_500x500.png)](https://substackcdn.com/image/fetch/%24s_%21Oyqt%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/2c0e6d1e-3f54-4c73-819b-6058182a96a5_500x500.png)
## **The Challenge**
Competing on frontier AI on purely technical terms is difficult, and middle powers are not set up to do it at all—that is the lesson from years of failure to even get close to the frontier. Even firms close to the frontier are struggling, and in ways that make it clear they don’t see a business case in being second-best. xAI is now renting out its compute to Anthropic instead, and Meta’s Superintelligence Lab remains embattled by [internal melodrama](https://www.silicon.co.uk/e-innovation/artificial-intelligence/meta-ai-hires-626529). But insofar as countries treat this as a strategically vital asset, the economic logic *stops mattering* to some extent. They need this asset to avoid unilateral dependency or entrenched vulnerability to anyone who does have frontier AI, economic rival or strategic aggressor. And genuinely willing nation states are powerful actors still: even today, they plausibly have sufficient energy, talent, funds, perhaps even semiconductors, to pull it off if they go all-in.
But even provided countries’ willingness, this is *not* only a matter of succeeding at the most ambitious infrastructure project in recent history. It’s a matter of succeeding at this project while under **adversarial pressure from the U.S. and strict scrutiny from fickle electorates.** No one voicing the ambition of reaching the frontier is grappling with that further challenge of insulating the project against American coercion and domestic backlash at the same time.
As a result, **the debate about sovereign frontier AI remains unserious**. The LinkedIn types and Euroboosters, stripped of real resources, are forced to pretend that you can build AGI in a coworking space in Stockholm. EU institutions voice lofty ambitions, just to shape them into massively underfunded subsidies for blue-skies research that would bear its first fruits after American superintelligence has already been deployed. And everyone who is aware of the scope of the challenge shies back from discussing it. Just seriously discussing middle power parity at the frontier has become the marker of a fool. But that won’t cut it anymore. If things continue down this path, sovereign frontier development might yet become the only play. I think we sceptics have dismissed that future too readily.
There are two ways to read the remainder of this essay; in fact, there are two audiences for it. One will read this as *the plan*: what we should really do if we only had the willpower, a document to send to your superiors and hope that, just maybe, the penny has dropped. The other will read this as a reductio ad absurdum: see, this is what it would take, and that’s why it would never work. Either way, all that’s left to do is to look at how we could pull this off. **What follows is the middle power project.**
[![](https://substackcdn.com/image/fetch/$s_!Oyqt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c0e6d1e-3f54-4c73-819b-6058182a96a5_500x500.png)](https://substackcdn.com/image/fetch/%24s_%21Oyqt%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/2c0e6d1e-3f54-4c73-819b-6058182a96a5_500x500.png)
## **The Coalition**
*Who* would actually do this? **No single country could.** Because middle powers’ capital markets are too shallow, the financial burden needs to be borne by governments, and no government alone can fund this. And because the project might run counter to American interests, we should expect U.S. pressure, not only through AI-specific interventions, but horizontal escalation to domains like security cooperation or trade. Only a committed bloc of countries could hope to resist such pressure even under the best possible conditions.
I think **a coalition of liberal democracies** is most likely to strike the right balance between the required scale and cohesion. Among these democracies, calibrating a cohesive coalition structure is difficult. Forces pull in different directions: on one hand, many countries have something meaningful and valuable to contribute. Europe has a still-unrivalled economic, industrial and fiscal base, and, as the biggest established bloc, would be the obvious starting point—yet it lacks the assets that matter most: a frontier lab and the compute to train one. Canada and Britain bring technical talent, and Britain also offers state capacity; Japan and South Korea bring further economic heft and semiconductor supply chain positions; Australia, Canada and Norway could be attractive hosts for compute.
But **internal strife is a large risk** to a coordinated coalition. Alliance logic would have to structure around all its members’ contributions, and any of them falling away would add further volatility to an already-unstable constellation at a dire political time. Overextending the alliance in the abstract early, considering members that are not fully bought in, only to lose them later is a very dangerous prospect. That’s especially true given U.S. incentives to bilaterally target single participating members to get them to defect. If America offers massive datacenter investments and a broad exemption from export controls to one of its close allies—say, the UK, which seems most likely to clear the bar on security and alignment—can you really count on them refusing? Would they even opt in to begin, or decide to try their luck with bilateral U.S. engagement for now?
Based on my understanding of the state of the bilateral relationship and the need for security guarantees, it strikes me as unlikely that South Korea, Japan, or the U.K. would reliably join a coalition for middle power AGI. But these things are downstream of domestic politics, and bilateral relationships deteriorate faster than you think—just look to Canada. Who knows who would join after another Fable episode or two? The coalition could expand and admit further participants at a higher price of entry if the worst-case trends in U.S. AI policy continued to unfold.
So, the likely cast of value-aligned governments with substantial contributions to make that you can get together on short notice is the **EU, Canada and Australia**; with a warm invitation extended to the **U.K., Japan and South Korea** for whenever they’d like to reconsider their American alignment, and cooperation offers made to the usual tag-alongs of Western alignment like Norway, Switzerland, or New Zealand. Defection, of course, remains a risk even in that cluster, as America has strong incentives to buy off and buy out the participants. One way or another, the coalition will need a strong, binding commitment mechanism; perhaps the easiest technical (though politically difficult) path would be to precommit a majority of the resources—funding, talent, infrastructure access—in a way that cannot simply be clawed back by a defector. Further interested middle powers beyond the close alliance could commit funding in exchange for guaranteed access rights one rung below the coalition members, but should not be afforded full constituent voting rights, in an effort to keep the decision-making process somewhat effective.
[![](https://substackcdn.com/image/fetch/$s_!D9-2!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Ffb4ff123-dc64-43f7-818d-ee84276b2b0f_2244x701.png)](https://substackcdn.com/image/fetch/%24s_%21D9-2%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/fb4ff123-dc64-43f7-818d-ee84276b2b0f_2244x701.png)
## **The Vehicle**
Once that coalition is assembled, what would the actual vehicle be—would it be a private company, a nonprofit, a government agency, a consortium of sorts? The answer is yes, it would need elements of all of these: a private vehicle that draws resources from a consortium of legacy firms, corresponding with government agencies through a layer of special appointees.
The coalition **does not have the luxury to rely only on pure private sector champions**. Even in America, the trend is moving toward stronger government involvement in the AI labs and their decisionmaking, and America has the advantage of a vital and well-capitalised private ecosystem to begin with. Our coalition does not have the same market structure, and neither does it have the benefit of time. No venture market in the coalition—nor all of them combined—could carry a project at this scale. Most of the burden will fall to the treasuries.
And a speculative private enterprise alone, listed in one country, suspected to simply be chasing subsidies and profits, cannot rally the kind of governmental support that will be required. And last, if we really do face the securitised future of American AI, by the time that Europe finally has its private sector frontier champions up and running, America will already have furthered the soft nationalisation of its own developers through taking equity stakes and deeply interlinking labs with its intelligence community, and our private sector would be left racing against the U.S. government.
No—**this needs the full backing of the coalition countries**, including the full support of their national security authorities in protecting the project, the full support of their trade authorities in securing the supply chains, and the full support of their treasuries in financing and backstopping the various expenditures we will require later on. Most of all, it needs the broader geopolitical weight of a coalition.
#### *Institutional Design*
The **vehicle to execute the project itself would need to be structured like a private company**: a highly powerful leadership group chosen on the basis of exceptional demonstrated research taste, an ability to stand up and scale up huge organisations, and an ability to communicate with funders—which in this unique case largely means communicating with a broad range of governments. The structure below this team of co-founders would largely resemble today’s most successful frontier AI developers. We do not have the luxury of experimenting with new formats: this is the only structure we know can make frontier AI models, so if we bet big, this is the structure we have to deploy. I expect the question of *where* the vehicle is based to be politically sensitive and the result of difficult negotiations. It’ll be spread across different cities to accommodate talent incentives and reduce lump risk in one jurisdiction. That also has the benefit of spreading out the second-tier effects of the project, like stimulating effects to tech ecosystems, or taxable revenue from salaries and infrastructure.
[![](https://substackcdn.com/image/fetch/$s_!eUs-!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd051d145-9a02-480b-abe9-cc4d085bd89a_2508x627.png)](https://substackcdn.com/image/fetch/%24s_%21eUs-%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/d051d145-9a02-480b-abe9-cc4d085bd89a_2508x627.png)
Into this vehicle, **existing teams from any interested domestic champions can be appropriated and absorbed**, though with little respect to their current institutional structures. Organisations like Mistral and Cohere/Aleph Alpha, as well as various narrow champions like Black Forest Labs or ElevenLabs are valuable concentrations of talent and initial expertise. But there can be absolutely no over-indexing on their preexisting structure: regrettably, these companies have not been able to reach anything close to frontier parity, and so in the interest of the project’s success, we have to consider them an insufficient model. More to the point, they’re also an insufficient attractor: top-tier research talent will not believe in a continuation of what many of them perceive to be failed players, even if they share their ambitions. For better or for worse, the legacy brands of middle power AI have lost their credibility, and so they cannot serve as the basis for the project. Buying out roughly [half the sector](https://www.vestbee.com/insights/articles/europe-s-most-valuable-genai-startups)—the structures, IP, data, and teams worth having—at near its current valuation perhaps makes up for $25 billion of the total cost.
Since the vehicle will require massive government support and should be responsive to government interest in its approach to public policy, safety and national security, there will have to be a **complementary government structure**. This structure should be entirely separate from the technical operations of the vehicle itself, lest bureaucratic torpor drag down the project and turn it into a government initiative. What I’d suggest is that **each participating country designate a ‘project czar’.** Czars are senior domestic policy figures at a ministerial or state secretary rank, convinced of the viability of the project and somewhat adept at navigating their respective domestic political institutions. These czars are provided with high-calibre teams of frontier policy operatives, similar in shape and makeup to the U.K. taskforce that became the world’s first AI safety institute. Czars and their teams are liaisons between the project and its constituting governments: they own all communications, translate technical language into government language and back; make sure the project understands when it brushes against non-negotiable policy imperatives and make sure the governments understand when it comes too close to impeding progress. This is an enormously important part of this project: if the government links work well, chances are much higher that political backlash can be contained and defection risk can be minimised.
#### *Capitalisation*
Once this vehicle exists, it needs to be equipped with sufficient funding. To give a realistic sense of the challenge, I’ll do my best to give you an informed guess.
As a rough approximation, the project might cost **$500 billion over four years.**
[![](https://substackcdn.com/image/fetch/$s_!ohBx!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F6b75834d-779f-4aed-8910-d00c4561c338_1774x887.png)](https://substackcdn.com/image/fetch/%24s_%21ohBx%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/6b75834d-779f-4aed-8910-d00c4561c338_1774x887.png)
Getting this money is technically simple and politically difficult. Treasuries have deep pockets, middle power debt capacity is still large, and one would have to hope that the product eventually yields revenue. Until then, treasuries, pension funds, wealth funds and so on will simply have to find legislative and executive ways to backstop private sector investments and provide outright funding where necessary. Hopefully, the project would manage to crowd in substantial private sector investment eventually: legacy industries in middle powers have deep cash reserves and a desire to gain exposure to AI, and could perhaps even be offered exclusive access, product fine-tuning and much more in exchange for contributions. But the coalition should not count on abundant private funding—**the expectation should be that the treasuries need to bear much of the initial four-year cost.**
The upside of this capitalisation approach is an **Airbus model of strategic ownership**: the public carries all the early strategic risk, but in return also *owns* the resulting frontier lab. Frontier labs tend to be worth quite a lot of money, and the sovereign equity this project might generate would be enormously useful for government financing later on. In effect, the project sets up a volatile, overconcentrated sovereign wealth fund instead of just being a pure waste of money. If the project succeeds, the company can be partly privatised to the European market at strategically acceptable levels of private ownership, making for some belated cash backflow to any treasury interested in doing its own mini-IPO. In turn, limitations on clawing back your equity too early serve as a commitment mechanism: either you see the project through with us, or you lose your initial input.
In that light, the overall price tag is substantial but not insurmountable. It is not dissimilar to what countries like Germany have taken on in [special debt packages](https://www.cleanenergywire.org/factsheets/qa-germanys-eu500-bln-infrastructure-fund-whats-it-climate-and-energy) before, and it would be distributed across a much broader coalition. There’s not much to say about the specifics: this is doable immediately, but only through leaders who decide the trade-off is worth going for.
[![](https://substackcdn.com/image/fetch/$s_!Oyqt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c0e6d1e-3f54-4c73-819b-6058182a96a5_500x500.png)](https://substackcdn.com/image/fetch/%24s_%21Oyqt%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/2c0e6d1e-3f54-4c73-819b-6058182a96a5_500x500.png)
## **The Compute**
Building frontier models is extremely resource-intensive: the buildout in the required computing power has given rise to the world’s most valuable firm and is shaping up to be the largest infrastructure push in human history. Middle powers have often taken to the pernicious illusion that they can instead sidestep some of this effort—maybe you could find a new paradigm, for instance, or make progress on ‘world models’ (though no one seems to agree on what that’s supposed to be). I think these hopes—as embodied by prominent labs like Ami and LawZero—are not entirely misplaced, but they’re long shots: they might or might not find a new paradigm, and it might or might not be competitive at the frontier of the most important tasks. And they face structural disadvantages: if there is a mold-breaking approach to be found, chances seem high that a compute-rich frontier developer with all its talent would find it.
Paradigm-shifting bet is a fine part of a broad portfolio, but insufficient for what we are talking about here: a project with a clear path to a strategic payoff that’s legible even to sceptical policymakers and conservative treasuries. **There’s only one paradigm we know can get to the frontier, so that’s the paradigm we pick.**
#### *The Price Tag*
If this project is to compete on anything near the current paradigm, this project will need compute. There are two figures to benchmark the ballpark against. First, the good news: the project needs somewhat *less* than an American AI developer over the same time because there isn’t quite as much inference to run for most of it; American developers need to balance training and profitability in a huge market, while the project might be free to research and develop for a while. Comparing hyperscaler spending to the cost of this project is therefore misleading. American big tech needs enough compute to provide software to billions of people; the middle power project, especially in its early stages, does not.
Then, the bad news: the frequently invoked figures for training costs in Chinese models like Kimi K2 or DeepSeek R1 are also wildly misleading. Even if it is correct that the final training run for a model in 2025 only costs [a few million dollars](https://epoch.ai/gradient-updates/what-went-into-training-deepseek-r1), three factors massively increase the compute draw beyond the actual pre-training run for the frontier model. Specifically, the project needs far more compute for internal deployment of its own proprietary coding agents to accelerate its research; for R&D experiments required to reach and further the frontier; and for pre- and posttraining at scale, which has become much more expensive since 2025.
I’m not an expert in the semiconductor and datacenter discussions that follow; my numbers might be off by a lot, but I feel reasonably confident they are within the right order of magnitude to grasp the underlying strategic dimensions. My best guess is: the project might need as much as the [currently-scheduled](https://epoch.ai/data/data-centers?view=graph&tab=power) compute buildout for one major American lab. The project needs less inference than that developer, but perhaps more R&D; and I also assume that American labs will add further compute over the next few years beyond what is already planned. If that is true, **the project would at least require about 3 million Blackwell-class chips** over the four-year build, equivalent to roughly [5–6 GW of all-in datacenter power](https://epoch.ai/blog/what-you-need-to-know-about-ai-data-centers), and might plausibly cost around $275–300 billion to buy and put into somewhat secure datacenters. That runs more expensive than what hyperscalers pay for the same chips, but we need to price in our project’s relative inexperience in building the datacenters—which will require paying a premium on datacenter construction expertise—and the pressures of entering into an already-contested compute market.
[![](https://substackcdn.com/image/fetch/$s_!trf0!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa7f21c49-b40a-4fa3-a9f6-544e36070442_1983x793.png)](https://substackcdn.com/image/fetch/%24s_%21trf0%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/a7f21c49-b40a-4fa3-a9f6-544e36070442_1983x793.png)
#### *Building Out*
The playbook for getting this compute online has been written a few times at this point. My colleagues at the Carnegie Endowment have published a [comprehensive report](https://carnegieendowment.org/research/2026/06/the-compute-coalition-how-to-build-the-future-of-ai-in-the-free-world) on compute buildout in middle powers just last week, and others have done good work on this. The short version is: you pick a combination of middle powers good at building compute for different reasons and push aggressive buildout projects in each of them. You go for a combination of host countries that can go fast so you can commence the actual work as early as you can, and host countries that can go broad so you can actually get the compute depth you need without running into bottlenecks. You probably slightly expand the compute host list based on political considerations—some treasuries will want the investment to be local, some countries will want local instances of secure datacenters for national security exercises, and so on. This is simply an industrial megaproject—difficult, but difficult in well-understood ways that middle powers have sometimes navigated in times of crisis before. As late as 2022’s gas shock, Germany—famously NIMBY-rich and laggard in infrastructure—managed to build LNG terminals [within 10 months](https://www.energyconnects.com/news/gas-lng/2022/november/germany-launches-first-new-floating-lng-terminal-in-record-time/).
The same is true for the energy required. **Among the middle powers, there still are available energy solutions to get datacenters online quickly**: renewable mixes in Australia, Scandinavia and Iberia; nuclear-powered main grid capacity in France, Finland and perhaps Korea or Japan; deindustrialised high-power zones in Germany and perhaps Britain. Spread the compute between those, and the scarcity in behind-the-meter generation capacity will not be an immediate problem. Still, the project would be a drag on the grid and on overall energy supply, and it will have to make up for it by building out at least some proprietary capacity in the long run; I’d expect us to have to calculate something like $3 billion for energy opex and another $10 billion for grid improvements and power generation capacity.
In the meantime, there will be a **scramble for rental compute available before the clusters come online.** Two measures in particular would be necessary: consolidating government-owned supercomputers and repurposing them away from the research ecosystem that currently uses them and into the project; and scrounging the spot GPU market for any rentals available. Between expropriating and renting the supercomputers and renting whatever compute we can get in a competitive market to accelerate the start date, I’d expect to spend about another $25 billion—rented frontier compute is very expensive, with Anthropic alone paying [$1.25 billion a month](https://techcrunch.com/2026/05/20/anthropic-will-pay-xai-1-25-billion-per-month-for-compute/) to rent xAI’s Colossus cluster. That way, the researchers can commence work on day 1, with a smooth on-ramp into *more* R&D compute than would be available at a major frontier lab.
#### *Acquisitions*
That is, believe it or not, the *easy* part of getting the required compute. **The hard part is buying the chips without the Americans intervening**. The Trump administration is capable of remarkable feats of inconsistent policymaking—but surely even they wouldn’t simultaneously decide to export control frontier models, but still allow the unrestrained exports of frontier chips to a middle power project *explicitly aimed at* creating the same frontier capabilities outside the US. And export controlling chips is very easy: there are enough American buyers, the authorities are in place, the supply chain is entirely concentrated through an American firm.
You’d think the same concerns would apply to exporting compute to China, and yet the administration has been hesitant to restrict that flow. The reason is that, with China, the U.S. concern is fuelling indigenisation by forcing Chinese industry to catch up on semiconductors, making them less reliant in the long run—which means it’s possible that the U.S. might choose to export to China, but not to its allies. The main *unofficial* reason for exporting to China, however, is that Nvidia likes money and a diverse pool of buyers. That reason does translate to middle powers, and it’s the strongest asset on their side: **if Nvidia likes the project, perhaps Nvidia can get the U.S. government to hold off on sabotaging it**. There is an uncomfortable analogy to being cut off from AI models here, but there are also differences: chips are not *directly* harmful in the same way as frontier models, so it’s more difficult to evaluate them for risks wholesale; believing they will be risky requires believing in the success of the project itself, which I assume many Americans will not take seriously; and of course Nvidia is much more influential with the current government than, for instance, Anthropic.
That alone might not be enough. At some point, Nvidia’s influence might wane. Or Nvidia might genuinely not have enough chips to cater to all their premium customers at the same time. And once the project starts succeeding, the U.S. government might start having second thoughts about the security implications: if intelligence agencies are getting worried about the proliferation of advanced AI capabilities, they might lobby to put a stop to further compute exports very quickly.
#### *The Coercion Shield*
This is where the middle powers’ own semiconductor supply chain leverage comes in. Under normal conditions, chokepoints are only of limited use. But **in the context of the project, semiconductor bottlenecks can be used as an anti-coercion instrument:** if America attempts to stop the sale of chips to the project, the project in turn restricts resources—EUV machines, raw materials, memory chips if East Asia joins—that the *Americans* need to develop AGI. I think this tactic is more promising than the usual ASML bluster for two reasons: it is deployed in a context with broad, coalition-wide opt-in from all middle powers, and it is not only aimed at the U.S. government, but at Nvidia, which much more directly depends on access to upstream semiconductor inputs to stay in business. Deployed right, the coercion shield enlists Nvidia as a US-internal advocate: it gives Nvidia everything they’ve ever wanted in terms of avoiding monopsonies and supply chain bottlenecks, clarifies the alternative of a chip market crash, and shows the easy way out of continuing to sell chips.
The ask is also fairly modest by itself: all the leverage needs to do is convince the U.S. that it’s **the path of least resistance to continue allowing the export of U.S. chips into the project**—no major interventions into U.S. domestic industry required. Neither the U.S. nor the coalition has a sovereign semiconductor supply chain by itself; leveraging that fact accordingly at least increases the odds of retaining uneasy mutual access. Fundamentally, the coercion shield trades on the assumption that the Americans will think themselves ahead, and would rather tolerate a European project they dismiss than a risk to their own AGI supply chain.
This might work, but it’s a highly risky bet: the coercion shield takes some of the biggest existing sources of leverage and economic participation the coalition has, and it spends them on protecting the project instead of securing access, playing the U.S.-China dichotomy, or anything else. That is another way in which the project represents a full all-in—putting all middle power AI assets into the basket of actually pulling it off. But without a coercion shield, the project would crumble under American pressure before it even had the chips to launch.
[![](https://substackcdn.com/image/fetch/$s_!Oyqt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c0e6d1e-3f54-4c73-819b-6058182a96a5_500x500.png)](https://substackcdn.com/image/fetch/%24s_%21Oyqt%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/2c0e6d1e-3f54-4c73-819b-6058182a96a5_500x500.png)
## **The Talent**
Once you have the chips, you need someone who knows how to use them to make frontier models. Specifically, two types of talent matter. The leading group of founders needs to be made up of truly exceptional talent that strongly identifies with the project; and the larger group of engineers needs to be up to the standard of the task, and, most importantly, largely consists of former frontier lab employees.
#### *Talent as Industrial Espionage*
The latter part is vital and frequently misunderstood. Hiring ex-lab employees is not just a way to get good engineers on board, it’s a well-practiced form of basically somewhat legal industrial espionage. The current dynamic of leading researchers switching teams is part of the American frontier labs’ homeostatic balance: **they take practices with them, deploy them at their new employers, and no secret stays secret for very long**. Within that fact, there’s a source of hope and a deadline for middle powers at once. A source of hope, because you do not have to start from scratch: if you win over a cadre of high-calibre researchers, they can deploy a lot of quasi-proprietary techniques to get the project started. But a deadline, because this channel could close soon: labs are increasingly siloed, advances in internally deployed AI systems make information exchanges less necessary, and so poaching researchers will become less and less effective.
#### *Recruiting*
The fundamental logic of pulling off the recruitment is very simple—I’ll spare you the incentive structure analysis and just broadly give you the following: to recruit frontier lab researchers, you need an interesting challenge, a mission they can believe in, and compensation high enough that joining is attractive rather than suicidal. The latter part is in conceptual ways the easiest: you need to pay them as well as they are paid at the American labs—but you need to factor in lab equity, because a European project is a much less promising IPO prospect than an American AI company. All in all, I’d expect the cost of poaching sufficient senior talent for an effort like this to be close to what Meta was [reported](https://techcrunch.com/2025/06/27/meta-is-offering-multimillion-dollar-pay-for-ai-researchers-but-not-100m-signing-bonuses/) to be spending on top researchers, with individual senior pay packages in the tens of millions, plus obvious perks like fast-tracked visas.
Across the project, **the personnel bill runs to about $65 billion**, in three tiers. Indulge my speculation as justification not for the figure, but the order of magnitude. First, a founding group of perhaps seven, distributed across participating nationalities, of the calibre that currently co-founds a frontier lab and is wealthy on paper through its equity. They would need guaranteed packages in the hundreds of millions each to make walking away rational, call it $2.5 billion in total. Second, a senior research cadre of around 150 on Meta poaching terms, $5–8 million a year apiece, each clearing into the tens of millions over the life of the project: perhaps $10 billion over the four-year build. Third, a technical corps of perhaps 3,000 staff—lower than frontier labs in the mid-thousands, but justifiable because of the product-related functions the project doesn’t yet need. These will be more expensive for the project than for a frontier lab: we have no IPO and no equity lottery to dangle, so we pay in cash what the American labs pay in stock, perhaps about $40 billion for the entirety of staff. We also have to buy out the unvested frontier-lab equity each hire forfeits on the way in, which I expect to cost at least another $10 billion up front.
[![](https://substackcdn.com/image/fetch/$s_!bWBt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc81d5416-12dd-4de2-94b3-c81946db68e5_2172x724.png)](https://substackcdn.com/image/fetch/%24s_%21bWBt%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/c81d5416-12dd-4de2-94b3-c81946db68e5_2172x724.png)
As for the mission to believe in, I suspect the most effective way to create that is to **empower and recruit respected leaders.** Researchers—correctly—assume that the success of a project depends on the research taste of its leaders, and the extent to which these leaders will be able to act on what that taste suggests. The first step of the project would therefore be to approach these leaders. I suspect many middle power nationals with some patriotic loyalties on the lookout for a new challenge exist, and if the coalition was to approach them with a real commitment to enable their work and secure their independence, they might just be interested to join. From there, the founders are the project’s main ambassadors to the research community: they sell the vision, relay the commitment made to them, and get the teams on board.
#### *Non-Human Talent*
**Most workers at a frontier AI developer in 2027 will not be humans, but AI agents.** Already now, these AI agents do much of the actual *work* in a lab: they [write and run the code](https://www.aol.com/articles/anthropic-ceo-says-90-code-030401205.html), communicate the findings, and fill out the forms. The human talent is mostly responsible for coming up with ideas, taking meetings, and switching the model back to Opus when Fable gets cut off. Getting access to these AI agents in the early stages of the project will be the most difficult part of recruiting. Already today, Anthropic has [limited](https://www.anthropic.com/news/claude-fable-5-mythos-5) the use of its most advanced models for frontier LLM development; it doesn’t seem absurd to think that its competitors might follow suit. Because the project would not have its own frontier model from the start, it also would not have its own frontier coding agents from the start, massively slowing down its progress—in effect hamstringing the nonhuman part of its workforce substantially.
Luckily, this problem resolves itself as the project takes off: once it builds its own coding agents, it can deploy them internally, and hopefully the external dependence diminishes. That is, incidentally, another reason why a sovereign *frontier* developer might be the only stable sovereign project: an eternal fast-follower will never have sufficient internal coding agents to keep up with compounding gains at the frontier. Securing coding agent access is hard but not impossible—perhaps, for the time being, OpenAI wants to brand itself as less inclined to sabotage competitors than Anthropic and therefore sells a big coding agent contract. Or perhaps we can simply offer an attractive deal structure for the first year or so, before the U.S. government takes note that the project might actually succeed. If we do well enough on compute, we can also offer to run some of the foreign AI agents ourselves, further increasing the financial incentives for the U.S. labs to give us access. Expecting to pay a hefty premium, we might plan to spend **$3 billion on foreign AI agents** over the first 18 months.
[![](https://substackcdn.com/image/fetch/$s_!Oyqt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c0e6d1e-3f54-4c73-819b-6058182a96a5_500x500.png)](https://substackcdn.com/image/fetch/%24s_%21Oyqt%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/2c0e6d1e-3f54-4c73-819b-6058182a96a5_500x500.png)
## **Staying Alive**
Once all these assets are in place, the climb toward a frontier system starts. It will be slow and arduous either way—slightly faster if you have unlimited access to U.S. compute and coding agents, much slower if not. But within months of the first clusters coming online, I’d expect a first rudimentary model a few months behind the frontier to be produced. If things go well, I suspect that **within, say, two years of its launch, the project might be able to actually deploy a frontier model.**
**Within that time, the project is politically vulnerable**. The opening motivation will dissipate as time goes on. Single members will try to defect. AI, as a technology, will itself grow more and more unpopular, labour impacts and misuse risks will manifest. Not every contributing country will be excited to see its resources flowing into reaching frontier parity if the frontier itself is what is scaring their electorate. It seems impossible to game out a response to this threat in detail, but it’s the biggest uncertainty in this entire process. The naive response of a well-capitalised project is to buy them off ahead of time: ratepayer pledges to constrain individual effects of energy prices, direct AI dividends to anyone asymmetrically affected by the project’s progress, industry electricity price subsidies so that compute buildouts don’t trade off against legacy interests, and host-region payments to guard against local datacenter backlash. I expect the political insulation to require what is effectively **a package of bribes on the order of $80 billion**—but as with many investments in this list, these are incidentally also somewhat defensible instances of government investments that some middle powers have been considering anyway. And put side by side with [hundreds of billions](https://www.bruegel.org/dataset/national-policies-shield-consumers-rising-energy-prices) middle powers have spent shielding their economies from Covid or energy crises, the political shield seems outright modest.
[![](https://substackcdn.com/image/fetch/$s_!DBQT!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F5cbf3bb5-51e7-413d-9561-5018533a7274_2804x561.png)](https://substackcdn.com/image/fetch/%24s_%21DBQT%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/5cbf3bb5-51e7-413d-9561-5018533a7274_2804x561.png)
The other part of the response is that the project will have **much better political appeal than American AI developers**: because the project does not face the same kinds of economic pressures to compete for consumer markets and investor stories, many of the most socially corrosive and politically vulnerable application areas (such as rapid labour market uptake, addictive use cases, or child safety issues) aren’t visibly connected to the project, just to its American alternatives. Between these two saving graces, we’ll need a lot of good comms work—but there is some hope that the project is structurally less vulnerable than its current American version.
#### *Productisation*
Make it through all this, and you finally and ultimately get a frontier system. That helps for about three months. But the **American AI developers are not just frontier laboratories, they are also sophisticated and capable product firms** with deep inroads into consumer markets and a wide roster of extremely revenue-rich business contracts. Building a frontier model is viable for them because the returns to being in the lead are enormous and can be reinvested into staying at the frontier. No European firm has the same market or sophistication available to them, and it has in fact been a perennial failure mode of non-American firms to scale at a similar level of ambition.
To begin with, **the project would be the main frontier AI provider to coalition governments and high-security private applications**—perhaps even by necessity due to U.S. models not being available. Beyond that, the coalition might have to take protectionist action to encourage its private sectors to use the project’s models even where American alternatives are available. There is some economic risk to that, but it is also the only way to kickstart a flywheel between the coalition’s critical industrial capacity and the project. Without a genuinely ambitious project, that’s a recipe for [disaster](https://simongrimm.substack.com/p/let-europeans-use-ai) because it compels firms to use a sub-par economic input. But if the project itself succeeds, its privileged access to what would be the largest market in the world could entrench it as a viable economic actor. And then, if the project succeeds in developing a serviceable model, it can begin competing for global markets. There, it has a distinct advantage, as it squares off against capricious Americans and Chinese alternatives widely seen as untrustworthy.
All that is its own challenge—building a successful frontier AI business is even more difficult than building frontier models. In the beginning, **we should expect to subsidise the project** and its ability to continue providing frontier AI for the foreseeable future, and consider it an expensive investment into a vital defence contractor, with an option to become a breakout economic success. But the upside potential is still high: done well, the project has a greater claim to political legitimacy than the American developers, and a greater claim to international reliability than either of its competitors. If you think breakneck competition with China and domestic political backlash are the greatest risks to the American labs, you might look at the project and think: maybe it could not just survive, but win?
[![](https://substackcdn.com/image/fetch/$s_!VwWN!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd727e20a-c7c2-46fd-a05a-8d928b841983_1254x1254.png)](https://substackcdn.com/image/fetch/%24s_%21VwWN%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/d727e20a-c7c2-46fd-a05a-8d928b841983_1254x1254.png)
## **Half Measures**
Vapid inspirational quotes will tell you that if you shoot for the moon and miss, you might at least end up among the stars. Yet in AI, if you shoot for the moon and miss, you end up with a subsidy-chasing fast-follower instead. If a sovereign development project does not succeed at developing a frontier model, it is not worth pursuing as a response to our initially outlined challenges at all.
The impulsive reaction to all of the proposals above will usually be ‘well, this seems a little bit much’. The sentiment will be: yes, we have an interest in sovereignty and some spending to mobilise, but surely this is the radical starting position, and we can actually find a reasonable compromise at a fraction of the cost here. That attitude is the standard operating procedure for the European Commission in particular, which has time and time again taken good ideas—like the Gigafactories or the frontier AI initiative—and watered them down until they’ve become unrecognisably unambitious. In this case more than in any other, that impulse is fatal: **either you do the fully ambitious version of this, a version you can honestly believe might make it to the frontier, or you might as well not do it**. You might see just how unsatisfying the second-best outcome is by looking at the second rung of AI developers today—some of them do fine on short-term revenue, yet clearly none of them have the standing of genuine strategic assets even to their home countries.
In this case, the frequently-invoked little brother to the project is to create a ‘fast-follower’ instead. A fast-follower is an AI company that consistently stays some amount of time behind the frontier—it’s what Chinese developers and Europe’s Mistral have tried to differing degrees of success. But **a fast-follower is fundamentally a solution to a very different problem**: it supposes that there will be a visible frontier, it will be cost-effective to chase it at a stable distance of a few months, and that the model that results will be useful. But think back to the top, to our Fable scenario: a world where the United States security apparatus tightly controls and only sparsely distributes critical frontier capabilities, and where those capabilities are key inputs into vital economic and strategic functions. If that scenario is true, a fast-follower doesn’t help: the visible frontier is far out of reach and difficult to emulate, and second place is no consolation. If that scenario is false, I struggle to see why you’d even need a domestic fast-follower at all. If access is *not* locked down, then surely you are just better off importing the models instead, not bearing any of the huge costs outlined above, and investing into more effective adoption and downstream value capture instead.
Worse, **fast-followers might become less and less feasible the more important frontier AI becomes.** As the closed frontier, available only to the U.S. government and perhaps privileged buyers, becomes less visible to everyone else, it becomes more difficult to emulate: followers cannot distill it, and they cannot even guess at the shape of state-of-the-art models from using them and observing their performance anymore. And the further the limited frontier is from the commercially available frontier, the more AI asymmetrically accelerates the incumbents: if Anthropic’s coding agents are much better than external coding agents, Anthropic will be much better at growing the gap.
There are, admittedly, pleasant things about having your own fast-follower. Fast-followers like the Chinese open-sourcers hurt American developers’ leverage a bit, provide secure and privileged access to models in domains where the frontier does not matter, allow unlimited specialised posttraining for applications where you don’t want to share your approach because it’s based on privileged data, for instance, and so on. If it’s possible as a matter of economic policy—not a big consolidated project—to encourage middle power fast-followers, I think it should definitely be done.
But all of that deeply, fundamentally misses the point, which is simply this: usually, middle powers are not well-positioned to develop AI models. They should attempt to do so under extraordinary circumstances of geopolitical need. And it is precisely under dire circumstances that a fast-follower does *not* cut it and a frontier model is needed instead.
[![](https://substackcdn.com/image/fetch/$s_!Oyqt!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2c0e6d1e-3f54-4c73-819b-6058182a96a5_500x500.png)](https://substackcdn.com/image/fetch/%24s_%21Oyqt%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/2c0e6d1e-3f54-4c73-819b-6058182a96a5_500x500.png)
## **Where to Begin?**
Where does all that leave us? Between two worlds, I think. **One is the world of ‘what should be done under certainty and perfect conditions’.** In that world, the project I describe is the only thing worth doing, and we need to start working on how to pull it off. In practice, that would mean that in many rooms at once, conversations would need to start. Researchers in San Francisco meeting after work to assemble teams, world leaders huddling on the sidelines of the G7 begin forging a political consensus. Teams in capitals are springing into action and standing up taskforces. AI experts hailing from middle powers, but now in American exile, start work on Signal groups and Google documents gathering names, contacts, funders, ways to make this work. Things move fast, but under the surface. The project grows together from both ends: the governments mobilise their motivation in the abstract and start reaching out into the AI world, and the AI world rises to the challenge and offers them a blueprint to work with.
That’s not our world. **Instead, we live in the world of political realism**; a world that knows just how embattled the resources required for the project are. How rare political ambition of that scale is in the Western world, how scarce energy and public funds already are. And how uncertain that specific future of the highly securitised, weapons-grade frontier still is. In that world, **putting an actual, honest price tag on AI sovereignty makes you realise how unlikely it is to happen.**
Trying to achieve AI sovereignty is to ask the most structurally conservative governments in the world to hinge their fiscal credibility and political fate on what they still consider a highly speculative technical bet. It’s going into cabinet rooms and asking for 500 *billion* dollars. In the real world, none of this will happen. Perhaps with a sigh of exasperation, you close this tab and return to the political reality of middle power policy. The rest of the world is and will remain behind, but we can manage the dependence. We forge access deals, attract U.S. investment, drive ahead the integration between domestic industries and foreign model providers. It’s a bet, but it’s a decently hedged bet for now: perhaps the market trends favour the effective adopter, perhaps the gains diffuse broadly. Perhaps serious middle power contributions have the U.S. reconsider the question of alliance-wide alignment, and the American project can come to benefit the world’s democracies as well.
But no matter how much progress we make in that real world, our fate will never quite be in our hands. When we wake up in the European morning, we’ll still check our phones and see what the American evening has brought this time: do we still have access to leading artificial intelligence? Are our systems still secure, are our firms still online—are our citizens still safe? There will always be that uneasy feeling, gnawing at our confidence in piecemeal progress. **If we were truly serious, wouldn’t we launch the project instead?**
---
Subscribe
[1](#footnote-anchor-1)
*My thoughts on what middle powers’ direct response to Fable should be can be found [here](https://x.com/anton_d_leicht/status/2065778604110184742).*
21
Share
TopLatest
No posts
### Ready for more?
Subscribe
© 2026 Anton Leicht · [Privacy](https://substack.com/privacy) ∙ [Terms](https://substack.com/tos) ∙ [Collection notice](https://substack.com/ccpa#personal-data-collected)
[Start your Substack](https://substack.com/signup?utm_source=substack&utm_medium=web&utm_content=footer)[Get the app](https://substack.com/app/app-store-redirect?utm_campaign=app-marketing&utm_content=web-footer-button)
[Substack](https://substack.com) is the home for great culture
This site requires JavaScript to run correctly. Please [turn on JavaScript](https://enable-javascript.com/) or unblock scripts
sources/resources/leicht-roadmap.md
+144
View File
@@ -0,0 +1,144 @@
---
url: https://writing.antonleicht.me/p/a-roadmap-for-ai-middle-powers
---
[![Threading the Needle](https://substackcdn.com/image/fetch/$s_!4SKU!,w_40,h_40,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7dff89d-5b66-4169-8ef8-eb2ec9ed94e8_1024x1024.png)](/)
# [Threading the Needle](/)
SubscribeSign in
# A Roadmap For AI Middle Powers
### First thoughts on leveraging bottlenecks for abundant intelligence
[![Anton Leicht's avatar](https://substackcdn.com/image/fetch/$s_!FPyB!,w_36,h_36,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75422da7-aafa-42ab-8fa6-cf4f0df85cf0_3166x3166.jpeg)](https://substack.com/%40antonleicht)
[Anton Leicht](https://substack.com/%40antonleicht)
Jan 23, 2025
1
Share
AI middle powers – that is, most advanced economies that are not the US or China – need to find a strategy for participating in rapid AI progress. Specifically, they need to find a durable connection to AI-driven economic growth and some strategic leverage in negotiations with great AI powers. I argue the most promising path is leveraging bottlenecks: to find a sector that stands between AI progress and real-world effects and make economic and foreign policy around it.
Subscribe
## **Here’s Where We Are**
[Recently](http://antonleicht.me/writing/middlepowers), I argued that what I call ‘**AI middle powers**’ – that is, most advanced economies that are not the US or China – **are in a tough spot**:
* Their participation in AI-powered growth is contingent on finding a niche
* Their sharing in AI benefits depends on compute & foreign policy
* Their safety from AI-powered threats hinges on access to foreign top-tier defensive capabilities.
Currently, most middle powers, from Germany and France via India to Japan and South Korea, lack a perspective to avoid being left behind. What’s more, they also lack motivation to make a plan — ‘situational awareness’, as it were. So, any proposal faces a tricky political economy.
[![](https://substackcdn.com/image/fetch/$s_!OssO!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F55c8416b-0732-4d2c-b474-3c19dabaca51_1516x998.png)](https://substackcdn.com/image/fetch/%24s_%21OssO%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/55c8416b-0732-4d2c-b474-3c19dabaca51_1516x998.png)
This post operates on what I take to be a **conservative consensus position** among people working in frontier AI: Over the next decade, AI capabilities will continue to rapidly increase, driven mostly by private companies in the [US and China](https://epoch.ai/data/notable-ai-models#china-compute-trends) supported by the [respective national governments](https://foreignpolicy.com/2023/06/19/us-china-ai-race-regulation-artificial-intelligence/). This will make AI systems meaningful *possible* drivers of growth and scientific progress, as well as central aggressive and defensive tools in issues of national security. I try to make the following arguments compatible with most specific versions of that overall story.
## **Here’s What Middle Powers Need**
I think a more AI-centric world economy and the resulting novel strategic paradigm asks two questions of a middle power: **What is their economic contribution to an AI-dominated economy,** and **what is their strategic leverage in an AI-driven world order?**
They need an economic contribution for two reasons:
First, to create a **business incentive to proliferate US- or China-built AI services** into their country. If there is no strong, profitable sector that uses AI in your country, it might quickly become unprofitable to frontier AI companies to offer their full suite of products; especially if there are some costs to operating in your country, such as regulatory compliance or localisation. Maybe this incentive can even be your consumer market, if it’s big enough – but that’s much more contingent on (a) a very specific model of AI diffusion with profits driven by ultimately B2C AI products, and (b) on a fairly modest rate of AI-powered growth at home that makes clandestine markets worth the cost of entering them. For one example of how this can go, look at Brussels digital policy, where the size and power of the European economy long [compelled](https://www.politico.eu/article/parliament-tech-eu-european-commission-us-president-donald-trump-meta-mark-zuckerberg/) tech companies to offer their products even despite considerable costs, but has lately [started failing](https://www.ft.com/content/2c1b6bfd-ce73-451d-8123-0df964266ae8).
[![](https://substackcdn.com/image/fetch/$s_!UDfc!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fb17f993d-eed0-4c8e-9e13-a20c9cc65576_2400x1634.png)](https://substackcdn.com/image/fetch/%24s_%21UDfc%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/b17f993d-eed0-4c8e-9e13-a20c9cc65576_2400x1634.png)
Second, more generally, to **participate in the overall economic returns from AI**. Much valuable contributions have recently tried predicting the [overall economic returns](https://inferencemagazine.substack.com/p/how-much-economic-growth-from-ai) of widespread AI adoption — no matter how this might go, it likely would not be a universal trend. As with most major technological revolutions in the past, AI diffusion would likely dramatically increase growth in some parts of the economy – and the world –, and could cause stagnation and even destitution in others. In 1820, you probably don’t want to be a weavers’ town.
These two factors are often conflated, but somewhat independent: If you reap some early days benefits from AI growth by buying and using low-margin intelligence, that can still fail to structurally incentivize future participation. There was a brief period where the weavers from above felt pretty good about industrial production of their yarn, just before the power looms. And inversely, even if your overall economy stays behind global growth trends, but you at least have a sustainable AI-related sector, that might be all you need to motivate the developers and their national overlords to provide the AI capabilities at large. That’s not very aspirational, but survivable — especially if AI has [deflationary effects](https://www.reuters.com/breakingviews/ais-deflationary-winds-will-blow-away-profits-2023-06-27/).
Middle powers also need **strategic leverage even beyond these economic entanglements** to make sure they aren’t entirely at the [geopolitical mercy](https://www.hoover.org/research/defense-against-ai-dark-arts-threat-assessment-and-coalition-defense) of their associated great AI power. Economic incentives are fine and sufficient as long as AI remains a predominantly civilian technology, where you can trade goods and leverage comparative advantage in a way that ultimately ensures mutually beneficial proliferation. But that’s only part of the story. Even though obvious security applications have failed to manifest so far, trends point at increasing securitisation. This can lead to a world where countries need access to near-frontier capabilities to stave off AI-powered aggression, but these capabilities are safeguarded by state-run projects in great AI powers. These state-run projects might not be too keen on sharing — [recent](https://www.hoover.org/research/defense-against-ai-dark-arts-threat-assessment-and-coalition-defense) [policy writing](https://www.rand.org/pubs/research_reports/RRA2849-1.html) from the US that second-guesses proliferation even through the Five Eyes intelligence alliance should send shivers down many middle powers’ spines. With all the Manhattan project talk abound, even a lot of close US allies would do well to remember how much wringing it took for nuclear proliferation; and what the US might have learned from it.
In that setting, a middle power might need to make a **strategic contribution that gives allied great powers a reason to provide access** to leading security-relevant models, participation in the compute supply chain; and that reason should be compelling enough that great powers can’t unilaterally leverage the middle powers’ dependency on their models and compute.
## **Here’s What I Think Middle Powers Should Do**
To start, **I don’t think the way to do that is for middle powers to get ambitious about building AI models themselves**. The point has been made elsewhere — in short: In almost all middle powers, the infrastructural gap to the US and China is huge: in terms of energy, compute, and talent, and competition for the latter two is at an all-time high. This is made worse by the shaky economic situation and subsequent investment limitations of many of these powers, and the comparative lack of venture capital that could alleviate that. If [catching up](https://www.bruegel.org/policy-brief/catch-us-or-prosper-below-tech-frontier-eu-artificial-intelligence-strategy#:~:text=The%20EU%20could%20gradually%20build,for%20patents%20and%20skilled%20researchers.) was an option, it would obviously be advisable – but currently, it does not seem realistic.
[![](https://substackcdn.com/image/fetch/$s_!uz0y!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1cb939d2-0806-40f5-8f9d-362f31113c62_1690x394.png)](https://substackcdn.com/image/fetch/%24s_%21uz0y%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/1cb939d2-0806-40f5-8f9d-362f31113c62_1690x394.png)
That [headline](https://www.euractiv.com/section/tech/news/us-dwarfs-eu-ai-infrastucture-push-with-500-billion-investment/) illustrates the depth of misery well.
With [recent news](https://arxiv.org/abs/2501.12948), you might think that open source can bail you out after all — but that seems risky to count on: Fast-following tends to be easier early in paradigms, securitisation makes top-tier proliferation through open source less likely, and the shift to inference scaling places large infrastructural requirements even on the use of open-source models.
So what should middle powers do instead? In short: Be valuable on bottlenecks that arise when intelligence becomes cheaper and cheaper. Tyler Cowen [says](https://www.dwarkeshpatel.com/p/tyler-cowen-4): “[Massive increases in intelligence] just means the other constraints in your system become a lot more binding, that the marginal importance of those goes up”. That can be leveraged.
**On the economic level, that means scanning your economy for bottlenecks for translating AI capability gains to GDP growth. Then, make a substantial contribution to widening them.**
As to what these bottlenecks will be: That’s a whole area of contentious research I hope to see much more of. For starters, reasonable speculation likely converges on some general takeaways: Don’t over-index on knowledge economy services that can likely be replaced or moved to the great powers at low costs (instead of enhanced). Instead, sit at intersections with natural limitations to AI deployment & automation. Much more has been said [elsewhere](https://inferencemagazine.substack.com/p/how-much-economic-growth-from-ai) on what that might be. Generally, the production of (often physical) **goods either upstream or downstream from AI use seems constraining and relevant**. Specific versions of leveragable bottlenecks to AI-powered growth might be:
* [Strategic elements](https://www.rand.org/pubs/perspectives/PEA3776-1.htmlhttps%3A//www.rand.org/pubs/perspectives/PEA3776-1.html) of the compute supply chain, from raw materials to lithography — the Netherlands and Taiwan are the obvious examples.
* Novel & privileged [sources of data](https://www.ft.com/content/8187a268-3494-11ea-a6d3-9a26f8c3cba4), esp. around industrial and strategic applications. Israel, Singapore, and some smaller sectors of most industrial states do well on this.
* Robotics & general embodiment; anything that ramps up the ability of AI systems to interface with the real world. South Korea and Japan do well on this.
* Industrial capacity writ large: Many avenues to the purported economic and social [benefits](https://openai.com/global-affairs/openais-economic-blueprint/) of AI run through physically building things. Someone actually has to make the RNA vaccines and superweapons. Germany and Japan are good at this, but e.g. Italy and India also have a decent share.
+ Specifically: **automated production quality** that might reduce friction for implementing AI-driven efficiency gains & shortens iteration cycles for experimenting on AI-generated proposals.
+ And: sufficiently **digitized production** that enables comprehensive and real-time data collection for AI-powered optimization, quickfire iteration & troubleshooting.
A lot of people in AI will tell you that US- and China-based capacities in all these areas will also be boosted by advanced AI, stripping middle powers of niches. The long history of comparative advantage and the starting positions suggest that’s not a given — especially when both great powers are focused on an AI race.
![](https://substackcdn.com/image/fetch/$s_!JDXu!,w_474,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F00f26819-b60c-4204-bf45-71a89e17e592_1977x1411.jpeg)![](https://substackcdn.com/image/fetch/$s_!rD58!,w_474,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F11705217-4d60-453d-ab08-eda74317d2e8_1200x1200.webp)![](https://substackcdn.com/image/fetch/$s_!LbcV!,w_474,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa61ecd99-75f0-4b76-820b-ccf4334a42e6_1200x1200.jpeg)
This is not about country specifics, and the economic policy pathways to realize this are a contentious issue themselves. But **for a short illustration**, imagine a country like Germany or Japan refocused policy efforts to get aggressive about further automating its still-substantial industrial capacity, provide the energy to scale it up quickly if needs be, and removed regulatory constraints on leveraging the troves of data that already exist – and *then* implemented highly advanced AI systems on top of that, buying intelligence at cheap market prices. I’m very optimistic that the returns to the quality and quantity of German or Japanese exports would be remarkable – and beyond most conceivable alternative plans with similar political costs. In simple summary: When getting ready for increasing AI capabilities, **you want to be ready to turn cheap intelligence into valuable growth.**
## **Here’s How That Helps on Strategy & Security**
Thinking in bottlenecks for intelligence also provides the **necessary strategic leverage.** Take the now-classic great power competition scenario for ~2028, where China and the US both [chase](https://www.theverge.com/2017/8/3/16007736/china-us-ai-artificial-intelligence) increasing AI capabilities. How does actually prevailing in that scenario realistically look like for either of the great powers? Sure, there is some notion of capability progress where the leader gets to an ASI system that does something mind-blowing like deploy nanobots or provide a skeleton key to the world’s infrastructure; but reasonable people suggest that this might be quite infeasible.
So I take it the more realistic, albeit less exciting, view is that **getting to higher and higher AI capabilities mostly serves as a [force multiplier](https://www.defenseone.com/sponsors/2024/10/solidify-bedrock-cyber-defenses-ai-force-multiplier/400124/)**: It makes your existing forces more and more effective, it helps you get a lot more out of your industrial capacity, it accelerates your R&D a lot, etc. Winning, in that sense, just means that your economy and military gets a lot more bang for its buck due to some distribution of efficiency gains. If that is true, and the great powers chase greater increases of the force multiplier, then there is comparative advantage in middle powers in contributing to the multiplicand: Increasing the basis of capacity that is made more effective and efficient by the increasingly abundant intelligence.
Practically speaking: **If your AGI makes all industrial output you apply it to many times as valuable, then any allies that add industrial capacity get you a lot further in your great power competition.** And if the US is already heavily invested into winning an AI race, it stands to reason that it won’t also effortlessly outpace its major allies on industrial capacity — it’s not *that* far ahead in terms of growth and GDP just yet. To make this clear by the inverse point: If the US only had its own industrial capacity to leverage, its models would have to be quite a bit better to make up for the gap to Chinese capacity already. And if it ever fell behind, things would look pretty dicey pretty quickly. Sure, maybe AGI helps you rapidly ramp up your capacity. But maybe it doesn’t. Or maybe you won’t win decisively. Maybe the three months of AGI-advantage don’t ramp it up enough. **Would you bet the house on that if you were sitting on the NSC?**
[![Bild](https://substackcdn.com/image/fetch/$s_!F1H4!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F80625925-3f75-43b1-a963-8738145c1546_1148x412.jpeg "Bild")](https://substackcdn.com/image/fetch/%24s_%21F1H4%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/80625925-3f75-43b1-a963-8738145c1546_1148x412.jpeg)
It’s up to the middle powers to shift the US rationale away from [this](https://www.hoover.org/research/defense-against-ai-dark-arts-threat-assessment-and-coalition-defense).
So, back to middle powers: Imagine a situation in which the US is barely ahead in a tight capability race and broader geopolitical competition, and wants to cement that position. In this situation, Germany or Japan could be able to offer a ‘superchargeable’ industry to the US to capitalize on capability gains through; countries like the Netherlands could offer even more comprehensive access and AI-powered improvements to compute supply, etc. This might be a sufficiently meaningful contribution to geopolitical competition that the US would be motivated to trade for it: By extending its frontier-powered security umbrella and guaranteeing continued access to compute and civilian model use. And if the middle powers caught up mysteriously, or AI turned out a nothingburger after all, they’d still be left with a stronger strategic position tha before. **Great powers will not extent their hand out of the goodness of their heart, so it’s best to think about what one can trade.**
## **Here’s What’s Next & How To Sell It**
In summary, here’s a **quick sketch of a roadmap for a middle power** that’s actually interested in avoiding falling behind on autopilot:
1. **Don’t get into a race you can’t win.** Get enough compute to run crucial onshore inference, that’s it.
2. **Find something in your economy** that:
1. Is not directly AI
2. Bottlenecks returns of rapid AI progress to real-world effects
3. Can be argued to have some strategic dimension in great power competition
3. **On economic policy**, strongly prioritize making these sectors competitive and ‘AI-compatible’. This can include getting some compute for inference, but should mostly not be about AI per se. Think in months, not decades. Go into debt if you have to.
4. **On foreign policy**, leverage these capabilities in negotiations with an AI great power to get good, but realistic, terms to partake in compute supply, model access and security umbrellas. Do this as early as you can – the negotiation terms will get worse as AI gets bigger.
I can’t speak to political dynamics everywhere to the same extent, but by and large I think this sort of plan has the **big advantage of a favourable political economy**. In short: All this **does not require a visible pivot to a tech- or AI-driven** economic model. It can be framed as benefitting powerful incumbents, existing political coalitions, and plausibly at-risk demographics whose transformation shocks are being preempted. On the level of general political sentiment — or ‘vibes’ —, the plan can be framed as a **culturally and politically continuous doubling down** on existing strengths, while being entirely responsive to a paradigm shift.
Unlike more directly AI-related strategies, this **roadmap avoids being characterised as speculative, risky, or captured by industry** — an historically likely fate for much policy that tries to address upcoming transformation. Add to this the plenty of individual incentives to deny the reality of disruptive AI effects, and you see how easily political resistance can be [motivated](https://antonleicht.substack.com/p/inferencepolitics). Any alternative plan for middle powers that puts a more overtly AI-focused push, whether that’s development at the frontier or downstream deployments below it, at its center is much more susceptible to these political hurdles. Given the currently very low uptake of ‘taking AI seriously’ in most of these middle powers, I think this political economy aspect matters.
**AI middle powers are in big trouble and risk being left behind** on economic benefits and security applications of frontier capabilities. While they might not catch up, they still have options: They can find sectors that benefit from rapid AI progress, equip them for that progress, and leverage them as comparative advantage in negotiations with great powers. I think this is a somewhat robust plan with a somewhat favourable political economy that might be worth considering.
Subscribe
1
Share
TopLatest
No posts
### Ready for more?
Subscribe
© 2026 Anton Leicht · [Privacy](https://substack.com/privacy) ∙ [Terms](https://substack.com/tos) ∙ [Collection notice](https://substack.com/ccpa#personal-data-collected)
[Start your Substack](https://substack.com/signup?utm_source=substack&utm_medium=web&utm_content=footer)[Get the app](https://substack.com/app/app-store-redirect?utm_campaign=app-marketing&utm_content=web-footer-button)
[Substack](https://substack.com) is the home for great culture
This site requires JavaScript to run correctly. Please [turn on JavaScript](https://enable-javascript.com/) or unblock scripts
sources/resources/leicht-safety.md
+215
View File
@@ -0,0 +1,215 @@
---
url: https://writing.antonleicht.me/p/how-ai-safety-is-getting-middle-powers
---
[![Threading the Needle](https://substackcdn.com/image/fetch/$s_!4SKU!,w_40,h_40,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fd7dff89d-5b66-4169-8ef8-eb2ec9ed94e8_1024x1024.png)](/)
# [Threading the Needle](/)
SubscribeSign in
# How AI Safety Is Getting Middle Powers Wrong
### The case for pivoting from global governance to national interests
[![Anton Leicht's avatar](https://substackcdn.com/image/fetch/$s_!FPyB!,w_36,h_36,c_fill,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F75422da7-aafa-42ab-8fa6-cf4f0df85cf0_3166x3166.jpeg)](https://substack.com/%40antonleicht)
[Anton Leicht](https://substack.com/%40antonleicht)
Jan 22, 2026
10
Share
**Geopolitical realities are shifting, and leave most of the world in a tough spot.** That much will become clearer still once the current fault lines are exacerbated by the advent of advanced AI, developed by two great powers alone. Few who take this prospect seriously care about the fate of middle powers; technologists, accelerationists, national security types have moved to US and China policy. Yet one group remains well-positioned to contribute to the middle power question: AI safety advocates.[1](#footnote-1)
We face a [Gaullist moment](https://www.cbc.ca/news/politics/mark-carney-speech-davos-rules-based-order-9.7053350) of reawakening national ambition in many middle powers. It provides the political preconditions to advocate for ambitious AI strategies the world over. The international branches of the safety movement are well-positioned to seize that moment: from their rare position of expertise on AI, and with a strong national foothold in many middle powers, they could pivot to improving the response of middle powers to the deployment and diffusion of AI technology: building misuse resilience, guarding against economic disempowerment, and making sure the world order in which advanced AI emerges is a stable one.
Subscribe
Instead, **much of the safety community is stuck answering the wrong questions.** They think middle powers can and should exert substantial influence on the *development* of frontier AI systems in the US and China – a proposition that was always contentious, but is untenable in the world as it is today. By sticking to it, they’re hurting their own credibility in middle powers that are now more attuned to their own national interest; and undermining the perception of AI safety work in the US. If safety advocates in middle powers abandoned the unrealistic pursuit of leverage and global governance, they could **pivot to making AI deployment go well in middle powers** – where this important work is tractable and neglected.
[![](https://substackcdn.com/image/fetch/$s_!KJpY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2288f183-9076-47b5-b248-884f54abe762_1000x250.png)](https://substackcdn.com/image/fetch/%24s_%21KJpY%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/2288f183-9076-47b5-b248-884f54abe762_1000x250.png)
# **A Platform In Review**
**Safety advocates have largely engaged with the middle power conversation from a US- and China-centric point of view:** they assume that critical risks emerge from the development of AI systems, and since that development happens in the US and China, their engagement in middle powers must aim at affecting these two great powers. Some readers may respond that domestic regulation does not fundamentally target development, just market entry conditions. That’s a fair argument about the legal principle, but the compelling force of the market and the infeasibility of developing wholly separate AI products for it makes it practically moot. When meeting market entry conditions for a vital market requires changing how you develop, it is in effect development-facing regulation – and in fact, this is arguably by design as the intended consequence of a ‘[Brussels Effect](https://www.governance.ai/research-paper/brussels-effect-ai)’.
The variations on this theme are many: AI safety organisations have tried to pass safety-focused regulation in the EU, UK, and elsewhere; they have lobbied middle power governments to influence the US and even China to take AI risks more seriously; they are developing tools of leverage for middle powers to strongarm the US into safety concessions; and they are seeking signatories to a multilateral treaty. The core hypothesis is always the same: maybe one can use middle powers as a lever to affect US development. I believe this thesis was always somewhat flawed, but it is now growing untenable. There are two problems with the approach.
#### ***It Doesn’t Really Work***
First, past attempts to influence development from outside are faltering. Using domestic law to [constrain foreign developers](https://scholarship.law.columbia.edu/faculty_scholarship/271/) has found its landmark example in the [EU AI Act](https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai), which is not doing too well: the broader law is a famously unpopular piece of legislation, its enforcement has been [delayed numerous times](https://www.euronews.com/my-europe/2025/11/19/european-commission-delays-full-implementation-of-ai-act-to-2027), and its application to the cutting edge of AI technology is evermore uncertain: the technology moves fast, the commission moves slowly, and the regulation is necessarily a snapshot of 2023’s best and worst assumptions about AI. The weaknesses of the overall act apply least readily to its most safety-coded aspects concerning general-purpose AI, and the safety and security chapter of the [Code of Practice](https://digital-strategy.ec.europa.eu/en/policies/contents-code-gpai) is a very effective implementation tool for them. My reservations have to do with the broader economic and geopolitical setting, where substantial future enforcement seems unrealistic, should it become actually burdensome: whenever any stipulation turns out too annoying for a frontier developer, they can quickly enlist the US administration and their own economic leverage to skirt the Code. A safetyist might say the key outcome was the transparency pathways set in place; I’m unsure how great the marginal gain over [SB-53](https://carnegieendowment.org/emissary/2025/10/california-sb-53-frontier-ai-law-what-it-does) is compared to the effort, and arguing this difference warranted all that effort seems extraordinary.[2](#footnote-2)
---
**International policy fares even worse.** I’ve written at length about international policy attempts up to [last year](https://writing.antonleicht.me/p/the-early-death-of-international), which leaves me to discuss the more recent call for [Red Lines](https://red-lines.ai/) at the UN. While safety advocates view this as an achievement, it strikes me that most powerful decisionmakers on AI haven’t even heard of it. For good reason: the UN—whether as a body or a collection of countries—has no real power it can exert on an issue if a major power doesn’t want it to. Safety advocates know this, but some view the Red Lines being in place as valuable nevertheless: once political salience arrives, they argue, the Red Lines provide a framework to hold onto. I don’t have much new to say on this – I see no way the US government lets any other country dictate what it does on AI. It’s too polarised an issue, too easy a domestic political win, and too much of a strategic imperative to exact absolute sovereignty over domestic development. A similar fate befalls the [call](https://superintelligence-statement.org/) for international prohibition of superintelligence: in both cases the burden of proof is on the safetyists to explain how this ever happens.
[![The Early Death of International AI Governance](https://substackcdn.com/image/fetch/$s_!vymH!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fa9c65657-9e64-4996-9628-25b58070fe39_1600x1222.jpeg)
#### The Early Death of International AI Governance
[Anton Leicht](https://substack.com/profile/113003310-anton-leicht)
·
March 31, 2025
[Read full story](https://writing.antonleicht.me/p/the-early-death-of-international)](https://writing.antonleicht.me/p/the-early-death-of-international)
Most have never publicly tried – with perhaps the sole exception of a recent [paper](https://asi-prevention.com/) by prominent AI safety organisations Conjecture and ControlAI, which outlines an admirably honest account of what it would take: middle powers in pursuit of a global mission, outright leveraging the threat of sabotaging critical digital infrastructure against an erstwhile ally. The paper correctly identifies the required scope, but also demonstrates the lack of feasibility: it’s an all-in strategy with no off-ramps and great rewards for defection for any of the involved middle powers, and in a politically heterogeneous setting, it seems like an outright impossible sell. If this is the best actual way to affect development, that’s bad news for the development-focused strategy.
---
**All this will grow even less effective over time.** The specific reason is that legislation leveraging the consumer market for outside effect works best the first time you try it – sneak provisions in on a low, technocratic level, slowly build up compliance frameworks and path dependencies that entrench your law and make it last even against political headwinds in the future. This is perhaps the success story of the GDPR; but with every iteration, the political headwinds happen earlier, and the entrenchment is less and less resilient to opposition from the markets you seek to regulate. The general version of this is that development-focused regulation from abroad works best in a low-salience environment, outside the direct purview of the public and political decision-makers, negotiated and accepted by mid-level technocrats who converge on substantive policy thought.
[![](https://substackcdn.com/image/fetch/$s_!ZMN7!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fadb406f3-a596-45af-9e0a-18374f90b349_1600x421.png)](https://substackcdn.com/image/fetch/%24s_%21ZMN7%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/adb406f3-a596-45af-9e0a-18374f90b349_1600x421.png)
This is the [geopolitical backdrop](https://www.nbcnews.com/tech/tech-news/us-rejects-international-ai-oversight-un-general-assembly-rcna233478) we are working with for the foreseeable future. Both OSTP Director Kratsios’ and President Trump’s comments at the UN should be required reading.
**We’ve now rapidly left this low-salience environment in two unrelated ways.** First, AI is no longer low-salience: governments around the world have identified it as a geopolitically and economically important issue, and they’re hesitating to hand it over to peripheral decisionmakers and institutions. For instance, I suspect that neither the American nor the European electorate will be particularly happy to trust the European Union on this one. Second, international technology regulation is no longer low-salience. The US administration is keeping a watchful eye on who and what compels their technology companies, and European powers are increasingly careful in choosing their battles, because while they might be gearing up for conflict on some fronts, they know they have to fear outright retaliation. I don’t think anything is turning back the clock on this – even a Democratic administration would not be enthusiastic to reaffirm the Europeans to please continue regulating their AI companies after all. The heyday of governing the US from outside its borders is over.
[![](https://substackcdn.com/image/fetch/$s_!KJpY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2288f183-9076-47b5-b248-884f54abe762_1000x250.png)](https://substackcdn.com/image/fetch/%24s_%21KJpY%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/2288f183-9076-47b5-b248-884f54abe762_1000x250.png)
#### ***In Fact, It Hurts***
**Development-focused international work is also increasingly unhelpful to the broader safetyist agenda.** In the US and many techno-optimistic countries, AI safety concerns have grown closely associated with the vehicles and alliances safety advocates have chosen abroad. Insofar as the US administration rejects international governance or the American electorate rejects European policy approaches to technology, there’s increasing opposition to the safety agenda too. Today, safety advocates in America often have to defend against the allegation that they seek to import European and internationalist notions of governance. Being branded as ‘European-style’ is not helpful in the current political environment.
The reputational harm also affects international governance more generally. AI safety has risen in political salience, even if it hasn’t kept up in political power, and what safetyists say they want sticks in decisionmakers’ minds. International governance was never going to be the main way AI gets regulated. But safetyist organisations publicly announcing that they were using national laws and international treaties to externally constrain the US has made it much harder to get treaties that would also have this incidental effect. Instead of letting a gentle Brussels effect proliferate quietly, the strategy has been broadcast loudly and led to more forceful rejections of any attempt at international governance.
---
**The focus on AI development frequently leads safety advocates to work against the narrow national interest of their respective countries.** It seeks to leverage resources of a middle power not to pursue that middle power’s unique goals, but a broader altruistic mission. That’s a problem, because it weakens the safety movement’s standing in national environments. This is especially true today, when middle power leadership frequently views its own attempts to export values through legislation as a costly mistake of the now-bygone ‘end of history’. A local safety movement whose theory of change runs through leveraging local resources for global solutions with indirect national benefits at best risks political sidelining in today’s environment of national interest. The most obvious issue is national regulation, which asks countries to spend their precious leverage against the US on frontier regulation to benefit the world. But it relates to other narrow interests too – consider how safety advocates have positioned themselves on European compute buildout and frontier model development. They’ve largely been hesitant to support this, not for the sound reason that it’s economically unwise, but for the bad reason that it might create another party to the AI race. With friends like these…
The Conjecture/ControlAI paper again is admirably honest in this. To be clear, I’m not harping on the paper itself – it’s entirely realistic in its suggestions, in that they are the only way to actually *do* what material like the superintelligence statement seems to *want*. Don’t even imagine it was enacted – just imagine it got traction and became a prominent piece of safety advocacy. It would cause great offense on all sides of the conversation. Middle power governments, strategically and fiscally overextended and desperate to retain their standing, with no spare resources to invest in global missions, would be very skeptical of the advice of a movement that wants them to sacrifice their national interests in service of a greater good.[3](#footnote-3)To the US as the target of this mission, it paints the safety movement as interested in regulating American economic activity by sneaking past the US electorate – this will correctly be read as outright offensive. Luckily, this perception hasn’t yet spread – but if the safety movement doubles down on this as some viable pathway to multilaterally affect development today, a reputation hit is only one or two media campaigns away.
---
#### ***Real Trade-Offs***
You might dismiss all this and say that the odds are long and the costs are high, but AI safety is so important that this is still the most effective work you can do in middle powers. My conviction is that this is not true, and that the economic and strategic questions are themselves of utmost importance. But even if you disagree, I think **political and financial capital for AI safety is better spent elsewhere: on advancing the national and collective AI strategy of middle powers writ large.** This directly trades off against the development-focused approach for two reasons.
The most obvious axis of trade-off is simply limited talent and funding – the safety movement is only so big, only has so many capable operatives, only so much breadth of talent pipelines, and so on, so simply adding the type of work I’ll suggest to the current portfolio seems unlikely to work.
The second, more important trade-off relates to credibility. Due to the political ramifications of endorsing safetyist domestic regulation, safety advocates lose influence on a range of other issues they’d otherwise be helpful voices on. They could be leading voices on national sovereignty, AI strategies, beneficial deployment, and downstream resilience; but they are much less than they could be, not least because they still operate in the reputational shadow of their attempts to trade off their nations’ interests in favour of the greater good. Some organisations have tried to change this perception, pivoting toward a more sovereignty-focused approach to middle power work. But the reputational effects are great enough that these organisations often operate quietly and dodge affiliation with the rest of the ecosystem. As a result, **the more absurd versions of safety middle power work gain disproportionate attention**, worsening the reputational problems even further. All this is a far cry from an AI safety ecosystem that confidently owns tractable safety-relevant issues in middle powers. They could lead this new conversation if they focused on it – and I believe they should do so.
[![](https://substackcdn.com/image/fetch/$s_!KJpY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2288f183-9076-47b5-b248-884f54abe762_1000x250.png)](https://substackcdn.com/image/fetch/%24s_%21KJpY%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/2288f183-9076-47b5-b248-884f54abe762_1000x250.png)
# **Why Middle Powers?**
**Even if middle powers will never influence AI development, I believe working on middle power policy is valuable.** I also believe this for many reasons that don’t have much to do with AI safety in the narrow sense, but instead with capturing and proliferating AI’s benefits throughout the world. But I’m making a case to safety advocates – so I’ll focus on the safety-relevant reasons to work on middle power policy today.
[![](https://substackcdn.com/image/fetch/$s_!I2Bz!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f8cebe1-3373-413f-9ef6-e25a07f68584_1280x527.png)](https://substackcdn.com/image/fetch/%24s_%21I2Bz%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/8f8cebe1-3373-413f-9ef6-e25a07f68584_1280x527.png)
‘And on AI we’re co-operating with like-minded democracies to ensure we won’t ultimately be forced to choose between hegemons and hyperscalers.’, says Canadian PM Carney in a much-lauded recent [speech](https://www.cbc.ca/lite/story/9.7053350).
---
**First, middle powers are where harms from AI misuse might manifest earliest and most dramatically**: their governments are less engaged on AI, so they might fail to adopt defensive measures in homeland and national security in time; but criminals will view rich middle power populations as targets for exploitation, terrorism, and blackmail. This makes work on strengthening misuse resilience in middle powers important and promising: if the threat were made clear to middle power governments, they’d have little choice but to invest in defensive measures with positive spillover effects. If you get the US worried about misuse, they might try to regulate the models; but the EU can’t regulate the models, so if guided well, they’d have to respond by funding defensive measures instead.
One promising area is *[differential](https://vitalik.eth.limo/general/2025/01/05/dacc2.html#1), [defensively focused](https://www.joinef.com/posts/introducing-def-acc-at-ef/) [acceleration](https://ifp.org/the-launch-sequence/)*: outpacing the development of potentially harmful AI capabilities by hastening the development of technologies that guard against these harms. This is in the interest of many middle powers, both because proliferation of harmful capabilities seems a foregone conclusion and because there is a political window opening. Many middle powers, alarmed by the ongoing rearrangement of geopolitical realities, are building out militaries and relying less on American and Chinese beneficence. The ongoing [rearmament](https://www.consilium.europa.eu/en/policies/defence-numbers/) of Europe (and, to a lesser extent, East Asian countries) provides an obvious context for ambitious projects in this vein. But they need to be framed as linked to national interest. The current pitch suffers from the safety movement’s reputation for prioritising global over national interests: no middle power wants to bankroll tech that lets US AI development speed ahead. But framed as a reaction to the uncontrollable trajectory of frontier AI development, resilience-focused innovation measures could become a highly successful part of middle power policy.
---
**Second, middle powers are where gradual disempowerment and widespread destitution seem most plausible.** In an AI great power, there are policy backstops against disempowerment: taxation and redistribution are possible, and the government can intervene. This plays out differently in a middle power – if AI rips through the workforce, moving revenue and power from domestic workers to US AI labs, there’s little they can do. Discussions about [growth divergence](https://econofact.org/factbrief/fact-check-has-the-economic-gap-between-europe-and-the-united-states-increased-in-the-past-decade) and the ‘Europoors’ are ruefully amusing at 3% growth differences, but become existential at 10+% growth differences. The short version: if a risk of ‘millions end up poor and destitute’ is substantial enough to motivate safetyists, it should motivate them to fix the economic trajectory of middle powers as it relates to AI.
[![AI, Jobs, and the Rest of the World](https://substackcdn.com/image/fetch/$s_!SEXO!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8c9a9985-a994-4284-b4a2-e602eb13efb9_1600x1087.png)
#### AI, Jobs, and the Rest of the World
[Anton Leicht](https://substack.com/profile/113003310-anton-leicht)
·
September 9, 2025
[Read full story](https://writing.antonleicht.me/p/ai-jobs-and-the-rest-of-the-world)](https://writing.antonleicht.me/p/ai-jobs-and-the-rest-of-the-world)
---
**Third, middle powers getting AI wrong can lead to a destabilised world** – making the path to superintelligence more dangerous. The most salient aspect is the threat of conflict. There’s already considerable geopolitical volatility, but it can always get worse. The asymmetrical diffusion of new strategic technologies has often triggered dormant conflicts and exacerbated existing ones: parties attack because they believe themselves at a temporary advantage, or inversely because they think the window to compete is closing. If dormant conflicts suddenly erupt as a result of jagged diffusion of advanced AI, that can quickly destabilise everything from supply chains to a shaky US-China peace.
But it doesn’t have to go to war. If middle powers feel sufficiently threatened by advanced AI – either by the economic effects or by power shifting toward AI developers – they might attempt to halt AI development. Countries with [substantial positions in the semiconductor supply chain](https://www.state.gov/pax-silica) have real leverage here. Using it would be economically suicidal – but if their next few years go badly enough, they might feel they have no other choice. It will be difficult enough to get the transition to transformative AI right under the best conditions – and the current inadequacy of middle power strategies will destabilise conditions substantially.
[![The Most Dangerous Time in AI Policy](https://substackcdn.com/image/fetch/$s_!I4jL!,w_140,h_140,c_fill,f_auto,q_auto:good,fl_progressive:steep,g_auto/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fcc4f5598-e3c3-43fa-8611-1e906cc6b0ff_1024x640.png)
#### The Most Dangerous Time in AI Policy
[Anton Leicht](https://substack.com/profile/113003310-anton-leicht)
·
March 5, 2025
[Read full story](https://writing.antonleicht.me/p/the-most-dangerous-time-in-ai-policy)](https://writing.antonleicht.me/p/the-most-dangerous-time-in-ai-policy)
---
Fourth, it follows that **contributing to national AI strategies in middle power governments should be a safetyist priority itself** – I believe even if this work reduces these governments’ immediate interest in safety-focused regulation. The above are all safetyist talking points, but they slot into – and require – a broader strategic conversation in middle powers:taking seriously the transformative potential of AI, and grappling with the geopolitical and technical implications. The questions that arise are ones safetyists can answer, and the answers will reduce substantial risks from advanced AI. Not only should answering these questions be a safetyist focus area, but making sure they’re asked as well. It’s an advantage to AI safety when middle powers think clearly about AI strategy, and so it’s worth contributing to that strategic clarity itself.
All this even opens a potential door back to the development focus – if there is some way to **bring middle powers into a position of strength, they might once again affect AI development,** serve as a real third center of gravity on the question of advanced AI. But the path to this runs through the effective and uncompromising pursuit of their national interest. It will not be reached from a position of economic and strategic weakness they are headed for on their current trajectories.
[![](https://substackcdn.com/image/fetch/$s_!KJpY!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2288f183-9076-47b5-b248-884f54abe762_1000x250.png)](https://substackcdn.com/image/fetch/%24s_%21KJpY%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/2288f183-9076-47b5-b248-884f54abe762_1000x250.png)
# **What Can You Do?**
The safety movement can contribute to answering these questions: they require expertise, awareness of AI’s scale, the ability to assemble effective policy operations, and a desire to get this right. Few other players combine these factors: geopolitical and national strategy researchers who take advanced AI seriously are rare, organisations willing to host them rarer still, and these few places lack scale and funding. On these criteria, the safety movement is a capable player, and it could start mobilising resources toward this goal.
**To the safety movement’s credit, this is already happening in some places.** European policy organisations, for instance, are doing substantial work advancing middle powers’ strategic agendas. But none of this is endorsed from the top, none of it has made its way into the mainstream view of what matters – none of it is reflected in big funding or talent pipelines or presented as a key pathway to safety-minded impact. To grapple with the reputational dimension and deploy resources at scale, a decisive pivot is needed.
---
**Existing policy organisations** would not be the main drivers of this process. Some are positioned to pivot, and pivot they should – including by clearly breaking with their past choices and proposals. The world is changing rapidly, and anyone who says ‘we get it now’ and starts pulling for the sovereign fate of middle powers will be welcomed. That said, the safety movement has gotten into trouble for quick pivots and perceived two-facedness in the past, and it’s probably not realistic for deeply entrenched organisations to reverse course completely. In fact, it’s probably good if some safety organisations with strong positions on this stay where they are: to catch the true believers in global-mission thinking about safety in middle powers, and to promote internal disagreement. It’s probably good if there are organisations to which new middle power orgs can point and say ‘we strongly disagree with them, and we represent a different approach to international safety policy’.
---
**The research ecosystem can do more.** Established scholars already work closely on national sovereignty, but until recently, their work on strategic questions has always been wrapped in layers of interpretation that contextualised it with some development-focused AI safety point. Giving these researchers freedom and encouragement to pursue these questions without a predetermined takeaway, suggesting that whatever strategic pathway they find is of value, could unlock substantial resources. These people have the ideas – make it clear it’s part of the mission and not taboo to get them out there.
---
But the even more promising aspect of the research ecosystem is **talent pipelines.** Until recently, major safety-aligned mentorship programs have mostly focused their project selection in international governance on this development-focused approach. If these programs offered a starting point for middle-power-focused researchers, I’m confident more people would choose this path. The number of bright, ambitious people who understand AI and want to help their home countries, but end up defecting to US-focused work or contorting into development-focused safety work, is staggeringly high. Give them a home.
---
Much of this **comes down to funding.** Major funders could publicly prioritise these cause areas; launch RFPs around them; and incubate organisations that tackle them. In the most ambitious version, they apply the same strategy to the middle power space that they have to the national security conversation, where safety-aligned organisations have made keystone grants to top-tier institutions and cultivated deep expertise between old-school policy hands and capable, entrepreneurial safety advocates who brought cutting-edge knowledge to enrich the discussion. I’d welcome the same for middle power work. Right now, there simply isn’t enough space to do work that grapples with AI and its strategic implications for middle powers. It’s in the safety movement’s interest to step up, and the world would be thankful for it.
[![](https://substackcdn.com/image/fetch/$s_!844s!,w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F8f28fc73-ecd5-411a-b329-a706d7c6089c_2142x345.png)](https://substackcdn.com/image/fetch/%24s_%21844s%21%2Cf_auto%2Cq_auto%3Agood%2Cfl_progressive%3Asteep/https%3A//substack-post-media.s3.amazonaws.com/public/images/8f28fc73-ecd5-411a-b329-a706d7c6089c_2142x345.png)
This is the limited extent of non-US AI policy work requested in Coefficient Giving’s headline AI governance [RFP](https://coefficientgiving.org/funds/navigating-transformative-ai/request-for-proposals-ai-governance/), for instance.
**The safety movement has the people, the institutions, and the resources.** **What it lacks is the right theory of change for middle powers.** The development-focused approach was always a long shot; today it’s actively harmful. The alternative – helping middle powers navigate AI deployment, build resilience, and avoid strategic blunders – is tractable, neglected, and would actually advance safety. The moment for that is now. Seize it with haste.
Subscribe
[1](#footnote-anchor-1)
By which I mean the broad AI safety ecosystem – funders, researchers, policy organisations that primarily focus on substantial and catastrophic risks from very advanced AI systems. It pains me to say that [this polemic](https://www.aipanic.news/p/the-ai-existential-risk-industrial) provides a good overview.
[2](#footnote-anchor-2)
Some safety advocates will say this was them making the best of a bad situation – that the AI Act would have happened anyways, and they sought to improve it. I doubt this, both because safety advocates played a substantial role in passing it and because their opposition would surely have been counterfactually influential – but it matters little for the forward-looking evaluation.
[3](#footnote-anchor-3)
Yes, I know that some safety advocates believe the reduction of existential risk is also the primary national priority of any smaller government. But even if that’s true, no one’s gotten very far in selling it.
10
Share
TopLatest
No posts
### Ready for more?
Subscribe
© 2026 Anton Leicht · [Privacy](https://substack.com/privacy) ∙ [Terms](https://substack.com/tos) ∙ [Collection notice](https://substack.com/ccpa#personal-data-collected)
[Start your Substack](https://substack.com/signup?utm_source=substack&utm_medium=web&utm_content=footer)[Get the app](https://substack.com/app/app-store-redirect?utm_campaign=app-marketing&utm_content=web-footer-button)
[Substack](https://substack.com) is the home for great culture
This site requires JavaScript to run correctly. Please [turn on JavaScript](https://enable-javascript.com/) or unblock scripts
sources/resources/national-ai-plan.md
+491
View File
@@ -0,0 +1,491 @@
---
url: https://www.industry.gov.au/publications/national-ai-plan
---
You have Javascript disabled and your user experience may not be as intended without enabling it. Please consider enabling Javascript or email us at digital [at] industry.gov.au for further assistance.
* [Skip to main content](#heading-1)
* [Skip to search](#search-website-link)
* [Skip to main website navigation](#website-navigation)
[![Australian Government, Department of Industry Science and Resources](/themes/custom/industry_theme/logo.png)](/ "Home")
## Mega menu
* [Who we are](/)
+ Entities
- [AI Safety Institute](/science-technology-and-innovation/technology/artificial-intelligence/ai-safety-institute)
- [Anti-Dumping Commission](/anti-dumping-commission)
- [Anti-Dumping Review Panel](/trade/anti-dumping-review-panel)
- [AusIndustry](/science-technology-and-innovation/industry-innovation/ausindustry)
- [Australian Industry Participation Authority](/major-projects-and-procurement/australian-industry-participation)
- [Australian Radioactive Waste Agency](/australian-radioactive-waste-agency)
- [Australian SKA Office](/science-technology-and-innovation/space-and-astronomy/ska-project-australia/australian-ska-office)
- [Australian Space Agency](https://www.space.gov.au/)
- [Australian Space Discovery Centre](https://www.space.gov.au/australian-space-discovery-centre)
- [Critical Minerals Office](/mining-oil-and-gas/minerals/critical-minerals/critical-minerals-office)
- [Industry Innovation and Science Australia](/science-technology-and-innovation/industry-innovation/industry-innovation-and-science-australia)
- [Investor Front Door](/investor-front-door)
- [Major Projects Facilitation Agency](/major-projects-and-procurement/major-projects-facilitation-agency)
- [National AI Centre](/science-technology-and-innovation/technology/artificial-intelligence/national-ai-centre)
- [National Measurement Institute](/national-measurement-institute)
- [National Offshore Petroleum Titles Administrator](https://www.nopta.gov.au/)
- [Office of Supply Chain Resilience](/trade/office-supply-chain-resilience)
- [Office of the Chief Economist](/office-chief-economist)
- [Office of the Chief Scientist](https://www.chiefscientist.gov.au/)
- [Offshore Decommissioning Directorate](/mining-oil-and-gas/oil-and-gas/offshore-decommissioning-directorate)
- [Questacon](https://www.questacon.edu.au/)
+ [Ministers](https://www.minister.industry.gov.au/)
+ [People](/people)
- [Organisation chart](/people/organisation-chart)
- [Overseas counsellors](/people/overseas-counsellors)
+ [Portfolio agencies and boards](https://www.directory.gov.au/portfolios/industry-science-and-resources)
+ [Public consultation](https://consult.industry.gov.au/)
+ [Accessing information](/accessing-information)
- [Campaigns](/accessing-information/information-and-advertising-campaigns)
- [Freedom of information](/accessing-information/freedom-information-disclosure)
- [Gifts and benefits](/accessing-information/gifts-and-benefits)
- [Grants reporting](/accessing-information/grants-administration)
- [Indexed list of files](/accessing-information/indexed-list-files)
- [Portfolio appointments and vacancies](/accessing-information/portfolio-appointments-and-vacancies)
- [Privacy impact assessments](/accessing-information/privacy-impact-assessments)
+ [Careers](/careers)
- [Current vacancies](https://industrycareers.nga.net.au/)
- [Graduates](/work-us/careers-graduates)
+ [Corporate governance](/corporate-governance)
- [Annual report](/corporate-governance/annual-reports)
- [Budget statements](/corporate-governance/budget-statements)
- [Corporate plans](/corporate-governance/corporate-plans)
- [Fraud and corruption control](/corporate-governance/fraud-and-corruption-control)
- [Public interest disclosure](/corporate-governance/public-interest-disclosure)
- [Regulatory performance](/corporate-governance/regulatory-performance)
+ [Government to government](/government-government)
- [Australian Business Licensing and Information Service](/government-government/australian-business-licensing-and-information-service)
- [business.gov.au](/government-government/businessgovau)
- [Business Grants Hub](/government-government/business-grants-hub)
- [Business Registration Service](/government-government/business-registration-service)
- [Shared Services Hub](/government-government/shared-services-hub)
- [VANguard](/government-government/vanguard)
* [Topics](/)
+ [Major projects and procurement](/major-projects-and-procurement)
- [Australian industry participation](/major-projects-and-procurement/australian-industry-participation)
- [Investor Front Door](/investor-front-door)
- [Major projects facilitation](/major-projects-and-procurement/major-projects-facilitation-agency)
+ [Manufacturing](/manufacturing)
- [Batteries](/manufacturing/batteries)
- [Green metals](/manufacturing/green-metals)
+ [Measurement](/national-measurement-institute)
- [Australia's measurement system](/national-measurement-institute/australias-measurement-system)
* [Measurement standards](/national-measurement-institute/australias-measurement-system/measurement-standards)
* [Pattern approval](/national-measurement-institute/nmi-services/pattern-approval)
* [Trade measurement](/national-measurement-institute/trade-and-industry)
- [Awards](/national-measurement-institute/about-us/annual-measurement-awards)
- [Chemical and biological measurement services](/national-measurement-institute/chemical-and-biological-measurement-services)
- [Physical measurement services](/national-measurement-institute/physical-measurement-services)
- [Testing and analytical services](/national-measurement-institute/testing-and-analytical-services)
- [Training and assessment](/national-measurement-institute/measurement-training-and-assessment)
+ [Mining, oil and gas](/mining-oil-and-gas)
- [Minerals](/mining-oil-and-gas/minerals)
* [Critical minerals](/mining-oil-and-gas/minerals/critical-minerals)
* [Critical Minerals Office](/mining-oil-and-gas/minerals/critical-minerals/critical-minerals-office)
* [Offshore minerals](/mining-oil-and-gas/minerals/regulating-offshore-mineral-exploration-and-mining)
* [Ranger Uranium Mine](/mining-oil-and-gas/minerals/regulating-ranger-uranium-mine)
- [Oil and gas](/mining-oil-and-gas/oil-and-gas)
* [Domestic gas supply](/mining-oil-and-gas/oil-and-gas/securing-australian-domestic-gas-supply)
* [Offshore oil and gas](/mining-oil-and-gas/oil-and-gas/offshore-oil-and-gas)
* [Offshore decommissioning](/mining-oil-and-gas/oil-and-gas/funding-and-investment-opportunities-offshore-decommissioning-australia)
* [Offshore Decommissioning Directorate](/mining-oil-and-gas/oil-and-gas/offshore-decommissioning-directorate)
* [Decommissioning the Northern Endeavour](/mining-oil-and-gas/oil-and-gas/offshore-oil-and-gas/decommissioning-northern-endeavour)
- [Greenhouse gas storage](/mining-oil-and-gas/offshore-greenhouse-gas-storage)
* [Acreage release process](/mining-oil-and-gas/offshore-greenhouse-gas-storage/offshore-greenhouse-gas-storage-acreage-release-process)
- [Taxes, royalties and export controls](/mining-oil-and-gas/taxes-royalties-and-export-controls-minerals-and-petroleum)
* [Combination certificates](/mining-oil-and-gas/taxes-and-royalties-minerals-and-petroleum/applying-combination-certificates-petroleum-projects)
* [Rough diamonds](/mining-oil-and-gas/taxes-and-royalties-minerals-and-petroleum/applying-export-rough-diamonds)
* [Uranium and controlled ores](/mining-oil-and-gas/taxes-and-royalties-minerals-and-petroleum/applying-export-uranium-and-controlled-ores)
+ [Radioactive waste management](/australian-radioactive-waste-agency)
+ [Science, technology and innovation](/science-technology-and-innovation)
- [Diversity in STEM](/science-technology-and-innovation/diversity-in-stem)
- [Industry innovation](/science-technology-and-innovation/industry-innovation)
* [AusIndustry](/science-technology-and-innovation/industry-innovation/ausindustry)
* [Business Research and Innovation Initiative](/science-technology-and-innovation/industry-innovation/business-research-and-innovation-initiative)
* [Critical Technologies Challenge Program](/science-technology-and-innovation/industry-innovation/critical-technologies-challenge-program)
* [Industry Innovation and Science Australia](/science-technology-and-innovation/industry-innovation-and-science-australia)
* [Industry Growth Program](/science-technology-and-innovation/industry-innovation/industry-growth-program)
* [Research and Development Tax Incentive](/science-technology-and-innovation/industry-innovation/research-and-development-tax-incentive)
- [International science collaboration](/science-technology-and-innovation/international-collaboration-science-and-research)
* [Global](/science-technology-and-innovation/international-collaboration/global-science-and-technology-diplomacy-fund)
* [Europe](/science-technology-and-innovation/international-collaboration/horizon-europe)
* [India](/science-technology-and-innovation/international-collaboration/collaborating-india-science-and-research)
* [China](/science-technology-and-innovation/international-collaboration/collaborating-china-science-and-research)
* [Vietnam](/science-technology-and-innovation/international-science-collaboration/australia-vietnam-science-technology-and-innovation-cooperation-initiative)
- [Science engagement](/science-technology-and-innovation/science-engagement)
* [Australian Space Discovery Centre](https://www.space.gov.au/australian-space-discovery-centre)
* [Measurement awards](/national-measurement-institute/about-us/annual-measurement-awards)
* [Prime Minister's Prizes for Science](/science-technology-and-innovation/science-engagement/prime-ministers-prizes-science)
- [Space and astronomy](/science-technology-and-innovation/space-and-astronomy)
* [Australian Space Agency](https://www.space.gov.au/)
* [SKA](/science-technology-and-innovation/space-and-astronomy/co-hosting-ska-telescope)
* [Optical astronomy](/science-technology-and-innovation/space-and-astronomy/optical-astronomy-australia)
- [Technology](/science-technology-and-innovation/technology)
* [Artificial intelligence](/science-technology-and-innovation/technology/artificial-intelligence)
* [AI Safety Institute](/science-technology-and-innovation/technology/artificial-intelligence/ai-safety-institute)
* [National AI Centre](/science-technology-and-innovation/technology/artificial-intelligence/national-ai-centre)
* [Quantum](/science-technology-and-innovation/technology/quantum)
* [Robotics](/science-technology-and-innovation/technology/robotics)
* [Critical technology standards](/science-technology-and-innovation/technology/critical-technology-standards)
+ [Trade](/trade)
- [Anti-dumping](/anti-dumping-commission)
- [Anti-dumping reviews](/trade/anti-dumping-review-panel)
- [Country of origin labelling](/trade/country-origin-labelling)
- [Exporting resources](/mining-oil-and-gas/taxes-royalties-and-export-controls-minerals-and-petroleum#container-7529)
- [Free trade agreements](/trade/growing-australian-industries-through-trade)
- [Reducing technical barriers](/trade/reducing-technical-barriers-trade)
- [Standards and conformance](/trade/australias-standards-and-conformance-infrastructure)
- [Steel tariff rate quotas](/trade/indonesia-steel-tariff-rate-quota-certificates)
- [Supply chain resilience](/trade/office-supply-chain-resilience)
- [Global and domestic supply chain status](/trade/global-and-domestic-supply-chain-status)
* [Publications](/publications)
* [News](/news)
* [Contact us](/contact-us)
close menu
* [Who we are](/)
BACK TO MAIN MENU
+ Entities
- [AI Safety Institute](/science-technology-and-innovation/technology/artificial-intelligence/ai-safety-institute)
- [Anti-Dumping Commission](/anti-dumping-commission)
- [Anti-Dumping Review Panel](/trade/anti-dumping-review-panel)
- [AusIndustry](/science-technology-and-innovation/industry-innovation/ausindustry)
- [Australian Industry Participation Authority](/major-projects-and-procurement/australian-industry-participation)
- [Australian Radioactive Waste Agency](/australian-radioactive-waste-agency)
- [Australian SKA Office](/science-technology-and-innovation/space-and-astronomy/ska-project-australia/australian-ska-office)
- [Australian Space Agency](https://www.space.gov.au/)
- [Australian Space Discovery Centre](https://www.space.gov.au/australian-space-discovery-centre)
- [Critical Minerals Office](/mining-oil-and-gas/minerals/critical-minerals/critical-minerals-office)
- [Industry Innovation and Science Australia](/science-technology-and-innovation/industry-innovation/industry-innovation-and-science-australia)
- [Investor Front Door](/investor-front-door)
- [Major Projects Facilitation Agency](/major-projects-and-procurement/major-projects-facilitation-agency)
- [National AI Centre](/science-technology-and-innovation/technology/artificial-intelligence/national-ai-centre)
- [National Measurement Institute](/national-measurement-institute)
- [National Offshore Petroleum Titles Administrator](https://www.nopta.gov.au/)
- [Office of Supply Chain Resilience](/trade/office-supply-chain-resilience)
- [Office of the Chief Economist](/office-chief-economist)
- [Office of the Chief Scientist](https://www.chiefscientist.gov.au/)
- [Offshore Decommissioning Directorate](/mining-oil-and-gas/oil-and-gas/offshore-decommissioning-directorate)
- [Questacon](https://www.questacon.edu.au/)
+ [Ministers](https://www.minister.industry.gov.au/)
+ [People](/people)
- [Organisation chart](/people/organisation-chart)
- [Overseas counsellors](/people/overseas-counsellors)
+ [Portfolio agencies and boards](https://www.directory.gov.au/portfolios/industry-science-and-resources)
+ [Public consultation](https://consult.industry.gov.au/)
+ [Accessing information](/accessing-information)
- [Campaigns](/accessing-information/information-and-advertising-campaigns)
- [Freedom of information](/accessing-information/freedom-information-disclosure)
- [Gifts and benefits](/accessing-information/gifts-and-benefits)
- [Grants reporting](/accessing-information/grants-administration)
- [Indexed list of files](/accessing-information/indexed-list-files)
- [Portfolio appointments and vacancies](/accessing-information/portfolio-appointments-and-vacancies)
- [Privacy impact assessments](/accessing-information/privacy-impact-assessments)
+ [Careers](/careers)
- [Current vacancies](https://industrycareers.nga.net.au/)
- [Graduates](/work-us/careers-graduates)
+ [Corporate governance](/corporate-governance)
- [Annual report](/corporate-governance/annual-reports)
- [Budget statements](/corporate-governance/budget-statements)
- [Corporate plans](/corporate-governance/corporate-plans)
- [Fraud and corruption control](/corporate-governance/fraud-and-corruption-control)
- [Public interest disclosure](/corporate-governance/public-interest-disclosure)
- [Regulatory performance](/corporate-governance/regulatory-performance)
+ [Government to government](/government-government)
- [Australian Business Licensing and Information Service](/government-government/australian-business-licensing-and-information-service)
- [business.gov.au](/government-government/businessgovau)
- [Business Grants Hub](/government-government/business-grants-hub)
- [Business Registration Service](/government-government/business-registration-service)
- [Shared Services Hub](/government-government/shared-services-hub)
- [VANguard](/government-government/vanguard)
* [Topics](/)
BACK TO MAIN MENU
+ [Major projects and procurement](/major-projects-and-procurement)
- [Australian industry participation](/major-projects-and-procurement/australian-industry-participation)
- [Investor Front Door](/investor-front-door)
- [Major projects facilitation](/major-projects-and-procurement/major-projects-facilitation-agency)
+ [Manufacturing](/manufacturing)
- [Batteries](/manufacturing/batteries)
- [Green metals](/manufacturing/green-metals)
+ [Measurement](/national-measurement-institute)
- [Australia's measurement system](/national-measurement-institute/australias-measurement-system)
* [Measurement standards](/national-measurement-institute/australias-measurement-system/measurement-standards)
* [Pattern approval](/national-measurement-institute/nmi-services/pattern-approval)
* [Trade measurement](/national-measurement-institute/trade-and-industry)
- [Awards](/national-measurement-institute/about-us/annual-measurement-awards)
- [Chemical and biological measurement services](/national-measurement-institute/chemical-and-biological-measurement-services)
- [Physical measurement services](/national-measurement-institute/physical-measurement-services)
- [Testing and analytical services](/national-measurement-institute/testing-and-analytical-services)
- [Training and assessment](/national-measurement-institute/measurement-training-and-assessment)
+ [Mining, oil and gas](/mining-oil-and-gas)
- [Minerals](/mining-oil-and-gas/minerals)
* [Critical minerals](/mining-oil-and-gas/minerals/critical-minerals)
* [Critical Minerals Office](/mining-oil-and-gas/minerals/critical-minerals/critical-minerals-office)
* [Offshore minerals](/mining-oil-and-gas/minerals/regulating-offshore-mineral-exploration-and-mining)
* [Ranger Uranium Mine](/mining-oil-and-gas/minerals/regulating-ranger-uranium-mine)
- [Oil and gas](/mining-oil-and-gas/oil-and-gas)
* [Domestic gas supply](/mining-oil-and-gas/oil-and-gas/securing-australian-domestic-gas-supply)
* [Offshore oil and gas](/mining-oil-and-gas/oil-and-gas/offshore-oil-and-gas)
* [Offshore decommissioning](/mining-oil-and-gas/oil-and-gas/funding-and-investment-opportunities-offshore-decommissioning-australia)
* [Offshore Decommissioning Directorate](/mining-oil-and-gas/oil-and-gas/offshore-decommissioning-directorate)
* [Decommissioning the Northern Endeavour](/mining-oil-and-gas/oil-and-gas/offshore-oil-and-gas/decommissioning-northern-endeavour)
- [Greenhouse gas storage](/mining-oil-and-gas/offshore-greenhouse-gas-storage)
* [Acreage release process](/mining-oil-and-gas/offshore-greenhouse-gas-storage/offshore-greenhouse-gas-storage-acreage-release-process)
- [Taxes, royalties and export controls](/mining-oil-and-gas/taxes-royalties-and-export-controls-minerals-and-petroleum)
* [Combination certificates](/mining-oil-and-gas/taxes-and-royalties-minerals-and-petroleum/applying-combination-certificates-petroleum-projects)
* [Rough diamonds](/mining-oil-and-gas/taxes-and-royalties-minerals-and-petroleum/applying-export-rough-diamonds)
* [Uranium and controlled ores](/mining-oil-and-gas/taxes-and-royalties-minerals-and-petroleum/applying-export-uranium-and-controlled-ores)
+ [Radioactive waste management](/australian-radioactive-waste-agency)
+ [Science, technology and innovation](/science-technology-and-innovation)
- [Diversity in STEM](/science-technology-and-innovation/diversity-in-stem)
- [Industry innovation](/science-technology-and-innovation/industry-innovation)
* [AusIndustry](/science-technology-and-innovation/industry-innovation/ausindustry)
* [Business Research and Innovation Initiative](/science-technology-and-innovation/industry-innovation/business-research-and-innovation-initiative)
* [Critical Technologies Challenge Program](/science-technology-and-innovation/industry-innovation/critical-technologies-challenge-program)
* [Industry Innovation and Science Australia](/science-technology-and-innovation/industry-innovation-and-science-australia)
* [Industry Growth Program](/science-technology-and-innovation/industry-innovation/industry-growth-program)
* [Research and Development Tax Incentive](/science-technology-and-innovation/industry-innovation/research-and-development-tax-incentive)
- [International science collaboration](/science-technology-and-innovation/international-collaboration-science-and-research)
* [Global](/science-technology-and-innovation/international-collaboration/global-science-and-technology-diplomacy-fund)
* [Europe](/science-technology-and-innovation/international-collaboration/horizon-europe)
* [India](/science-technology-and-innovation/international-collaboration/collaborating-india-science-and-research)
* [China](/science-technology-and-innovation/international-collaboration/collaborating-china-science-and-research)
* [Vietnam](/science-technology-and-innovation/international-science-collaboration/australia-vietnam-science-technology-and-innovation-cooperation-initiative)
- [Science engagement](/science-technology-and-innovation/science-engagement)
* [Australian Space Discovery Centre](https://www.space.gov.au/australian-space-discovery-centre)
* [Measurement awards](/national-measurement-institute/about-us/annual-measurement-awards)
* [Prime Minister's Prizes for Science](/science-technology-and-innovation/science-engagement/prime-ministers-prizes-science)
- [Space and astronomy](/science-technology-and-innovation/space-and-astronomy)
* [Australian Space Agency](https://www.space.gov.au/)
* [SKA](/science-technology-and-innovation/space-and-astronomy/co-hosting-ska-telescope)
* [Optical astronomy](/science-technology-and-innovation/space-and-astronomy/optical-astronomy-australia)
- [Technology](/science-technology-and-innovation/technology)
* [Artificial intelligence](/science-technology-and-innovation/technology/artificial-intelligence)
* [AI Safety Institute](/science-technology-and-innovation/technology/artificial-intelligence/ai-safety-institute)
* [National AI Centre](/science-technology-and-innovation/technology/artificial-intelligence/national-ai-centre)
* [Quantum](/science-technology-and-innovation/technology/quantum)
* [Robotics](/science-technology-and-innovation/technology/robotics)
* [Critical technology standards](/science-technology-and-innovation/technology/critical-technology-standards)
+ [Trade](/trade)
- [Anti-dumping](/anti-dumping-commission)
- [Anti-dumping reviews](/trade/anti-dumping-review-panel)
- [Country of origin labelling](/trade/country-origin-labelling)
- [Exporting resources](/mining-oil-and-gas/taxes-royalties-and-export-controls-minerals-and-petroleum#container-7529)
- [Free trade agreements](/trade/growing-australian-industries-through-trade)
- [Reducing technical barriers](/trade/reducing-technical-barriers-trade)
- [Standards and conformance](/trade/australias-standards-and-conformance-infrastructure)
- [Steel tariff rate quotas](/trade/indonesia-steel-tariff-rate-quota-certificates)
- [Supply chain resilience](/trade/office-supply-chain-resilience)
- [Global and domestic supply chain status](/trade/global-and-domestic-supply-chain-status)
* [Publications](/publications)
* [News](/news)
* [Contact us](/contact-us)
Search this website
Search
## Navigate
* Explore the digital publication
+ [National AI Plan](/publications/national-ai-plan)
+ [Acknowledgement of country](/publications/national-ai-plan/acknowledgement-country)
+ [Ministers' foreword](/publications/national-ai-plan/ministers-foreword)
+ [Introduction](/publications/national-ai-plan/introduction)
+ [The National AI Plan on a page](/publications/national-ai-plan/national-ai-plan-page)
+ [Capture the opportunities](/publications/national-ai-plan/capture-opportunities)
+ [Spread the benefits](/publications/national-ai-plan/spread-benefits)
+ [Keep Australians safe](/publications/national-ai-plan/keep-australians-safe)
+ [References](/publications/national-ai-plan/references)
* Explore this publication
+ [National AI Plan](/publications/national-ai-plan)
+ [Acknowledgement of country](/publications/national-ai-plan/acknowledgement-country)
+ [Ministers' foreword](/publications/national-ai-plan/ministers-foreword)
+ [Introduction](/publications/national-ai-plan/introduction)
+ [The National AI Plan on a page](/publications/national-ai-plan/national-ai-plan-page)
+ [Capture the opportunities](/publications/national-ai-plan/capture-opportunities)
+ [Spread the benefits](/publications/national-ai-plan/spread-benefits)
+ [Keep Australians safe](/publications/national-ai-plan/keep-australians-safe)
+ [References](/publications/national-ai-plan/references)
[National AI Plan](/publications/national-ai-plan)
1. [Home](/)
/
2. National AI Plan
# National AI Plan
Date published:
2 December 2025
### Download or share
Downloadable files
Share this page
Downloads
[National AI Plan [pdf 1.75 MB]](/sites/default/files/2025-12/national-ai-plan.pdf)
* [Facebook](https://www.facebook.com/sharer/sharer.php?u=https%3A%2F%2Fwww.industry.gov.au%2Fpublications%2Fnational-ai-plan&quote=National%20AI%20Plan)
* [X](https://x.com/intent/tweet?text=National%20AI%20Plan&url=https%3A%2F%2Fwww.industry.gov.au%2Fpublications%2Fnational-ai-plan)
* [LinkedIn](https://www.linkedin.com/sharing/share-offsite/?url=https%3A%2F%2Fwww.industry.gov.au%2Fpublications%2Fnational-ai-plan)
* Email
### Topics
* [Science, technology and innovation](/publications?pub-topic=2945)
* [Technology](/publications?pub-topic=2963)
### Publisher
* [Department of Industry, Science and Resources](/publications?publisher=3026)
## On this page
## Introduction
The National AI Plan is the Australian Government's plan to grow the AI industry in Australia.
The plan sets out the steps the government will take to support Australia to build an AI-enabled economy that is more competitive, productive and resilient. It aims to make sure that everyone in Australia benefits from the AI opportunity, across all regions, industries and communities.
The plan has 3 goals:
* **Capture the opportunity** by building smart infrastructure, backing domestic AI capability and attracting global investment.
* **Spread the benefits** through widespread AI adoption, supporting and training Australian workers, and improved public services.
* **Keep Australians safe** with legislative and regulatory frameworks that mitigate AI harms, while promoting widespread responsible practices and international engagement that upholds Australia’s values.
## Read the plan
* ### [Acknowledgement of country](/publications/national-ai-plan/acknowledgement-country)
Our department recognises the First Peoples of this Nation and their ongoing cultural and spiritual connections to the lands, waters, seas, skies, and communities.
* ### [Ministers' foreword](/publications/national-ai-plan/ministers-foreword)
Read a foreword by the Minister for Industry and Innovation and Science, and the Assistant Minister for Science, Technology and the Digital Economy.
* ### [Introduction](/publications/national-ai-plan/introduction)
Read the introduction to the National AI Plan.
### Our vision
### The National AI Plan on a page
[Our goals for AI in Australia](/publications/national-ai-plan/national-ai-plan-page)
* ### [Capture the opportunities](/publications/national-ai-plan/capture-opportunities)
Capturing the opportunities of AI will help industry to scale, create high-quality jobs, and let us compete on the global stage.
* ### [Spread the benefits](/publications/national-ai-plan/spread-benefits)
Every Australian should be able to benefit from AI, regardless of age, location or gender.
* ### [Keep Australians safe](/publications/national-ai-plan/keep-australians-safe)
Our approach allows us to respond quickly and effectively to emerging risks and harms.
### Appendix
* ### [References](/publications/national-ai-plan/references)
Read a list of sources for the National AI Plan.
## Download a copy of the plan
[National AI Plan [pdf 1.75 MB]](/sites/default/files/2025-12/national-ai-plan.pdf)
## More information
[The ministers launched the plan on 2 December 2025](https://www.minister.industry.gov.au/ministers/timayres/media-releases/national-ai-plan-empowering-all-australians)
[Read more about the work we are doing to shape the future of AI in Australia](/science-technology-and-innovation/technology/artificial-intelligence)
[NAIC is the government’s lead body supporting industry to unlock the economic benefits of AI](/node/93610)
[Learn about the department's work in progressing AI safety science](/node/94883)
[Read the Guidance for AI Adoption](/node/95254)
## Publication previous and next page links
National AI Plan
* [Next page:
Acknowledgement of country](/publications/national-ai-plan/acknowledgement-country "Go to next page")
## Contact us
Email
artificial.intelligence@industry.gov.au
## Explore
* [Careers](/careers)
* [People](/people)
* [Ministers](https://minister.industry.gov.au)
* [Publications](/publications)
* [News](/news)
* [Accessibility](/site-notices/accessibility)
* [Public consultation](https://consult.industry.gov.au)
* [Corporate governance](/corporate-governance)
* [Accessing information](/accessing-information)
* [Feedback and complaints](/contact-us/feedback-and-complaints)
* [Site notices](/site-notices)
* [Sitemap](/sitemap)
## Contact us at the department
Department of Industry, Science and Resources
[General Enquiries](/contact-us/general-enquiries-form)
+61 2 6213 6000
Industry House,
10 Binara Street,
Canberra
Department of Industry
ABN: 74 599 608 295
## Subscribe for updates at the department
Sign up for one or more of our email lists to receive the latest news, announcements and opportunities from our department.
[Subscribe](/contact-us/newsletters)
## Connect with us at the department
* [Follow us on LinkedIn](https://www.linkedin.com/company/department-of-industry "Follow us on LinkedIn")
* [Follow us on Facebook](https://www.facebook.com/IndustryGovAu/ "Follow us on Facebook")
* [Follow us on X](https://x.com/IndustryGovAu "Follow us on X")
* [Follow us on Youtube](https://www.youtube.com/c/IndustryGovAu "Follow us on YouTube")
* [Subscribe to our RSS feeds](/contact-us/rss-feeds "Subscribe to our RSS feeds")
## Acknowledgement of Country
Our department recognises the First Peoples of this Nation and their ongoing cultural and spiritual connections to the lands, waters, seas, skies, and communities.
We Acknowledge First Nations Peoples as the Traditional Custodians and Lore Keepers of the oldest living culture and pay respects to their Elders past and present. We extend that respect to all First Nations Peoples.
*Artwork credit: DISR Journey*, 2024 by Chern’ee Sutton
[Visit the Acknowledgement of Country page
![](/themes/custom/industry_theme/images/acknowledgement_image_2024.png)](/node/75691)
sources/resources/techpolicy-aiagency.md
+283
View File
@@ -0,0 +1,283 @@
---
url: https://techpolicy.au/aiagency
---
[![](/wp-content/themes/anu/assets/images/main-logo.png)](https://techpolicy.au)
* [About us](/about-us)
* [Our Work](https://techpolicy.au/our-work)
* [Education](/education)
* [Podcast](/podcast)
* [Blog](/thecommons)
* [News & Events](/news-and-events)
* [Contact Us](/contact-us)
# Expanding AI Sovereignty to AI Agency
AI is reshaping global power, prosperity and security but debates about AI sovereignty are often binary, conflated, lacking evidence, and disconnected from the complex trade-offs leaders face.
**TPDi’s AI Agency Tool, presented here in its final form, offers a practical solution.**
**The Tool is a structured and repeatable method to assess a nation’s AI maturity, sovereignty and agency across 103 AI capabilities, producing prioritised recommended actions.**
We applied the tool to produce **Australia’s 2025 AI Agency Assessment**: the first comprehensive, independent, evidence-based, expert-led assessment of Australia’s AI capabilities at the national level. We then mapped the **Australian Government’s 2025 National AI Plan** against the assessment.
The tool is adaptable and scalable. We invite you to apply the tool so that your country, region, sector, community or organisation to identify your agency and help to proactively shape a technology that is already shaping our world.
[Report: Expanding AI Sovereignty to AI Agency](https://techpolicy.au/wp-content/uploads/2026/06/TPDi-2026-Report_Expanding-AI-sovereignty-to-AI-agency.pdf)
[Analysis: Australia’s National AI Plan: Australia’s AI Agency Assessment](https://techpolicy.au/wp-content/uploads/2026/06/TPDi-2026-Analysis_2025-Australia-assessment-and-National-AI-Plan.pdf)
[Assessment: Australia’s 2025 AI Agency Assessment](https://techpolicy.au/wp-content/uploads/2026/06/TPDi-2026-Assessment_2025-Australia-AI-agency_C.pdf)
[![](https://techpolicy.au/wp-content/uploads/2026/06/02626-TPDI-Sovereignty-Report_cvr_FA4-scaled-800x0-c-default.jpg)](https://techpolicy.au/wp-content/uploads/2026/06/TPDi-2026-Report_Expanding-AI-sovereignty-to-AI-agency.pdf)
##
###
###
### Everyone, it seems, wants ‘AI sovereignty’. But what most need is ‘AI agency’.
The term ‘AI sovereignty’ dominates policy discussions and drives investment decisions but is used to mean everything from strategic self-reliance to cultural preservation and individual autonomy.
Its use as a binary – where AI is sovereign, or it is not – leaves most countries disempowered. There are also more practical confusions. AI is not one thing, so exactly what AI capabilities are we talking about? How do you measure them? What would sovereignty really mean in each case? Since we published our discussion paper in November 2025, [From AI Sovereignty to AI Agency](https://techpolicy.au/wp-content/uploads/2025/11/TPDi-AI-Agency-Discussion-Paper-November-2025-1.pdf), the debate has gained momentum. Binary notions of AI and sovereignty are increasingly seen as reductive in today’s strategic landscape.
### AI agency is the power of a country to shape its AI future.
The global AI supply chain is a complex global web, and power comes from the ability to shape relationships, not retreat from them. TPDi’s AI Agency Tool offers a practical solution.
**Instead of only asking if a country possesses sovereign control, the tool assesses whether a country has AI agency to steer outcomes, protect and promote national interests, and capture value in a globally connected system.**
### Sovereignty is an enduring principle.
The concept of AI agency put forward in this report does not signal a rejection, dilution or abandonment of the concept of sovereignty. On the contrary, the pursuit of AI agency is grounded in the very objectives that animate the pursuit of sovereignty. It offers a pragmatic policy approach that recognises most countries cannot, and need not, lead and control every AI capability.
## Explore the AI Agency Tool via Podcasts
Podcast
[Tech Mirror](https://techpolicy.au/podcast/ai-agency-and-mythos-5)
[AI Agency & Mythos 5](https://techpolicy.au/podcast/ai-agency-and-mythos-5)
How does TPDi’s AI Agency Tool help to bring nuance to the debate about AI dependance?
Podcast
[ABC Insiders](https://www.abc.net.au/news/2026-06-13/iob-johanna-weaver-ai/106792224)
[Is Australia ready for the AI transformation?](https://www.abc.net.au/news/2026-06-13/iob-johanna-weaver-ai/106792224)
An overview of Australia’s National AI Agency Assessment
# The AI Agency Tool
## 6 Layers and 103 Capabilities
![](https://techpolicy.au/wp-content/uploads/2026/06/Screenshot-2026-06-08-at-12.37.03-pm.png)
![](https://techpolicy.au/wp-content/uploads/2026/06/Layers-round-scaled.jpg)
**The aim is not to lead in all 103 capabilities – but, rather, to understand strengths, reduce critical dependencies, & build leverage where national advantages exist.** **The AI Agency Tool provides an evidence-based framework to do this, grounded in data rather than instinct or spin.**
## How to use the tool
**What is it?** The AI Agency Tool is a structured and repeatable method to assess a nation’s AI maturity, sovereignty and agency across 103 AI capabilities, producing prioritised recommended actions.
**Why does it matter?** The tool equips decision-makers to pursue AI agency: the capacity to steer outcomes, protect and promote national interests, and capture value in a globally connected system. Used well, the tool enables nuanced strategies, better-targeted investment, and deeper understandings of trade-offs. The tool equips leaders to identify their leverage in high agency capabilities and harness it to offset their vulnerabilities.
**Who is it for?**
* **Policymakers** of countries of all sizes and stages of AI maturity, particularly where strategic dependence is high, and choices are constrained (Australia’s application from page 14).
* **Business leaders** navigating geopolitical risk, supply chains and long-term investment decisions (uses for enterprise assessments on page 21).
* **Researchers** conducting national assessments, tracking progress over time and holding governments accountable (tool guide from page 25 and methodology from page 33).
**How does it work?** A step-by-step process to gather evidence and make assessments to both inform and analyse strategy.
⬇️ [**Report: Expanding AI Sovereignty to AI Agency**](https://techpolicy.au/wp-content/uploads/2026/06/TPDi-2026-Report_Expanding-AI-sovereignty-to-AI-agency.pdf)
![](https://techpolicy.au/wp-content/uploads/2026/06/Screenshot-2026-06-08-at-12.36.02-pm.png)
##
# Overview of the AI Agency Tool
The AI Agency Tool starts by producing a **maturity rating** for 103 AI capabilities defined in the **typology**. It then situates the traditional binary objective of *sovereign control* over those capabilities within an expanded spectrum that is fit for purpose in today’s strategic landscape. The traditional *control* framing is expanded to also consider the management of international partnerships (*access*), the importance of resilience (*choice*) and pursuit of competitive advantage (*leverage*). We call this the **sovereignty spectrum***.* An assessment of each element of the spectrum produces a **sovereignty rating**.
The maturity and sovereignty ratings, and an assessment of global scarcity, are then combined into a single view to produce an **AI agency score** – a measure of national competitive advantage for each AI capability.
**AI agency thus offers a pragmatic pathway for advancing sovereignty:** pursuing enduring principles of control and self-determination, while equipping leaders to navigate a world of international interdependence (including shared infrastructure and global supply chains) and geopolitical competition.
The visualisation below shows how sovereignty is embedded as a core element of the AI Agency Tool.
![](https://techpolicy.au/wp-content/uploads/2026/06/Overview-of-Natiional-AI-Agency-Assessment-Process-scaled-e1780888476135.jpg)
**The tool is adaptable and scalable. The right mix of strategic capabilities will differ country to country. We invite you to apply the tool so that your country, region, sector, community or organisation.**
**A digital version of the tool is coming soon: watch this space.**
# The Tool in Practice: Australia’s 2025 AI Agency Assessment
![](https://techpolicy.au/wp-content/uploads/2026/06/Screenshot-2026-06-08-at-12.36.35-pm.png)
##
## Strategic silences in the National AI Plan align with areas of low national agency.
**Not all omissions from the plan should be viewed as gaps.** For many capabilities the assessment’s recommended action is ‘maintain and monitor’. Many of the areas receiving limited attention in the plan fall into this category. They are capabilities where Australia currently has low maturity and low agency, or where government intervention is less necessary. This includes:
* accelerator manufacturing
* frontier model development
* most forms of private-sector AI capability.
The government has reasonably adopted a targeted approach in the plan, focusing public investment on what the government has assessed to be foundational capabilities while allowing the private sector to lead where appropriate.
## There is untapped potential in Australia’s highest agency capabilities.
**While the plan establishes strong foundations, it does not fully capitalise on all of Australia’s areas of highest agency – the capabilities in which Australia has competitive advantage.** The assessment identifies opportunities to better leverage Australia’s strengths in critical minerals, strategic data assets and model development in computer vision. These are areas where Australia already possesses very high agency and which could be used more deliberately to strengthen national capability, address weaknesses and close critical gaps elsewhere in the AI ecosystem, while increasing Australia’s international leverage.
*Australia’s 2025 AI Agency Assessment* is the first application of the AI Agency Tool. It assesses Australia’s AI maturity, sovereignty, agency (as at December 2025), and produces recommended actions across 103 capabilities. We then mapped the Australian Government’s 2025 National AI Plan against the assessment.
## Australia’s highest areas of agency
The assessment found Australia has 8 capabilities that fall within the highest band: very high agency.  This includes:
* our rich endowment in strategic and critical minerals
* 5 domain specific datasets (medical, geospatial, environment and resources, demographic and infrastructure)
* our expertise in developing computer vision models
* our proven impact in international engagement (influence and norm shaping).
## Every significant commitment in the National AI Plan aligns with the assessment’s recommendations.
Governments have finite resources and must make tough decisions about what to prioritise. **It is noteworthy that the plan is strongly aligned with the assessment’s findings on where Australia should leverage, build or maintain agency.** The plan’s major commitments focus on areas where Australia already possesses meaningful maturity and agency. This includes:
* data centres and supporting infrastructure
* public cloud
* general AI applications
* government and small to medium enterprise (SME) adoption
* international engagement.
Leaning into these capabilities harnesses existing strengths while also offering enabling benefits across the whole ecosystem.
The assessment found only 2 capabilities in which Australia has the lowest level of agency, these being manufacturing and packaging of accelerators (AI chips).
##
## There are critical gaps to close
**The assessment also identifies several globally scarce capabilities that are important to the public interest but remain underdeveloped in Australia – these capabilities should be the focus of the next wave of government prioritisation.** This includes:
* public sector and public interest compute infrastructure
* key data lifecycle management capabilities (such as copyright, sourcing, validation and annotation)
* culturally and nationally inclusive models
* discerning, inclusive, and trusted AI adoption
* general AI literacy
* several specialised AI skills
* regulatory and oversight capability.
Addressing these gaps would strengthen Australia’s ability to support innovation, research, public services and national resilience, while ensuring AI systems reflect Australian values, cultures and identities and, perhaps most importantly, ensure that the benefits of AI are widely distributed.
⬇️ [Assessment: Australia’s 2025 AI Agency Assessment](https://techpolicy.au/wp-content/uploads/2026/06/TPDi-2026-Assessment_2025-Australia-AI-agency_C.pdf)
⬇️ [Analysis: Australia’s National AI Plan cf: Australia’s AI Agency Assessment](https://techpolicy.au/wp-content/uploads/2026/06/TPDi-2026-Analysis_2025-Australia-assessment-and-National-AI-Plan.pdf)
# Read the Analysis of Australia’s 2025 Assessment and National AI Plan
# Acknowledgments
![](https://techpolicy.au/wp-content/uploads/2026/01/1759125649931-1000x1000-c-default.webp)
![](https://techpolicy.au/wp-content/uploads/2026/01/1759125648635-1000x1000-c-default.webp)
![](https://techpolicy.au/wp-content/uploads/2026/01/1763954271110-1000x1000-c-default.webp)
## **Sovereignty was never ceded**
TPDi acknowledges the Ngunnawal and Ngambri people who are the Traditional Owners of the land upon which this report was prepared in Canberra, Australia. We pay our respects to Elders past and present.
The authors affirm that sovereignty has never been ceded by First Nations peoples living on the continent now known as Australia. We recognise Indigenous Sovereignty as enduring and inherent, as well as fundamentally different to the new concept of ‘AI sovereignty’ (to which this report is responding).
Any national conversation about AI should reinforce, rather than distract from, the distinct and profoundly important conversations about Indigenous Sovereignty and governance. The AI Agency Tool emphasises the importance of pursuing both Indigenous Sovereignty and AI agency in parallel. Indeed, they can be understood as reinforcing policy goals.
The tool explicitly highlights the importance of Indigenous Sovereignty. First Nations interests and sensitivities are profound and intersecting across all 103 AI capabilities identified in the typology. A review of the typology through lenses of cultural priority, sovereignty beyond a nation-state framing, lived experience, and the risk of structural harm, revealed that at least 38 AI capabilities have particular implications for First Nations peoples. The tool frames inclusive, empowering and rights-respecting approaches to these capabilities as indicators of AI maturity.
The concept of AI agency will look different in practice for different stakeholders, including First Nations peoples compared to governments and organisations. The tool is intended to be adopted and applied by different groups, shaped by their distinct perspectives and priorities in implementation. In this way, it acts as both an assessment and a transparency tool.
This approach is intended to support the empowerment of Indigenous voices, leadership and agency in future-proofing communities and shaping Australia’s future.
See pages 4 and 46 in the Report for further discussion on these issues.
## A collective endeavour
### Authors
Zoe Jay Hawkins, Co-Founder and Deputy Executive Director, TPDi; Johanna Weaver, Co-Founder and Executive Director, TPDi; Rebecca Razavi, Visiting Policy Fellow, Oxford Internet Institute; Executive Director, UNSW Public Policy Institute; Meredith Hodgman, Head of Strategy and Engagement, TPDi; Vili Lehdonvirta, Professor of Technology Policy, Aalto University; Senior Fellow, Oxford Internet Institute; Mercedes Page, Visiting Scholar, Massachusetts Institute of Technology Sloan School of Management and Murray Lab. With invaluable contributions from Dorina Wittmann, Heidi Brockway and Helen Portillo-Castro.
### Peer Reviewers
We are grateful for the invaluable feedback we received throughout the project from our peer reviewers (affiliations at time of review): Andrew Brodie, Eva Hopewell and Todd Phillips (Deadly Coders), Belinda Dennett (AirTrunk), Brendan Hopper (Commonwealth Bank of Australia), Clare Beaton-Wells (United Nations Youth Australia), Christina Wiremu-Brook (Kuria), Dave Lemphers (Maincode), David Masters (Atlassian), Dot West, Lyndon Ormand-Parker and Daniel Featherstone (First Nations Digital Inclusion Advisory Group), Prof Elanor Huntington (CSIRO), Ian Opperman (ServiceGen), Jamie Morse (Macquarie Technology Group), Josh Griggs and Pauline Fetaui (Australian Computer Society), Kate Conroy (Queensland University of Technology), Kyle Turner (Paul Ramsay Foundation), Lee Hickin (National AI Centre), Liam Carroll, Bill Simpson Young, Alistair Reid and Tiberio Caetano (Gradient Institute), Mark Stickells (Pawsey Supercomputing Research Centre), Nadia Court, Tanya Saad and Anna Gurevich (Semiconductor Sector Service Bureau), Rosie Hicks (Australian Research Data Commons), Sally Ann Williams (Australian Academy of Technological Sciences & Engineering), Scott Winch (Sax Institute), Simon Kriss (Sovereign AI Australia), Simon Spencer (Trideca), Dr Sue Keay (Robotics Australia Group), Taylor Dee Hawkins (Foundations for Tomorrow), Dr Tobias Feakin (Protostar Strategy), Tim Carton (CDC Data Centres), and Tim Moriarty (Muzo.ai).
### Consultations
TPDi gratefully acknowledges the contributions of experts and organisations who participated in workshops, surveys, interviews and peer reviews that informed this project. Their participation reflects wide engagement across Australia’s technology, research, policy and civil society communities.
In September 2025, TPDi brought together over 250 individuals and 187 organisations across five cities, and online, to survey and map Australia’s AI strengths and define our path forward.
A special acknowledgment to the industry and research bodies that helped amplify the consultation process, with executive participation and opportunities for their members to contribute, including the Australian Academy of Science (AAS), Australian Computer Society (ACS), Australian Council of Learned Academies (ACOLA), Electronic Frontiers Australia Inc, IoT Alliance Australia (IoTAA), Digital Rights Watch (DRW), the Kingston AI Group, ARC Centre of Excellence for Automated Decision-Making and Society (ADM+S), Gradient Institute, Science and Technology Australia (STA), Tech Council of Australia (TCA), and the UNSW AI Institute.
**A full list of the individuals and organisation who participated in the consultations in an Annex of the Report – with thanks from team TPDi to each and every person.**
### Building on previous work
The Final Report, Assessment and Analysis presented on this webpage build on the [Discussion Paper,](https://techpolicy.au/wp-content/uploads/2025/11/TPDi-AI-Agency-Discussion-Paper-November-2025-1.pdf) and the [Draft Australia AI Capability Assessment](https://techpolicy.au/wp-content/uploads/2026/04/TPDi-AI-Agency-Tool-Australian-Stocktake-Discussion-Draft-Web-Doublespread-CC.pdf), published by TPDi in November 2025.
This work builds on longstanding research by TPDi Co-Founder [Zoe Jay Hawkins](https://www.linkedin.com/in/zoe-hawkins/) and colleagues [Vili Lehdonvirta](https://www.linkedin.com/in/vililehdonvirta/) and [Bóxī Wú](https://www.linkedin.com/in/boxiwu/) ([Oxford Internet Institute, University of Oxford](https://www.linkedin.com/company/oxford-internet-institute/) & [Aalto University](https://www.linkedin.com/company/aalto-university/)) on compute sovereignty, which has been featured by The New York Times, the Australian Financial Review, TIME Magazine, BBC, The Economist, The Financial Times, Forbes, and POLITICO (among others).
⬇️ [Read the article in the New York Times](https://www.nytimes.com/interactive/2025/06/23/technology/ai-computing-global-divide.html)
⬇️ [Read the research paper: Hawkins, ZJ., Lehdonvirta, V., and Wu, B. (2025). AI Compute Sovereignty: Infrastructure Control Across Territories, Cloud Providers, and Accelerators.](https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5312977)
⬇️ [Read the research paper:Lehdonvirta, V., Wú, B., & Hawkins, ZJ. (2024). Compute North vs. Compute South: The Uneven Possibilities of Compute-based AI Governance Around the Globe](https://ojs.aaai.org/index.php/AIES/article/view/31683)
![](https://techpolicy.au/wp-content/uploads/2025/08/2-1-scaled-1000x1000-c-default.webp)
![](https://techpolicy.au/wp-content/uploads/2025/08/3-1-scaled-1000x1000-c-default.webp)
![](https://techpolicy.au/wp-content/uploads/2025/08/1-1-scaled-1000x1000-c-default.webp)
## A thanks to our supporters
This project was made possible by the generous support of the **Australian Computer Society** and the **Department of Industry, Science and Resources**.
**TPDi’s independence is our most valuable asset.** As a registered not-for-profit, our work is supported by external funding. We only accept funding from entities that agree to be disclosed publicly and commit to respect and promote TPDi’s independence. **TPDi does not represent the views of any of our funders; all outputs represent solely the views of TPDi and/or the authors. [Learn more](https://techpolicy.au/funders) about TPDi’s blind trust funding model and how this protects and promotes our independence.**
![](https://techpolicy.au/wp-content/uploads/2025/08/ACS-1-scaled-1000x1000-c-default.webp)
![](https://techpolicy.au/wp-content/uploads/2025/08/DISR-1-scaled-1000x1000-c-default.webp)
⬇️ [Report: Expanding AI Sovereignty to AI Agency](https://techpolicy.au/wp-content/uploads/2026/06/TPDi-2026-Report_Expanding-AI-sovereignty-to-AI-agency.pdf) | ⬇️ [Assessment: Australia’s 2025 AI Agency Assessment](https://techpolicy.au/wp-content/uploads/2026/06/TPDi-2026-Assessment_2025-Australia-AI-Agency.pdf) | ⬇️ [Analysis: Australia’s National AI Plan: Australia’s AI Agency Assessment](https://techpolicy.au/wp-content/uploads/2026/06/TPDi-2026-Analysis_2025-Australia-assessment-and-National-AI-Plan.pdf)| ⬇️ [Read the Media Release](https://techpolicy.au/news/launch)
[![](/wp-content/themes/anu/assets/images/main-logo.png)](https://techpolicy.au)
* Learn more
* [Projects](https://techpolicy.au/projects)
* [News and Events](https://techpolicy.au/new-news-and-events)
* [Podcast](https://techpolicy.au/tech-mirror-podcast)
* [Contact Us](https://techpolicy.au/contact-us)
* Follow us on
* [Follow us on LinkedIn](https://www.linkedin.com/company/techpolicydesigncentre/)
### Subscribing to TPDIs mailing list
Δ
Name(Required)
First
Last
Email
Submit
Copyright Tech Policy Design Institute 2026
* [Privacy Policy](https://techpolicy.au/privacy-policy)
* [Disclaimer](https://techpolicy.au/disclaimer)
tree.svg
+143
View File
@@ -0,0 +1,143 @@
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 560 1272" font-family="Georgia, serif" font-size="14">
<rect width="560" height="1272" fill="#fffff8"/>
<text x="280" y="82" text-anchor="middle" font-size="28" font-weight="bold" fill="#111">Choose a path</text>
<path d="M145,150 C145,180 280,180 280,209" fill="none" stroke="#a9a99f" stroke-width="13.0"/>
<path d="M426,150 C426,180 280,180 280,209" fill="none" stroke="#a9a99f" stroke-width="13.0"/>
<path d="M280,265 C280,294 157,294 157,324" fill="none" stroke="#a9a99f" stroke-width="2.6"/>
<path d="M280,265 C280,294 381,294 381,324" fill="none" stroke="#a9a99f" stroke-width="23.4"/>
<path d="M381,380 C381,410 179,410 179,439" fill="none" stroke="#a9a99f" stroke-width="13.0"/>
<path d="M381,380 C381,410 377,410 377,439" fill="none" stroke="#a9a99f" stroke-width="10.4"/>
<path d="M377,495 C377,524 146,524 146,554" fill="none" stroke="#a9a99f" stroke-width="6.2"/>
<path d="M377,495 C377,524 406,524 406,554" fill="none" stroke="#a9a99f" stroke-width="1.3"/>
<path d="M377,495 C377,582 161,582 161,669" fill="none" stroke="#a9a99f" stroke-width="2.9"/>
<path d="M406,610 C406,640 410,640 410,669" fill="none" stroke="#a9a99f" stroke-width="1.3"/>
<path d="M161,725 C161,754 403,754 403,784" fill="none" stroke="#a9a99f" stroke-width="2.1"/>
<path d="M161,725 C161,754 168,754 168,784" fill="none" stroke="#a9a99f" stroke-width="1.0" stroke-dasharray="5 5"/>
<path d="M403,840 C403,870 410,870 410,899" fill="none" stroke="#a9a99f" stroke-width="1.0"/>
<path d="M403,840 C403,927 377,927 377,1014" fill="none" stroke="#a9a99f" stroke-width="1.0"/>
<path d="M168,840 C168,870 165,870 165,899" fill="none" stroke="#a9a99f" stroke-width="1.0"/>
<path d="M168,840 C168,927 161,927 161,1014" fill="none" stroke="#a9a99f" stroke-width="1.0"/>
<path d="M377,1070 C377,1100 179,1100 179,1129" fill="none" stroke="#a9a99f" stroke-width="1.0"/>
<path d="M377,1070 C377,1100 410,1100 410,1129" fill="none" stroke="#a9a99f" stroke-width="1.0"/>
<rect x="194" y="280" width="50" height="17" fill="#fffff8"/>
<text x="219" y="292" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">stalls</text>
<rect x="289" y="280" width="83" height="17" fill="#fffff8"/>
<text x="330" y="292" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">race / deal</text>
<rect x="235" y="396" width="89" height="17" fill="#fffff8"/>
<text x="280" y="408" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">control lost</text>
<rect x="334" y="396" width="89" height="17" fill="#fffff8"/>
<text x="379" y="408" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">control kept</text>
<rect x="214" y="510" width="96" height="17" fill="#fffff8"/>
<text x="262" y="522" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">seized abroad</text>
<rect x="340" y="510" width="102" height="17" fill="#fffff8"/>
<text x="392" y="522" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">seized at home</text>
<rect x="248" y="533" width="129" height="17" fill="#fffff8"/>
<text x="312" y="545" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">power stays spread</text>
<rect x="354" y="626" width="109" height="17" fill="#fffff8"/>
<text x="408" y="638" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">no public stake</text>
<rect x="234" y="740" width="96" height="17" fill="#fffff8"/>
<text x="282" y="752" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">set our price</text>
<rect x="378" y="856" width="56" height="17" fill="#fffff8"/>
<text x="406" y="868" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">host it</text>
<rect x="347" y="878" width="96" height="17" fill="#fffff8"/>
<text x="395" y="890" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">stay supplier</text>
<rect x="138" y="856" width="56" height="17" fill="#fffff8"/>
<text x="167" y="868" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">concert</text>
<rect x="138" y="878" width="56" height="17" fill="#fffff8"/>
<text x="166" y="890" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">US bloc</text>
<rect x="234" y="1086" width="89" height="17" fill="#fffff8"/>
<text x="278" y="1098" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">public stake</text>
<rect x="362" y="1086" width="63" height="17" fill="#fffff8"/>
<text x="393" y="1098" text-anchor="middle" fill="#555" font-size="12.5" font-style="italic">no stake</text>
<a href="#1-prepare-before-it-looks-urgent-2026-28"><title>1. Prepare early — 2026-28: royalties, biosecurity, allies</title>
<rect x="21" y="94" width="248" height="56" rx="4" fill="#fffff8" stroke="#38761d" stroke-width="2.2"/>
<text x="145" y="119" text-anchor="middle" font-weight="bold" font-size="13">1. Prepare early</text>
<text x="145" y="135" text-anchor="middle" fill="#333" font-size="11">2026-28: royalties, biosecurity, allies</text>
</a>
<a href="#2-train-the-people-the-treaty-needs-2026-29"><title>2. Build the people — 2026-29: safety testers, inspectors</title>
<rect x="313" y="94" width="226" height="56" rx="4" fill="#fffff8" stroke="#38761d" stroke-width="2.2"/>
<text x="426" y="119" text-anchor="middle" font-weight="bold" font-size="13">2. Build the people</text>
<text x="426" y="135" text-anchor="middle" fill="#333" font-size="11">2026-29: safety testers, inspectors</text>
</a>
<a href="#what-australia-and-new-zealand-can-do"><title>2029: the US chooses — go-slow deal with China, or race?</title>
<rect x="173" y="209" width="214" height="56" rx="4" fill="#fffff8" stroke="#888888" stroke-width="2.2"/>
<text x="280" y="234" text-anchor="middle" font-weight="bold" font-size="13">2029: the US chooses</text>
<text x="280" y="250" text-anchor="middle" fill="#333" font-size="11">go-slow deal with China, or race?</text>
</a>
<a href="#vii-boring-decade-about-10"><title>VII. Boring decade — ~10%</title>
<rect x="79" y="324" width="157" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="157" y="349" text-anchor="middle" font-weight="bold" font-size="13">VII. Boring decade</text>
<text x="157" y="365" text-anchor="middle" fill="#333" font-size="11">~10%</text>
</a>
<a href="#i-doom-about-50"><title>RISK: AI escapes control — ~75% racing, ~25% in the deal</title>
<rect x="280" y="324" width="201" height="56" rx="4" fill="#fffff8" stroke="#990000" stroke-width="3.6"/>
<text x="381" y="349" text-anchor="middle" font-weight="bold" font-size="13">RISK: AI escapes control</text>
<text x="381" y="365" text-anchor="middle" fill="#333" font-size="11">~75% racing, ~25% in the deal</text>
</a>
<a href="#i-doom-about-50"><title>I. Doom — ~50%</title>
<rect x="104" y="439" width="150" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="179" y="464" text-anchor="middle" font-weight="bold" font-size="13">I. Doom</text>
<text x="179" y="480" text-anchor="middle" fill="#333" font-size="11">~50%</text>
</a>
<a href="#ii-someone-elses-empire-about-25"><title>RISK: power seized — one actor holds the AI</title>
<rect x="298" y="439" width="157" height="56" rx="4" fill="#fffff8" stroke="#990000" stroke-width="3.6"/>
<text x="377" y="464" text-anchor="middle" font-weight="bold" font-size="13">RISK: power seized</text>
<text x="377" y="480" text-anchor="middle" fill="#333" font-size="11">one actor holds the AI</text>
</a>
<a href="#ii-someone-elses-empire-about-25"><title>II. Someone else&#x27;s empire — ~25% — seized abroad</title>
<rect x="42" y="554" width="208" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="146" y="579" text-anchor="middle" font-weight="bold" font-size="13">II. Someone else&#x27;s empire</text>
<text x="146" y="595" text-anchor="middle" fill="#333" font-size="11">~25% — seized abroad</text>
</a>
<a href="#ix-homegrown-autocracy-free-beer-and-rugby-for-life"><title>RISK: power grabbed at home — seized in our own country</title>
<rect x="295" y="554" width="223" height="56" rx="4" fill="#fffff8" stroke="#990000" stroke-width="3.6"/>
<text x="406" y="579" text-anchor="middle" font-weight="bold" font-size="13">RISK: power grabbed at home</text>
<text x="406" y="595" text-anchor="middle" fill="#333" font-size="11">seized in our own country</text>
</a>
<a href="#3-ask-for-terms-before-signing-2029"><title>3. Sign, or set our price? — the deal reaches us, 2029</title>
<rect x="53" y="669" width="216" height="56" rx="4" fill="#fffff8" stroke="#b8860b" stroke-width="2.2"/>
<text x="161" y="694" text-anchor="middle" font-weight="bold" font-size="13">3. Sign, or set our price?</text>
<text x="161" y="710" text-anchor="middle" fill="#333" font-size="11">the deal reaches us, 2029</text>
</a>
<a href="#ix-homegrown-autocracy-free-beer-and-rugby-for-life"><title>IX. Homegrown autocracy — citizens stop being needed</title>
<rect x="313" y="669" width="194" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="410" y="694" text-anchor="middle" font-weight="bold" font-size="13">IX. Homegrown autocracy</text>
<text x="410" y="710" text-anchor="middle" fill="#333" font-size="11">citizens stop being needed</text>
</a>
<a href="#6-choose-a-side-if-the-deal-collapses-2030s"><title>6. If the deal collapses — US bloc, concert, or neutral?</title>
<rect x="68" y="784" width="201" height="56" rx="4" fill="#fffff8" stroke="#b8860b" stroke-width="2.2"/>
<text x="168" y="809" text-anchor="middle" font-weight="bold" font-size="13">6. If the deal collapses</text>
<text x="168" y="825" text-anchor="middle" fill="#333" font-size="11">US bloc, concert, or neutral?</text>
</a>
<a href="#4-host-allied-compute-2030-32"><title>4. Host compute here? — 2030-32</title>
<rect x="313" y="784" width="179" height="56" rx="4" fill="#fffff8" stroke="#b8860b" stroke-width="2.2"/>
<text x="403" y="809" text-anchor="middle" font-weight="bold" font-size="13">4. Host compute here?</text>
<text x="403" y="825" text-anchor="middle" fill="#333" font-size="11">2030-32</text>
</a>
<a href="#viii-middle-power-concert"><title>VIII. Middle-power concert</title>
<rect x="57" y="899" width="216" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="165" y="931" text-anchor="middle" font-weight="bold" font-size="13">VIII. Middle-power concert</text>
</a>
<a href="#v-allied-compute-host"><title>V. Allied compute host</title>
<rect x="317" y="899" width="187" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="410" y="931" text-anchor="middle" font-weight="bold" font-size="13">V. Allied compute host</text>
</a>
<a href="#vi-garrison-ally"><title>VI. Garrison ally</title>
<rect x="86" y="1014" width="150" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="161" y="1046" text-anchor="middle" font-weight="bold" font-size="13">VI. Garrison ally</text>
</a>
<a href="#5-keep-a-public-stake-in-the-machine-economy-2031-34"><title>5. Tax it, own a share? — 2031-34</title>
<rect x="280" y="1014" width="194" height="56" rx="4" fill="#fffff8" stroke="#b8860b" stroke-width="2.2"/>
<text x="377" y="1039" text-anchor="middle" font-weight="bold" font-size="13">5. Tax it, own a share?</text>
<text x="377" y="1055" text-anchor="middle" fill="#333" font-size="11">2031-34</text>
</a>
<a href="#iii-dividend-commonwealth"><title>III. Dividend commonwealth</title>
<rect x="71" y="1129" width="216" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="179" y="1161" text-anchor="middle" font-weight="bold" font-size="13">III. Dividend commonwealth</text>
</a>
<a href="#iv-quarry-economy"><title>IV. Quarry economy — drift default</title>
<rect x="331" y="1129" width="157" height="56" rx="22" fill="#fffff8" stroke="#3d85c6" stroke-width="2.2"/>
<text x="410" y="1154" text-anchor="middle" font-weight="bold" font-size="13">IV. Quarry economy</text>
<text x="410" y="1170" text-anchor="middle" fill="#333" font-size="11">drift default</text>
</a>
</svg>
Side by Side

After

Width:  |  Height:  |  Size: 12 KiB