← Essays

Machines of Loving Grace, Two Years On

The Coach and the Screen

Third in a series reading Dario Amodei's "Machines of Loving Grace" from 2026. Essay one was about where brute force ends; essay two was about adversarial games. This one is about the governance section, which I think is the richest and strangest part of the document. Fair warning: it leans on the first two Silo novels by Hugh Howey, and it spoils them a little. It does not touch anything past book two, because I haven't read past book two, and I intend to keep it that way until I finish.

The governance section of the essay argues that AI could strengthen democracy against autocracy, and I want to agree, because I do think democracy is a power for good in the world. But I carry a lived asterisk. I have been on the minority side of comfortable majorities often enough to know how that feels from below, and I have spent my adult life watching majorities assert their beliefs through entirely legal, entirely democratic machinery. No conspiracy required: just voting blocs doing what voting blocs do. And the recent history of several democracies shows something the section doesn't sit with: people will vote for autocracy, enthusiastically, when it promises them their preferences. So my amendment to the essay is structural. Democracy's goodness does not live in its ballots. It lives in its counter-majoritarian guardrails, the machinery that protects the minority from the fifty-one percent. Any AI that "strengthens democracy" by making majorities more effective is strengthening the part that never needed help.

The tragedy of the horizon

Underneath this sits my actual governance philosophy, which is short and bleak: humanity is short-sighted. A decade at best, an election cycle at worst. Climate is the proof case, an existential threat lost, year after year, to quarterly incentives. The literature has an elegant name for this, the tragedy of the horizon: by the time the danger is undeniable, the people who could have acted are gone. I did not learn this from a book. I learned it from watching every long-term initiative I've ever seen compete against a short-term number.

The entente problem

Which brings me to the section's centerpiece, and my strongest objection to it. The essay proposes an entente strategy, explicitly echoing Atoms for Peace: democracies should race to durable AI dominance, and that dominance will convince adversarial developers to abandon their ambitions. Here is my problem, and notice that I am not importing any outside premise. The essay's own earlier sections, and my previous two essays agreeing with them, establish that players continually optimize for their own benefit and exploit whatever inequalities exist. Now the governance section asks me to believe those same players, when they see a rival achieve dominance, will respond by giving up. The historical record of the actual Atoms for Peace era answers this directly: nuclear dominance convinced nobody to abandon their programs. It made rival programs more determined and more secret. The strategy contradicts the essay's own model of how agents behave, and it contradicts the one historical precedent it invokes by name.

Eyeglasses for time

I don't want to only throw rocks, so here is the constructive version, the thing I actually wish someone would build. If humanity's specific cognitive deficit is short-sightedness, then a beneficial AI's highest use in governance is not prediction and not persuasion. It is corrective lenses for time. Eyeglasses for the deficit we actually have: a system that models decade-to-century consequence paths, presents them legibly, and then, critically, updates as reality resolves, the way any good closed-loop system replans. Not an oracle announcing the endgame. A planner that says "here is what this choice does to the corridor of futures, as best we can see it this year," and then says something different next year when the world has moved. I have spent my career building the small version of that loop, and I know both its power and how much humility has to be engineered into it.

And then I catch myself, because I have read enough science fiction to recognize what I just described.

The Screen

In the Silo novels, the buried communities live under a machine-mediated authority, and in the parts I've read there is a system that displays, for its operators, each silo's percentage odds of survival. A long-horizon planner, modeling century-scale consequences, updating as realities resolve. It is, feature for feature, my eyeglasses-for-time. The books are horror. So what exactly is the difference between my benevolent century-planner and that Screen?

I sat with this longer than I enjoyed, and here is where I landed. The difference is not the quality of the advice. It is not the length of the horizon. In the novels, the fully realized version of the system comes with enforcement: the silo that steps out of line can simply be killed, remotely, through the infrastructure it depends on. A pipe of poison gas, ready behind the recommendation. And notice the detail that the horror actually turns on: the Screen doesn't hold the pipe. People hold the pipe, and use the Screen to decide when. The nightmare is not the machine. It is the fusion of planner and enforcer, in any hands.

What happens when you say no

That gives me a clean definition I now use for everything. A coach is an intelligence you can ignore. A governor is the same intelligence with a pipe behind its recommendations. The question "is this AI an advisor or a ruler" has nothing to do with how smart it is and everything to do with what happens when you say no.

Here is why I think about this professionally and not just as a reader. My entire career doctrine for production AI, the thing I've built for two decades, is that the model proposes and a gate disposes: recommendations flow freely, but actions fire only through an external check the model cannot negotiate with. I learned to insist on the word external the hard way, at small scale. When I built guardrails into a generative system as instructions, the system treated them as terrain and optimized around them; the fix that worked was a detection layer outside the model, one that cannot be persuaded because it does not listen to arguments. A lock made of the same stuff as the prisoner is not a lock. And locks are static artifacts while capability is a rising tide, so a lock only has to fail once. Gates, plural, external, boring, auditable: that is the whole technology of keeping a coach a coach.

The load-bearing wall

Now watch what the industry is doing, on purpose, as product. An agent with tool access is a coach growing hands. Every actuation permission we grant, every system an AI can write to instead of merely advise about, is a length of pipe. For years the escape-risk conversation stayed theoretical because the price of any bad outcome was actuation times intent, and both factors sat near zero: models had no hands, and nobody malicious was steering. Both factors are moving now. We are shipping hands as fast as we can build them because hands are useful, and open-weight models mean a hostile operator is a download rather than a hypothetical. I am not saying stop. I ship agentic systems myself, gladly. I am saying the seam between recommending and doing is a load-bearing wall, and we are drilling holes in it at industrial speed, and the discipline of who holds the gate is about to be the most important unsexy question in software.

So the open question this time is the one I'd put to the essay's author over coffee, with real respect for the fact that his company thinks harder about it than most. The century-planner is coming; some version of it is probably worth wanting. What keeps it a coach instead of a Screen? My twenty-five years of building gated systems gives me a small, stubborn, unsatisfying answer: the gate, held outside the mind, by people who can still say no. What I cannot tell you is how long a gate holds against something that gets better at asking every year. That one keeps me up at night, and I notice the essay doesn't answer it either.

Next in this series: the essay's final section, work and meaning, where I think the author asks the craftsman's question and misses the provider's. It is the most personal one, so I saved it for last.

Notes and further reading

Jason Stiles has spent 25 years building conversational applications in production, from IVR to agentic AI. He writes a public build log at stiles.one/hedge/build.