Friday, September 18

The Question We Are Not Asking About Autonomous AI

We are building machines that act on their own. The path is easy to trace: from tools that answer when asked, to systems that carry out tasks, to autonomous agents that pursue goals across long stretches of work with no human in the loop. Each step hands more of the doing, and more of the deciding, to the machine. Argue about the timeline all you like; the direction is not in dispute.

The public conversation fixes on the spectacle. The headline figures, the billion-robot forecasts, the named founders and their release dates. Those are the wrong thing to watch. The forecasts will be wrong in their particulars, as forecasts always are. What matters is the shape of the thing they are all pointing at, and it is a threshold no earlier technology ever crossed: production that can expand without a matching expansion of human effort.

For the whole of human history, making more of anything required more human effort, or at least more human decisions about whose effort to hire. The loop always ran back through people. A factory needed workers, workers needed wages, wages required employers selling to customers who earned their money working somewhere else. The circulatory system of modern life ran on one quiet fact: human effort was scarce, and because it was scarce it was valuable, and because it was valuable the people who supplied it had leverage.

Autonomous production cuts that loop. When machines carry out the work and increasingly direct it, the growth of output stops depending on the supply of human labor. This is not automation as we have known it since the spinning jenny, where each wave displaced some workers and created demand for others. It is the loosening of the economy's dependence on human effort itself, across whole categories of work at once.

My new book, Custodialism: How Machine Abundance Ends the Age of Human Necessity, is about what happens next, and the argument runs against the optimistic consensus.

Four things we built on a foundation we cannot see

The comfortable story says abundance frees us. Freed from drudgery, we turn to art and leisure and each other. That story might come true. But it assumes something it never examines: that a population which has become economically unnecessary will keep the standing, the claims, and the dignity it won, historically, precisely because it was necessary.

Four of our most cherished institutions rest on the same hidden foundation, and it is the one this trajectory loosens. Meritocracy rewards talent because talent is needed; weaken the need and the reward has less to attach to. The market coordinates scarce human effort by pricing it; when goods cost almost nothing to produce, prices stop carrying the signal that once organized human work. Property spread broadly because effort was the ordinary road to owning it; sever effort from acquisition and ownership pools around those who already hold the machines. And rights, the ones we call inherent, expanded most reliably when rulers needed their people to work, to fight, to pay. The powerful conceded because they could not do without the many. That leverage is what erodes.

Each looks like an independent pillar. They are four structures raised on one piece of ground.

What I am not claiming

Here is where the argument is most often misread. I am not claiming that machines will surpass human capability everywhere, that nothing will be left for people to do. That is a forecast about the ceiling of technology, and it is genuinely contestable. My argument does not rest on it. What it requires is far weaker and far harder to deny: not that human capability disappears, but that broad human economic necessity weakens, across enough of the economy, for enough people, that the leverage necessity always conferred begins to fail. And it weakens unevenly, domain by domain, which is exactly why there is still time to act.

The part that should give us hope

There is a reason to doubt the most triumphant version of this future, and it has nothing to do with rooting for humans. A system trained on the record of what has been thought is very good at recombining that record. It is far weaker at the thing that actually drives discovery: finding the question worth asking in the first place. Answering was never the bottleneck in human progress. Asking was, and the genuinely new question rarely sits in the existing data, because if it did, someone would already have asked it.

So I do not believe the machines will simply outgrow us. Which changes what custodialism actually is. It is not something the machines impose on capable people. It is something that can happen only if we let our own capacities atrophy, hand over the questions, cede the judgment, and accept the tool as a substitute rather than an extension, until we have become the diminished thing a machine can in fact replace. The danger is not that the machines rise to our level. It is that we sink to theirs. And that is a choice, made or refused in how we use these tools, not a fate the technology hands down.

A system with no name

If human beings are provided for by an apparatus they do not own and are no longer needed to run, what do you call the result? It is tempting to say communism, since everyone is provided for regardless of contribution. But communism placed human labor at the center of value. This removes labor from the center entirely. It is not the endpoint of the line from capitalism to communism; it departs from that line into territory the line never described.

I call it custodialism, borrowing the word from the mid-twentieth-century literature of institutional care, where it named the management of a dependent population: housed, fed, kept safe, kept managed, and given no say. A custodial order can be generous. It can deliver more comfort to more people than any society in history. But comfort is not the same as standing, being provided for is not the same as having a claim, and a population maintained by a system it cannot hold to account has lost something even if it has gained everything else. I am not describing a future I expect. I am describing one I am warning against, and one we can still refuse.

The window

This is why the book is a warning and not a prophecy. The mechanism that makes the transition possible also makes this moment decisive. There is a window, defined by how much leverage human beings still hold and how much of their own capability they keep in exercise, in which the terms of what comes next can still be written. Whether custodialism arrives at all, and if it does, whether as broad ownership and preserved voice or as comfortable dependency, is not yet decided. It is being decided now, mostly by default, while we argue about the spectacle.

That is the conversation I wrote the book to start.

 Custodialism: How Machine Abundance Ends the Age of Human Necessity