From Machine God to AI Pluralism
Keep AI Open. Keep AI Human.
“These collisions of values are of the essence of what they are and what we are.” — Isaiah Berlin
Almost all the capital deployed in AI so far has gone to building and serving a single kind of artifact. I will call it the Singleton: large, horizontal models, trained and frozen, generalized across every use case.
Today, the leader in this category is Anthropic.
We have gotten good at making Singletons cheaper. MoE routes each token through a sparse slice of the network, only using a fraction of the parameters on any given pass. The industry is also increasingly aware of the cost, performance, and sovereignty trade-offs of using single model providers. Routers like OpenRouter arbitrage providers; agentic harnesses like Nous’s Hermes agent stitch models together, though largely within the same paradigm where weights are cast in bronze and shipped.
Yet, the frontier is shifting. Moving away from static training and frozen models toward models that learn continuously - a merger of training and inference.
This shift is happening along two different axes at once.
It is happening inside the Singleton, as the labs chase recursive self-improvement. The large labs are increasingly pursuing this direction, joined by newcomers Recursive Intelligence and Ineffable.
And it is being pursued on a more distributed scale: researchers pursuing specialized, continually-learning models embedded inside the institutions with their own, protected data sets - a network of different “sovereign AI” stacks.
Perhaps one of the most important questions of the 21st century is which quadrant we end up in?
The Paradox of Pluralism
One of my favorite thinkers is Isaiah Berlin, best remembered for his defense of pluralism: the idea that human values can each be fully legitimate and still be incompatible, colliding with no clean resolution outside the context of a particular choice.
For example, liberty and equality can both be good, and yet are often held in tension. You can both be a Christian (do unto others) and a capitalist (survival of the fittest). Sometimes you must lie to keep a promise.
Pluralism is not relativism. You do not get to decide your own truth. However, there are many valid ways to live, to prioritize, to ascribe value between different goods, and no single ranking of them is definitive.
Pluralism, in this sense, is the load-bearing wall of liberalism; a way to hold incompatible values in tension without chaos.
So, what happens when more and more decisions - of production, of policy, of consensus itself - are handed over to intelligent systems?
The Singleton: Dr. Jekyll & Mr. Hyde
Singletons today have done a surprisingly good job of walking a centrist line amidst a highly-polarized political backdrop. In particular, where truth is objective and verifiable - math, code, etc - a single scaled model works very well. Yet it is the slippery, subjective, colliding, human middle which seems much more difficult for a single oracle - even one with millions of GPUs - to adjudicate. Especially in a society with free choice.
I almost see today’s Singletons as a rerun of mid-century broadcast media. One-to-many megaphones. Tens of millions tuned to the same three channels, downloading the consensus to discuss at the Monday water cooler.
I would expect a similar fragmentation in AI. From three channels to three hundred. From the timeline the algo. From one-to-many to many-to-many. Singletons appear poised to cede ground to a proliferation of models tuned to the wildly different tastes, cost preferences, and priors - reflecting the globally diverse user bases they serve.
And yet, unexpectedly, this very fragmentation - of media, of music, of commerce - led to renewed centralization within internet giants, as Ben Thompson’s aggregation theory took hold in the 2010s.
The audience got infinite choice and yet that long-tail was collapsed down into a handful of platforms that increasingly surfaced a mix of the 1) most popular (uniform timelines optimizing for attention) and 2) the most engaging (rage bait).
A walled garden.
In the AI wave, the concentrated nature of compute may lead to a similar phenomenon, just trading a model layer duopoly for a hyperscaler oligopoly.
So far, the paradigm has involved bringing data to the large clusters of compute in exchange for large, up-front payments. Businesses are beginning to realize this is a deal with the devil. Slow, extended seppuku.
Data is, increasingly, the moat.
The Reward Function is the Bottleneck
Increasingly, it seems that data - real world data - is the primary constraint to useful intelligence. In the scaling era of LLMs, data was abundant in the form of internet artifacts (”distilled” by frontier labs without public compensation, I would add). Compute was scarce.
Increasingly, however, data seems to be the real constraint. Or, more precisely, the cost per unit of verified experience, particularly in difficult to verify domains.
AI labs are now spending billions on data foundries: at first in expert contractors like Mercor and Surge which built rubrics for LLMs in less verifiable domains, next for specialized teams to build “RL environments” (i.e. AI reproductions of popular software suites), and increasingly on automated lab data in bio, materials, and robotics. These data sets are not publicly available at internet scale. They are owned by whoever runs the process - the assay result, the yield, the shipped part that worked - you cannot scrape a reward function embedded in someone else’s operations.
The reward function is the bottleneck, and the reward function is… distributed.
Meta installing computer use spyware on its thousands of employees is telling. Perhaps the best data set to automate an engineer or an AI researcher at meta comes from.... an engineer or AI researcher at Meta?
Companies are now wising up to the Palantir critique: what if providing data to the Frontier labs comes to be synonymous with slow, inevitable suicide?
The reality is that any single Mercor-esque network of experts across broad domains pales in comparison to the tacit knowledge spread across individuals, companies, and institutions which make up our economies and societies.
What if the answer is not extracting the data and bringing it to the tower of compute?
What if the answer is bringing compute to the verified reward signal?
Learning to Garden
“The world was a great garden in which different flowers and plants grew, each in its own way, each with its own claims and rights and past and future.”
— The Crooked Timber of Humanity, Isaiah Berlin
We are entering the “era of experience“. AI’s need to act on the world and learn from these interactions.
Given the structure of the global economy - the competing incentives and colliding truths and protected data and tacit knowledge and fragmented talent - the inherent complexity of it all; it would seem likely that single closed models trying to be the best at everything, would be suboptimal; especially if data owners start to play long-term games.
The other vision - one recently outlined by Mira Murati - is what I will call the Garden: networks of models of various shapes and sizes and specializations which learn on the job (using the actual data produced in completing a task), updating their weights in real time without catastrophic forgetting.
As opposed to pursuing recursive self-improvement of the God-model, this would be a co-evolution of AI with humans in their existing, diverse competing institutions.
AI pluralism.
The singleton would have us all serving as individual contractors for Mercor or uncompensated, data repositories for Claude; insights stripped and harvested and brought back as an offering to the towers of compute. However, distributed networks of hybrid compute and open models that can be updated continuously based on ground truth interactions with real economic tasks - that might be the least dystopian vision of AI yet.
Oddly enough, this is the vision championed by the US’s chief geopolitical adversary, not exactly known for its ideological pluralism.
What gives?
Make America Pluralist Again
Peter Thiel famously said that “AI is centralizing” or more ideologically “Crypto is libertarian, AI is communist.” On its current trajectory, AI is indeed shaping up to be the greatest concentration of power the world has ever seen.
Paradoxically, within the AI race, the US has been pursuing a closed, Singleton-dominant approach to AI, while a more authoritarian China as been enthusiastically supporting an open ecosystem of models.
And yet, this simplified inversion is also incomplete.
China has cultivated a genuinely open ecosystem of compounding models across size and capability tier, and yet wary of the disharmony which comes from a fracturing of consensus, still pursues robust censorship in sensitive areas. The US, meanwhile, has concentrated on closed, frontier-scale Singletons, increasingly censored based access and safety as opposed to ideology.
China has architectural pluralism sitting on top of ideological monism. The West has architectural monism sitting on top of value pluralism.
Neither is the ideal quadrant.
Ideally, we will find our way to the garden. Ideally one that meshes a shared reality (for building consensus around truth) with a plurality of values (respecting beliefs held in tension).
This is a very hard mix to get right.
A view of AI not as a machine god, but as a ubiquitous, highly productive general purpose utility - which is not shut up behind a walled garden or enforcing a particular ideology but flows easily between people, groups, and private-public divides - competing, colliding, and constantly evolving networks of intelligences just as diverse and chaotic and beautiful as humans themselves.
If we are on the cusp of birthing a new species, let us at least create one in our image.
Keep AI Open. Keep AI human.
The author can be found on x @ponderingdurian