Training a large model outside a data center has stopped being an experiment

On November 6, 2025 Bitcoin closed at $101,346, −2.4% against the previous day, while the day’s news was about the distributed training of artificial intelligence models.

What actually got finished

In November 2024 one group finished a model of 10 billion parameters trained on 112 cards spread across three continents, keeping the machines busy 96 percent of the time under the best conditions. In May 2025 the same group finished one of 32 billion, with more than 100 mismatched nodes and more than 400 hours of training. The whole process was public and anyone could repeat it.

That first model’s scores on the standard tests were far from records: thirty-seven and a half out of a hundred on one, seventy-two on the other. The people who built it never claimed otherwise. The result lay elsewhere. Volunteers scattered around the world, with machines joining and leaving while the training ran, had finished the job using 400 times less bandwidth than the normal method.

Meanwhile a project running inside a network of subnetworks has more than 200 cards in production, has finished a model of 1.2 billion parameters and is aiming for 70. All these figures are declared by the people who produced them: you can check them in the published work, but we haven’t redone them.

The wall was bandwidth, and how they lowered it

The first step was to stop synchronizing at every step and do it every 500. The machines work alone for a stretch and realign afterwards, and that alone cuts communication five hundredfold. The second was to compress what gets sent, with a transform and a reduction down to a single bit. The declared reductions run from a thousand to 10,000 times against the standard methods.

The third goes further still: synchronize only 0.1 percent of the parameters at each step, and the model converges anyway. In December 2024 one of these methods trained a model of fifteen billion parameters over eleven thousand steps, on machines supplied by four different operators. So what brought the wall down was three tricks stacked on top of each other, rather than any single idea.

The other road: split the model instead of the data

In September 2024 a group completed a training run of this kind for the first time, across different participants and unequal machines: seven and a half billion parameters, three weeks, thirty-six billion words. A year later they showed a model of eight billion trained with its blocks in four different places joined by the ordinary internet. The results matched a centralized setup, which until then had been thought impossible.

What makes it workable is how failures are handled. Another group built a system that watches which computing routes are free and skips slow or dead nodes. It keeps 93 percent of its output even when 50 percent of them go down. In a data center a node that disappears is a fault; here it’s a normal operating condition.

Why reinforcement learning fits so well here

Out of that came a setup that separates three things that used to sit together: the training, the use of the model to generate answers, and the delivery of the updated weights. Each machine runs its own round and delivers when it’s done. A light check verifies that it really learned from what it saw, instead of copying or making things up.

Another group put a three-stage mechanism on its test network. Each node answers on its own, then the nodes criticize each other, then they settle on a shared answer, and the reward depends on how close each one came to that agreement. It looks more like a newsroom than a server farm, and it works for the same reason: nobody has to trust anybody.

How you check that somebody is really working

There are four different answers going around, and they’re worth keeping apart because they fail in different ways. The first is economic: whoever works puts down a bond, whoever cheats loses it, and anyone can report them. The second watches behavior, checking that the corrections delivered match the data that machine saw. The third has validators vote on the quality of what comes in, and also penalizes validators who inflate their votes.

The fourth, not yet running, tries to make the work itself the proof: every answer generated carries a stamp showing which model produced it. None of the four is settled. One of these networks already suffered a real attack over the 2024 holidays, and defending it is the usual cat-and-mouse game.

The four approaches to verification: economic, based on a bond put down that is lost by cheating; behavioral, which checks the consistency between the data seen and the corrections delivered; by vote, where validators judge the quality and whoever inflates the votes is penalized; by stamp, where every answer generated carries cryptographic proof of which model produced it.

The money inside it

For now there’s essentially one live token: the token of the network of subnetworks. It touched $5 billion of market value in December 2024, and since February 2025 it has had separate tokens for each of its 118 subnetworks. All the others are still to come, and the people waiting for them say so openly: these are capital raises with a token promised later.

Capital raised by the groups named: nous research, $65m (declared valuation $1bn); gensyn, $43m (test network, no token); prime intellect, $20.5m (no token announced); pluralis, $7.6m (shares of the model, not a fixed wage).

The estimate doing the rounds is that the artificial intelligence market will be worth $15 trillion by 2030. It’s a self-interested estimate, made by people who have put money into these projects, and it’s worth what every five-year estimate is worth. It tells you what its authors hope for, and very little about what will actually happen.

What doesn’t add up, and what to watch

The fourth is the most awkward, and it’s about real money. If the finished model’s weights are public, who pays for training it? One of the groups tries to solve this by giving participants a share of the model instead of a fee. But that share is only worth something if the model makes money, and there’s no guarantee at all that it will.

From outside there are four things to watch, all of them accounting. How many cards really take part: two hundred today in the most advanced project, when the useful threshold is fifty times that. How many training runs finish without being aborted. What a unit of computing costs here against traditional suppliers. And how far these models are from the best centralized ones. Bitcoin, meanwhile, closed at $101,346, down 2.4 percent. As usual, the price doesn’t notice the day a piece of an industry moves.

Put bluntly: in a data center a node that disappears is a fault to repair; in a network of volunteers it’s the normal condition, and all the research went into building on top of it instead of preventing it.