Normal People

Before we go deeper in return on tokens later, we should talk about the investment and return on owning intelligence first.

In computer vision, there has been a similar pattern. Early methods conceived of vision as searching for edges, or generalized cylinders, or in terms of SIFT features. But today all this is discarded. Modern deep-learning neural networks use only the notions of convolution and certain kinds of invariances, and perform much better. The Bitter Lesson

Rich argued that across 70 years of AI, general methods that scale with computation (search and learning) consistently outperform approaches built on human knowledge of a domain¹. The recent advancement in continual learning and reinforcement learning²³⁴ might have changed the game. (Speaking of games, there is a self-learning Mario you can play around). Practical examples in the industry are moving this from paper into product. Do we need it? Most likely. Does continual learning make production inference as costly as training time? The return on reinforcement learning and fine tuning is something we will go into more details later.

I have had 30+ conversations with people telling me "people are going to have their own energy source in the company", "home use compute and home use GPUs!", "cost of buying from labs are too expensive!". Marco actually home-baked a small cluster so I don't see why people can't do it. To put the numbers down into perspective:

During training

• A small model (1B to 7B parameters) can be trained on a single H100, but it will take longer time. For practicality it's more often to be trained on a cluster of 8 to 32 H100s. ⁵

• A medium model (7B to 70B parameters) requires 64 to 512 H100s.

• A frontier / Mega Models (400B+ parameters) requires 1,000 to 10,000+ GPUs (And there were a lot orchestration project built to support this problem!)⁶⁷

Production inference/Test time compute

GPU requirement is much smaller than training but it depends heavily on precision, context length, batch size, and latency target. Inference mostly needs memory for model weights and KV cache. a 400B+ model takes up about 810GB before KV cache. More details about what happens during training and serving inferences for a model.

• A small model (1B to 7B parameters), minimally requires 1 GPU, and sometimes maybe not even H100. But depending on the traffic you might need 1-8 H100s.

• A 7B to 70B parameter model, minimum spec is 1-2 H100s for small/medium; 2+ for 70B, but usually need 8-256+ H100s for real API services.

• 400B+ densed/frontier model takes 8-16+ H100/H200 per serving replica, but it can also be hundreds to thousands of GPUs depending on user volume.

So we clustermaxxing now?

A small 8xH100 cluster within Supermicro SYS-322GA-NR gives a useful physical baseline:

Let's say we put 10 of these in a Vertiv VR3350:

Great, now we have one rack with 80 GPUs. Training compute is usually counted in GPU-hours: one GPU running for one hour is one GPU-hour, so total demand is roughly GPU count × wall-clock hours. If a frontier pre-training run needs on the order of 20 million GPU-hours and you want it done in about three months (~2,160 hours), you need roughly 10,000 GPUs — 125 racks like this. That is one lab, one model, one training push.

Training alone only gets you to ~45,000 GPUs. Inference is what blows the number up: tens of millions of users do not each get a GPU, but peak traffic still needs thousands of replicas in parallel, and a single 400B+ replica can already consume 8-16+ GPUs before batching helps. 300,000 GPUs is also ~3,750 of these 80-GPU racks — the same unit math as above, just scaled by 30×.

Take H100 NVL (80 GB HBM3 and up to 3.35 TB/s), on average it costs between $29,900 and $48,400 as a standalone unit (it can go upwards to $90,000+ depending on supplier/reseller).

Let's take a medium number, say each of them costs $35,000. Combined together:

300,000 GPUs costs $10,500,000,000.

This is just the cost on GPUs and we completely ignored the costs on acquiring land permit, hiring, electricity, networking build out, ongoing maintenance etc. When we add up the numbers together monster rounds are not as crazy as it seems.

You can play around with the cost of building a data center on the Cluster TCO Calculator made by SemiAnalysis.

If things are so large, how do we optimise?

These numbers looks quite scary. There are recent advancement from the post transfomer community where smaller models that can perform the same as GPT-3 be deployed on a small Rasberry Pi. These smaller models can be deployed, trained on much smaller machines and requires much less resources, building on python and Rust There were significant investment going into fine tuning small models with LLM insights. The debates between who is wining the context window haven't end.

Autonamtic AI research is coming quickly than anyone could have imagine.

It can be done, but how much is needed

We can do some dumby calculation to estimate this:

If someone's maximum spent on token is 90k, and some people could be spending nothing on AI. Currently there are approximately 8.3 billion people on the planet. In 20 years time we will reach roughly 9.5 billion.

If we split the worlds' population into four groups:

• teenagers (0 - 14yr): to simplify this we imagine a large group of them are spending a lot less or close to zero
• adults (15-24yr): they might spend a lot more for work
• mid-age (25-64yr): senior executive or pre-retirement they would still spend some tokens but less
• elderlies (65yr - above): retired/not using AI

The formula to calculate annual electricity cost is simple:

Economics of data centers

GPU Capital Market