Technological Speed Limit

I pondered Moore’s Law for decades. I watched PC clockspeeds rise from 10Mhz, my family’s first PC, until they started to top out around 3Ghz. When people breathlessly invoke Moore’s Law it usually to talk about how amazing it was that processor technology improved so quickly. But I didn’t understand the phenomenon deeply until I ask two related questions: why does the technology improve so steadily, and why doesn’t it improve faster?

One day walking our dog it hit me. Steady improvement over sixty years meant that there was fundamental speed limit in how quickly we could improve the technology. There’s no other way to explain the steadiness except that we are bumping up against a limit. There’s no way we’d be balancing on a saddle point, happening to hit the same speed year after year. To be that stead we must be tucked up agaist a wall. Through a never ending parade of companies, individuals, approaches and funding cycles, improvements came like clockwork.

I realized the reason was this: our ability to improve the technology has maximum theoretical “learning rate” and we are able to push improvements exactly to to that rate, but not further. The entire industry, I realized, is complex physical system, including the humans and their human brains, such that physical system can only evolve so quickly, no matter how hard you push it.

What does this “technological speed limit” tell us about AI. For starters I think it fully explains why the legacy tech giants, with seemingly unlimitted resources, cannot catch Anthropic and OpenAI. Those two startups are well funded and well staffed enough to be operating at full speed, bumping up against the speed limit for improving AI. Unless they suffer an implosion of some kind, literally no one can catch them, because no one can exceed that same limit, even while other comapanies and open source models yip at their heels.

Viewed up close the smooth steady growth appears to be chaotic and varied, with breakthroughs announced weekly. But if you zoom out enough you’ll see that humans are on a trajectory of improving AI technology that is maxed out right at the limit of what’s possible.