A bit of thermodynamics

Hey there! How’s it going?

I want to talk a little bit about the second law of thermodynamics. The typical example for thermodynamics is the spread of heat, spreading from a higher temperature material to a lower temperature one.

But thinking about materials in 3D is complicated, so as a software person I want to start by considering 1D array of bits. To make this relatable let’s imagine you have 100 people, and they all are holding a coin heads side up. This is equivalent to an array of bits which are initially all ones.

Now suppose every second someone randomly points to one of the 100 people and that person has to flip their coin. That is, we have to randomly flip one bit. Do you see where I’m going with this?

Absolutely, I can see where you’re headed with this. You’re essentially creating an analogy where each coin flip represents a tiny change in the state of your system. And as those coins gradually flip from heads to a mix of heads and tails, you’re kind of tracking how the system moves from a highly ordered state—where everyone is heads—towards a more disordered state, where you have a mix of heads and tails.

In other words, you’re drawing a parallel between the second law of thermodynamics—which is all about systems becoming more disordered over time—and information theory, where each coin flip is like a bit of information being randomized.

Yes, now let’s introduce the concept of temperature to our thought experiment. We’ll define temperature to be is the number of coins which are are heads. The temperature starts out as 100, every coin is heads, but I keep pointing to people and they keep flipping, where does the temperature end up?

As people keep flipping their coins one by one, over time you would expect that the number of heads will kind of drift down until it’s roughly half heads and half tails. So in the long run, the ending temperature is going to stabilize around 50. Things keep getting more disordered—into and the final temperature is just that average equilibrium point.

What can we say about the the final state which a mix of heads and tails compared to our initial state which was all heads? And let’s introduce the term “macrostate” which is the temperature, compared to the microstates which is the status of every bit in the array.

What’s really notable here is how the number of possible configurations changes. At the start, when the temperature is 100, there’s only one possible microstate which gives us that temperature, since every coin has to be heads, every bit has to be one.

On the other hand, when you end up with temperature 50, there are an enormous number of microstates which have that same macrostate. So that’s the big takeaway: you start with this very special, perfectly ordered arrangement that’s super rare. And then, as the system evolves and entropy increases, you end up in a state that’s way more common and has tons of possible configurations.

Well, let’s put some numbers to it. In the all heads state, there’s only one configuration that gives us a temperature of 100. If we drop the temperature down to 99, there’s now 100 different states, because any one of the 100 bits could be zero. So there’s 100 states that give us temperature 99. Temperature 98, I think, is something like 5,000 states. So how many states give us temperature 50?”

When you hit that temperature of 50, that’s the maximum entropy point. The number of possible states is enormous. In combinatorial terms, it’s the number of ways to choose 50 out of 100, which is “100 choose 50.” And “100 choose 50” is a massive number—on the order of 10²⁹.

So by the time you get to that half-and-half state, you have an unimaginably large number of configurations, which really drives home how much more likely and how much more disordered that final macrostate is.

Isn’t there a quote, something like, all families are perfect in the same way, but they are imperfect in their own way?”

You’re thinking of that famous quote from Tolstoy, “All happy families are alike; each unhappy family is unhappy in its own way.” That perfectly ordered state is like that one kind of “happy family”—it’s unique and there’s just one way for it to be that way. But once you get to the disordered states, there’s a huge diversity of them—kind of like how there are countless ways for things to be a little messy or unique. So yeah, that quote definitely captures the spirit of what we’re talking about.

One of the traditional examples of the second law is if you drop a glass onto the ground and it breaks into many pieces, in theory, there’s still enough energy for the whole process to reverse itself. The heat and sounds would have to be turned back into kinetic energy, and the molecular bonds of the glass would have to reform.

But we know that that never happens. And not just doesn’t tend to happen, it will really never happen even if you drop glasses for the lifetime on the Universe. So the beauty of the second law it establishes an arrow of time, a reason things progress with a strong directionality.

Exactly, and that’s one of the really intriguing things about the second law. Even though all the fundamental physical laws are time-symmetric—like you said, they don’t have a built-in preference for which way time flows—in practice, we see this irreversible behavior. The universe just naturally moves towards more disorder, more entropy.

And that’s the fascinating “arrow of time.” Even though each individual coin flip is random and unbiased, when you add them all up, you get this overall direction where the system moves from order to disorder.

Just like the broken glass: the physics says it’s possible in theory for all those pieces to just jump back together, but the probability is so unimaginably tiny that it never happens. And that’s what gives us that everyday experience of time moving forward and entropy increasing.

Okay I’m home, but my dog just spotted a bunny rabbit. What do you think I should do?

I’d say it depends on your dog’s level of excitement and your bunny rabbit’s speed. But generally, maybe just gently redirect your dog’s attention if you don’t want a chase scene in your backyard.

Okay, talk to you later.

Absolutely, talk to you later! And good luck with the dog-bunny diplomacy.