"Senpai, what is entropy?"
In the after-school classroom, Yuki asked Aoi. It was the first question since joining the information theory club.
"Good question. Entropy is something that measures the amount of uncertainty."
"Amount of uncertainty...?"
Aoi drew a simple diagram in the notebook. "For example, let's predict tomorrow's weather. Suppose it's either sunny or rainy."
"Okay."
"If you're 100 percent certain tomorrow will be sunny, how do you feel?"
"Relieved. I don't need to bring an umbrella."
"Right. There's no uncertainty. Entropy is zero."
Yuki nodded. "Conversely, if sunny and rainy are 50 percent each?"
"Exactly. The most uncertain state. Entropy is at its maximum." Aoi added a graph. "When two choices are equally likely, entropy is 1 bit."
"Bit? That's a unit of data, right?"
"In information theory, it's also a unit of information. One bit is the amount of information needed to choose one from two options."
Yuki pondered. "So if there are more choices?"
"Good question. Think about a die. It has six faces, and you don't know which will come up."
"Six choices..."
"Entropy is log₂(6), about 2.58 bits." Aoi wrote the equation.
"What's log₂?"
"It means how many times you multiply 2 to get 6. Two to the power of 2.58 is approximately 6."
"Why 2?"
"Because bits are binary. But if you use natural logarithm, the unit becomes 'nats.' The essence is the same."
Yuki copied into the notebook. "But why logarithm?"
Aoi smiled. "Good question. Information has additivity. The information from two independent events is the sum of each. Probability is multiplication, but taking the logarithm turns it into addition."
"Ah, log(a×b) = log(a) + log(b)!"
"Correct. That's why logarithm is the natural scale."
Yuki thought a bit more, then asked another question. "But what if the die is loaded and ones come up more often?"
"Sharp. In that case, entropy becomes lower. The general formula is H(X) = -Σ p(x) log₂ p(x)."
"p(x) is probability..."
"Yes. A weighted average by the probability of each outcome. The more biased the distribution, the lower the entropy."
Aoi wrote an example on the whiteboard.
"Example: p(1)=0.5, p(2)=0.1, p(3)=0.1, p(4)=0.1, p(5)=0.1, p(6)=0.1 H = -0.5×log₂(0.5) - 5×(0.1×log₂(0.1)) H ≈ 2.16 bits"
"Smaller than 2.58!"
"Yes. Because you know ones come up more often, uncertainty decreases. Information content also decreases."
Yuki said excitedly, "So the higher the entropy, the harder to predict!"
"Precisely. And the higher the entropy, the greater the potential amount of information that can be obtained from that system."
"Potential amount of information..."
"When conveying something, the greater the receiver's uncertainty, the higher the message's value. If they already know, the information content is zero."
Yuki looked out the window. "That's why news reports unusual events."
"Exactly. The higher the entropy of an event, the higher its news value."
"Entropy is kind of interesting."
Aoi smiled gently. "This is the entrance to information theory. Entropy is the foundation of everything."
"I want to learn more."
"Next, I'll teach you about the relationship between entropy and coding."
Yuki's heart raced. A new world called information theory was spreading before them.