Mira was quietly writing numbers in her notebook today, as usual.
"What are you measuring?" Yuki peeked over.
Mira wrote. "Entropy"
"Entropy," Aoi translated. "A measure of uncertainty."
"Measuring uncertainty?"
"Yes. Mira measures the entropy of all phenomena around her."
Yuki looked at the notebook. "Weather, dice, conversation topics..."
"Everything has a probability distribution. And entropy can be calculated," Aoi explained.
Mira wrote an equation.
"H(X) = -Σ p(x) log₂ p(x)"
"This is Shannon entropy," Aoi said. "Calculating average information content from probability distribution."
"Looks difficult," Yuki frowned.
"Let's think with concrete examples," Aoi drew on the whiteboard. "Flip a coin. What's the probability of heads?"
"Half and half, so 0.5."
"Correct. Then what's the entropy?"
Aoi calculated. "H = -0.5 log₂ 0.5 - 0.5 log₂ 0.5 = 1 bit"
"There's 1 bit of uncertainty."
Mira nodded and wrote the next example. "Biased coin: p=0.9"
"A biased coin. Heads comes up 90 percent," Aoi continued. "What's the entropy?"
Yuki tried calculating. "H = -0.9 log₂ 0.9 - 0.1 log₂ 0.1 ≈ 0.47 bit"
"Correct. Lower than a fair coin."
"Because uncertainty decreased?"
"Yes. When results are somewhat predictable, entropy decreases."
Mira wrote a new example. "Dice: 6 outcomes"
"A die has 6 outcomes. If all equally probable," Aoi calculated. "H = log₂ 6 ≈ 2.58 bits"
"Higher than a coin," Yuki noticed.
"More choices increases uncertainty. But this is for equal probabilities."
Mira wrote further. "Weighted dice"
"If it's a loaded die where 1 comes up more often, entropy decreases."
Yuki began to understand. "Because predictability increases."
"Perfect. Entropy is maximized when it's hardest to predict."
Aoi stated an important theorem. "Uniform distribution maximizes entropy."
"Why?"
"If all outcomes have equal probability, it's most uncertain."
Mira wrote an everyday example. "Lunch menu entropy"
"Interesting," Aoi laughed. "Someone who eats the same thing every day has entropy close to zero."
"Someone who randomly eats various things has high entropy," Yuki continued.
"Right. Mira measures that kind of everyday entropy."
Yuki looked at Mira's notebook. "Senpai's behavior patterns, club room topics, weather..."
"All can be treated as probabilistic phenomena," Aoi said. "And entropy can quantify the uncertainty."
"But why measure it?" Yuki asked.
Mira spoke for the first time. "...To predict."
Aoi supplemented. "If you know entropy, you know how much information is needed."
"Information is needed?"
"Yes. High-entropy phenomena need many bits to compress. Low-entropy needs fewer bits."
Yuki was surprised. "It's related to file compression?"
"Greatly. A file's entropy determines the theoretical compression limit."
Mira wrote. "Shannon's source coding theorem"
"Shannon's source coding theorem," Aoi explained. "You cannot compress below entropy."
"Amazing."
"But you can compress close to entropy. That's the theoretical limit."
Yuki summarized. "Measuring entropy is knowing uncertainty. And thinking about how to deal with it."
"Correct," Aoi acknowledged.
Mira smiled. To her, the world was a collection of entropies.
"Maybe I'll try measuring too," Yuki said. "The entropy of my own life."
"Might be interesting," Aoi agreed.
The three began measuring their respective uncertainties. Knowing entropy was the first step to understanding the world.