"Senpai, I have a question."
In a corner of the library, Yuki spoke to Aoi and Mira.
"What is it?"
"Entropy measures uncertainty. But what about uncertainty after knowing something?"
Aoi smiled. "Good question. That's conditional entropy."
Mira quietly opened her notebook and wrote an equation. "H(X|Y) = H(X,Y) - H(Y)"
"H(X|Y) is the uncertainty of X after knowing Y," Aoi explained.
"Please teach me with a concrete example."
"Alright, a weather example. Today's weather and tomorrow's weather, two variables."
Aoi drew a table.
"Today sunny, tomorrow sunny: 40 percent Today sunny, tomorrow rain: 10 percent Today rain, tomorrow sunny: 10 percent Today rain, tomorrow rain: 40 percent"
"When you don't know today's weather, what's the entropy of tomorrow's weather?"
Yuki calculated. "Tomorrow sunny is 50 percent, rain is also 50 percent, so 1 bit."
"Correct. Now, when you know today is sunny, what's the entropy of tomorrow's weather?"
"If today is sunny, tomorrow is within 40+10=50, sunny is 40, rain is 10..."
"The probabilities are 40/50=0.8 and 10/50=0.2."
Yuki continued calculating. "H = -0.8 log₂(0.8) - 0.2 log₂(0.2) ≈ 0.72 bits"
"Perfect. By knowing today's weather, tomorrow's uncertainty decreased from 1 bit to 0.72 bits."
Mira added. "Information gain = 1 - 0.72 = 0.28 bits"
"Information gain. The amount of information about X obtained by knowing Y," Aoi supplemented.
"So when you know today is rainy?" Yuki started calculating.
"Similarly, 0.72 bits. In this case, it's symmetric."
"But on average?"
"H(tomorrow|today) = 0.5×0.72 + 0.5×0.72 = 0.72 bits. That's conditional entropy."
Yuki was surprised. "Even knowing today, tomorrow's uncertainty remains 0.72 bits."
"Yes. We can't predict perfectly. But it's 0.28 bits more certain than knowing nothing."
Mira drew another diagram. A Venn diagram with two overlapping circles.
"I(X;Y) = H(X) - H(X|Y)"
"Mutual information. The amount of information X and Y share," Aoi explained.
"Shared information..." Yuki pondered.
"If X and Y are independent, mutual information is zero. If perfectly correlated, it equals H(X)."
"Today's and tomorrow's weather are slightly correlated."
"Exactly. I(today;tomorrow) = 0.28 bits. A weak correlation."
Yuki wrote in the notebook. "So what's an example with stronger correlation?"
"For example, viewing the same coin twice. The first and second views are exactly the same."
"I(X;Y) = H(X)?"
"Yes. If you know Y, X is completely determined. H(X|Y) = 0"
Mira quietly stood up and looked out the window. Rain had started falling.
"The weather forecast was right," Aoi said.
"Are forecasts also conditional probability?" Yuki asked.
"Yes. Past data, air pressure, wind direction. Using all as conditions, predict today's weather."
"Minimize H(today|all data)."
"Precisely. The more information we gather, the less uncertainty."
Mira held out a note. "Uncertainty = Ignorance"
"Not knowing is the source of uncertainty," Aoi nodded.
Yuki said seriously, "So by gaining information, the world becomes more certain."
"But there's no complete certainty. Residual uncertainty always remains somewhere."
"That's conditional entropy..."
"Yes. It tells us the limits of prediction."
The rain intensified. The three gazed out the window.
"We don't know tomorrow's weather," Yuki murmured.
"But with today's information, we can guess a little," Aoi replied.
Mira smiled. Her expression always held information.
"Living with uncertainty. That might be information theory's teaching."
Yuki thought while listening to the rain. The world is uncertain. But by gathering information, we can understand bit by bit. That might be what learning means.