Short Story ⟡ Informatics

A Small Promise with KL Distance

Understanding the gap between promises and expectations through Kullback-Leibler divergence.

  • #KL divergence
  • #distance
  • #expectation
  • #promise
  • #distribution difference

"Have you ever had a promise broken?"

Riku suddenly asked. Break time in the club room.

"Yes," Yuki answered. "Small ones, though."

"How did you feel?"

"Disappointed. Because I had expectations."

Aoi showed interest. "That can be explained with KL divergence."

"KL?" Riku tilted his head.

"Kullback-Leibler distance. A measure that quantifies the difference between two probability distributions."

Aoi wrote a formula in the notebook. "D_KL(P||Q) = Σ P(x) log(P(x)/Q(x))"

"P is the true distribution, Q is the predicted distribution."

Yuki thought. "Is it related to promises?"

"Yes. Promises create expectations. That expectation is probability distribution Q."

"Then what's P?"

"What actually happened. The distribution of reality."

Riku asked for an example. "Specifically?"

Aoi explained. "For example, a friend promised 'I'll come at 10 tomorrow.'"

"Yeah."

"In your mind, expectation distribution Q forms: '90 percent probability of coming at 10, 10 percent of not coming.'"

Yuki continued. "But actually they didn't come. P(came)=0, P(didn't come)=1"

"This gap between P and Q can be measured with KL divergence."

Riku tried to calculate. "What's the value?"

"In this case, it approaches infinity. Because P(x)=1 but Q(x) is small."

"Infinity?"

"Mathematically yes. But the meaning is clear. The gap between expectation and reality is extremely large."

Yuki said thoughtfully, "So that's why we get disappointed."

"Yes. The larger the KL divergence, the greater the disappointment."

Riku asked, "Then what's a good promise?"

"A promise with small KL distance," Aoi answered.

"Expectation and reality are close. It's realized as promised."

Yuki gave another example. "But if expectations are too low, that's also a problem, right?"

"What do you mean?"

"They say 'I probably can't come' and really don't. KL distance is small but I'm not happy."

Aoi was impressed. "Sharp. Can't be measured by KL distance alone, the absolute value of expected value also matters."

"Balance," Riku understood.

"Right. Creating moderately high expectations and realizing them. That's a good promise."

Yuki laughed. "Analyzing promises with information theory."

"But it makes sense," Riku said.

Aoi continued. "Conversely, what about surprises?"

"Surprises?"

"Something better than expected happens. P(good thing)>Q(good thing)"

"KL distance becomes large then too," Yuki noticed.

"Yes. But it's a gap in a pleasant direction."

Riku was amused. "Positive KL divergence?"

"KL itself is non-negative, but as an interpretation, yes."

Yuki thought. "So promises are about managing expectations."

"Good expression," Aoi nodded.

"Appropriately setting the other person's expectation distribution Q so that reality P exceeds it, or at least matches it."

Riku said practically, "So when making promises, be a bit conservative?"

"One strategy. Lower expectations, then actually do more."

"But lying is wrong, right?" Yuki pointed out.

"Of course. Adjust expectations within an honest range."

Aoi supplemented. "In communication theory, minimizing KL divergence is the goal."

"Bringing predicted distribution closer to true distribution."

"Maybe the same with human relationships. Matching other's expectations and your actions as much as possible."

Riku laughed. "I probably have large KL distance."

"Because you're unpredictable?" Yuki agreed.

"But I keep important promises," Riku said seriously.

Aoi said gently, "That's enough. You don't need to make all behavior predictable."

"Moderate surprise makes relationships interesting."

Yuki looked out the window. "Keeping small promises carefully."

"Keeping KL distance small."

"That's how you build trust."

The three quietly nodded. Promises are probability distributions. Expectations are prediction models. And trust is the accumulation of small KL distances.

Riku said, "From now on, I'll keep promises. Reduce KL distance."

Yuki smiled. "I'm expecting it."

"That's pressure," Riku laughed.

But there was no lie in that smile. Small promises gradually change the world.