Blog — Tim-Henrik Buelles

Thoughts on research, applied stats, and machine learning. Experiences as a mathematician in industry.

James–Stein

December 28, 2025

James–Stein changed statistical inference forever – ML didn't get the memo. At least many practitioners seem oddly unaware of it. In 1955, Charles Stein found that maximum likelihood estimators for Gaussian models are inadmissible (bad). Shrinkage towards zero beats it uniformly (good). In this note, I review the estimation theory and connect James–Stein to two popular methods: L2 regularization and weight decay.

Online Attention Entropy

November 26, 2025

Attention entropy collapse is a failure mode in ML where attention logits converge to a small number of states. In practice, this is often poorly monitored and patched with normalization. In this note, I discuss how to extend the attention algorithm to compute entropy online with the same GPU kernel.