• What structure lives inside mHC's residual-mixing matrices?

  What kind of structures emerge in the linear transformations added by Manifold Constrained Hyper Connections (mHC) used in modern language models to solve the information bottleneck problem in the residual stream? Studied from geometric, eigenvalue, and other perspectives.

  2026-06-24 · 33 papers · 149 links

• Why was BPE tokenization crucial?

  We know BPE was crucial to get language models to learn anything. Why? It's all compression anyway — why couldn't character-level tokens reach the same level? Is char-level provably complete (would catch up with more training, just inefficient) or is there an unhobbling we gain from subword tokenization?

  2026-06-23 · 44 papers · 182 links