• Parameters vs. thinking tokens: is there an exchange rate?

  You can increase the amount an LLM thinks in two axes: by increasing the size of the model, or by increasing the number of tokens it outputs. Can we quantify the mathematical equivalence between the two? Given an architecture (depth, embed dim, param count), what's the equation that establishes the relationship between model size and reasoning effort (test-time compute)?

  2026-08-23 · 55 papers · 263 links

• What structure lives inside mHC's residual-mixing matrices?

  What kind of structures emerge in the linear transformations added by Manifold Constrained Hyper Connections (mHC) used in modern language models to solve the information bottleneck problem in the residual stream? Studied from geometric, eigenvalue, and other perspectives.

  2026-06-24 · 33 papers · 149 links

• Why was BPE tokenization crucial?

  We know BPE was crucial to get language models to learn anything. Why? It's all compression anyway — why couldn't character-level tokens reach the same level? Is char-level provably complete (would catch up with more training, just inefficient) or is there an unhobbling we gain from subword tokenization?

  2026-06-23 · 44 papers · 182 links