← Research ReadingWhat structure lives inside mHC's residual-mixing matrices?
What structure lives inside mHC's residual-mixing matrices?
What kind of structures emerge in the linear transformations added by Manifold Constrained Hyper Connections (mHC) used in modern language models to solve the information bottleneck problem in the residual stream? Studied from geometric, eigenvalue, and other perspectives.