← Research ReadingStructure of the residual-mixing matrices in mHC
Structure of the residual-mixing matrices in mHC
What kind of structures emerge in the linear transformations added by Manifold Constrained Hyper Connections (mHC) used in modern language models to solve the information bottleneck problem in the residual stream? Studied from geometric, eigenvalue, and other perspectives.