← Research ReadingStructure of the residual-mixing matrices in mHC

Structure of the residual-mixing matrices in mHC

What kind of structures emerge in the linear transformations added by Manifold Constrained Hyper Connections (mHC) used in modern language models to solve the information bottleneck problem in the residual stream? Studied from geometric, eigenvalue, and other perspectives.