← Research ReadingWhy did language models need BPE tokenization?

Why did language models need BPE tokenization?

Byte-pair encoding (BPE) tokenization is widely credited with making language models trainable. Why? Tokenization is a form of compression, so why couldn't character-level models reach the same quality? Are character-level models provably able to learn the same things given more training (just less efficiently), or does subword tokenization give models a capability they would otherwise lack?