← Research ReadingWhy was BPE tokenization crucial?

Why was BPE tokenization crucial?

We know BPE was crucial to get language models to learn anything. Why? It's all compression anyway — why couldn't character-level tokens reach the same level? Is char-level provably complete (would catch up with more training, just inefficient) or is there an unhobbling we gain from subword tokenization?