Tokenization in Transformers v5: Simpler, Clearer, and More Modular

Why it matters

Short-term noise or genuine inflection point? Dig into the tokenization details before drawing conclusions about transformers.