GigaToken: ~1000x faster Language model tokenization

Interesting :

Q: Did you just way over-optimize for a specific CPU and tokenizer? How is it so fast? No, I way over-optimized for every combination of these! The results are very consistent across CPUs (modern x86 and ARM), and across specific tokenizers.

The major improvements are in optimizing heavily an implementation that usually is outsourced to a Regex engine (pretokenization) using SIMD, minimizing branching and other tricks, as well as heavily optimizing caching of pretoken mappings (if a word has been seen before, look it up its encoded tokens efficiently). Caching is a very hard problem in this domain since the cache grows very quickly, and pretoken distributions are very long-tailed.

Finally, interactions with Python are minimized, and threads have minimal interactions with each other.

This is awesome, but tokenization is typically <0.1% of total inference time.

Presumably there's a host of applications that just need to tokenize, though, and this would be great for those!

So the question becomes, how many other parts of the inference pipeline have left 1000x optimization opportunities lying on the table?

I had to stare at that chart for a minute just to let the numbers sink in. It's genuinely mind-bending, incredible ship OP

We should just rewrite everything in Rust, especially bloated Python code, and the world would be a better place. ;) Disclosure: I'm a Rust advocate!

What sort of setups do people have that are bounded by the speed of the tokenizer?

This is really cool, great work!

Quality software here.

Very interesting project! Are there benchmarks for the "compatibility mode" or are all the numbers for the Gigatoken API?

wow, best release all week.

quite excellent software