Wanted to share something I’ve made recently - a visualisation for ML number formats: thecharlieblake.co.uk/visual…… Hopefully this gives some intuition for the trade-offs involved (e.g. FP16 vs BF16). It also introduces the two newly proposed FP8 formats. Feedback much appreciated🙏

Oct 17, 2022 · 7:42 AM UTC

5
37
3
186
0
Just wanted to tag a couple of people on my feed who might be interested: @charles_irl - did some similar stuff a few months back which got me into all this! and @Tim_Dettmers - whose LLM.int8() paper shows some v interesting aspects of using low-precision formats
4
Replying to @thecharlieblake
Sorry i don't understand it, but is it somehow related to how we can use quantization/lower precision for resource constrained edge devices?
1
I have a section on this! thecharlieblake.co.uk/visual… (see the "What are number formats and why should I care about what values they can represent? What are these new FP8 formats?" question)

Visualising ML number formats

A visualisation of number formats for machine learning --- I couldn’t find any good visualisations of machine learning number formats online, so I decided to make one. It’s interactive, and hopefully...

thecharlieblake.co.uk
3
Replying to @thecharlieblake
Love it!
1
Replying to @thecharlieblake
Yo, this is incredible! Thanks for sharing
2
Replying to @thecharlieblake
Really cool, Charlie! Your visualization really helps to understand the idiosyncrasies of the format zoology 🔝
1