Teaching Question: What are there technical ideas around Scaling (but not Alignment) that grad-level students should know or study? Is it thinking about systems? Order of magnitude thinking? Abstraction? Experimental design?
17
6
2
89
Replying to @srush_nlp
I find people unfamiliar with scaling are shocked by this:

Oct 12, 2022 · 12:31 AM UTC

17
22
8
252
Replying to @stephenroller @srush_nlp
Also all modern scaling strategies are highly synchronous. Which means one bad node can tank the entire system. I would love our future researchers to be thinking about this.
2
1
27
is this really true? it's not worth ddp
2
more replies
Replying to @stephenroller @srush_nlp
Would be nice to have a column for number of layers. If you change hidden dim, but keep number of layers same, then some of this is less surprising?
1
Hidden from view in screenshot. Left as an intentional exercise to the reader
1
1
2
more replies
Replying to @stephenroller @srush_nlp
Tbh building a flops calculator is a pretty good homework assignment…
1
10
Yeah, this makes a ton of sense. Quantized, backward, and forward memory as well.
1
4
more replies
Replying to @stephenroller @srush_nlp
This is pretty interesting. If the original Transformer has almost a linear computational cost wrt seq len (at scale), why so many people spend time on trying to make the attention cost O(n) (e.g., LongT5, etc)?
3
8
I suspect bc so many people are still thinking in BERT regime
1
7
more replies
Replying to @stephenroller @srush_nlp
Time to start pushing up those sequence lengths? :)
1
4
It’s still n^2 and one of the most memory heavy operations. But yeah, O(n) attention doesn’t have the same appeal as it did at BERT scale.
1
4
more replies
Load more