@ggerganov's LLaMA works on a Pixel 6! LLaMAs been waiting for this, and so have I

Mar 13, 2023 路 7:57 AM UTC

9
92
65
545
Generations are very slow, seems it's due to load time. Might be cause I'm running it on Termux
4
27
Replying to @thiteanish @ggerganov
Does it uses Pixel 6 TPUs?
1
6
No, it's not even optimized for the pixel. I quantized it on my mac and transferred the weights over due to space issues on my phone
2
39
more replies
Replying to @thiteanish @ggerganov
Oh man, I need to deploy this on my pixel 6! Is there a repo you have posted anywhere? I can't keep up with all these ai LLaMA advances!
1
Replying to @thiteanish @ggerganov
Only 26 seconds / token but it鈥檚 something!
1
17
Replying to @thiteanish @ggerganov
324secs on Pixel 6 is too slow for building an app around, but maybe this model version params cut be cut down somehow?
Replying to @thiteanish @ggerganov
I wonder how it will do with @kdrag0n NestBox!
1
2
Replying to @thiteanish @ggerganov
Does the pixel have that new tensor chip? Wonder if we could get it to leverage it to speed up generation
3
3
Replying to @thiteanish @ggerganov
Holy shit is termux still a thing?
3
3