"Hosting SQLite databases on Github Pages" is absolutely brilliant: it adds a virtual filesystem to SQLite-compiled-to-WebAssembly in order to fetch pages from the database using HTTP range requests phiresky.github.io/blog/2021…
47
969
137
4,069
Check out this demo: I run the SQL query "select country_code, long_name from wdi_country order by rowid desc limit 100" and it fetches just 54.2KB of new data (across 49 small HTTP requests) to return 100 results - from a statically hosted database file that's 668.8MB!

May 2, 2021 · 7:09 PM UTC

9
70
5
373
GIF
Looks like the core magic here is only around 300 lines of (devastatingly clever) code github.com/phiresky/sql.js-h…
2
16
1
192
phiresky points out that my "order by row id desc" demo is actually a worst case scenario - if you try the same thing without the desc it only takes 6 requests! select country_code, long_name from wdi_country order by rowid limit 100 news.ycombinator.com/item?id…
2
1
88
Here's an application of this technique at a much larger scale: static.wiki runs HTTP range requests against a 43GB SQLite database on S3 to provide search-by-title and display of Wikipedia articles
1
7
37
Replying to @simonw
does it work with the 33GB Who's on First sqlite DB? :)
2
3
oh hang on it actually just might be able to do that... at least it should work fine for "where id = XXX" queries against an index - maybe even ancestor/descendant queries with carefully designed indexes cc @alloftheplaces
1
1
more replies
Replying to @simonw
This is pretty cool.
1
Replying to @simonw
So next version of datasette will be written in Rust and running in the browser? :D
1
19
Replying to @simonw
That’s insane, but awesome
1
Replying to @simonw
Cool cool cool. We tried to do something similar back in 99 or so integrated in a desktop publishing tool. I guess it was a bit before its time because no one bought the stuff and the company went bankrupt later.
Replying to @simonw
"across 49 small HTTP requests" <<< why the large number of requests? That sounds a bit high, and potentially problematic for anyone on latent or unreliable connections.
1
Load more