“Archiving GitHub”, Gwern2011-03-20 (Haskell, Internet archiving, tutorial)⁠:

Scraping and downloading Haskell-related repositories from GitHub

Tutorial of how to write a Haskell program to scrape Haskell-related repositories on GitHub and download them for offline installation, search, reference, and source code analysis, using TagSoup & curl.

Obsolete

• Why Download?
• Archiving GitHub
  • Parsing Pages
  • Downloading Pages (the Lazy Way)
  • Spidering (the Lazy Way)
  • Filtering Repositories
  • Transforming Links (the Lazy Way)
  • Shelling out to Git
    • Unique Repositories
  • The Script
    • The Script Golfed
• Exercises for the Reader