I've been a long time follower of the darknet community and various hidden services (I still remember thoose nucleus days...).
When /u/gramsadmin announced the shutdown of Grams I immidiately started to work on my long time dream project, making a deepweb spider which collects data from hidden services. The cross-market search of Grams was broken for the past few months before the shutdown, so I started with scraping down markets. Currently I have a demo database of nearly 10k products (out of about 95k on Dream), 500 vendors, 50k ratings and a total purchase volume of ~ 3.1M USD from Dream Market (which is uploaded to a git repo, see the end of post).
The code name of the project is ounces (another common measurement unit on markets).
The program is currently in a very early state, the code structure is still not stable, and some features still missing. At the moment I'm busy writing the documentation, and making it ready to be deployed on my Raspberry Pi. After it I plan to add Tochka support.
I want share the source code under GPLv3 license. If you have some experience with Python programming and hidden service hosting (with probably a python-flask frontend) and would like to contribute to this project feel free to dm me here or send a message to the email linked to my pgp public key.
git repo: http://m6su7s3ir7dxggwg.onion/ounces/ounces
If you started calculating Dream's revenue from the numbers above, keep in mind that the max number of ratings displayed per product is 100 and I only calculated with the fix price of products (not dealing with bitcoin price changes), so their revenue is much bigger than that. However if you have plenty of time to play with the database, feel free to do it and give us interesting statistics ;)
My PGP key:
-----BEGIN PGP PUBLIC KEY BLOCK-----
mQINBFpEBNgBEACtmuQ8CutVHJKTx5mqUuFDCDd+8RwRpnYHKoYcjyPhvt0ZsJ0s
lhjVTiFq+Ga3KJcgvA6SFZHbIV2C9tfQ0tF51kvfLFTGl+JhnKWuP2EeEi8wuhJU
MkMTvak/Mwojhs5LH8uK1TFcQcAKTo3NuRNfR+osA66L93JwWUONWJWQ/B8ckIe1
sy1Uu1PoHY1VHYLCPQZKXlRoj3OwG7YouVZtj/VTaRZbTadD8fTkj6eFvXuhV3Mp
3SBosmO5fQIYnbqWCkIoWTSBXYTwtJVRbV0GNSdAXULyItYeJ1sLBkJk3R8n8n9M
1Fhy1a9rNJnObvS0M2SrGkYDgapYVBT83F3CnCFT6rQsHuwxv3nOrsKYM1GlDi0r
OWcnqr2mytPnd67Tz4eaKvfo0RmXJ87+6XjNecAM7si0Ody3jI13/U027JWANrXq
xRXG9IVYCnQDvTP2CsKDizVgB/LAwBYkpR6MkK7TH7/P8OQCUT+369fUdM18ySRW
WI13aaa27miNAogo2gtjc9P3PwFd0zKjJA+5s59Y5B4O3Gp/1rawAYE1sE0M5iE1
M0+XxPeZ8rIuOADDV/+jLEnKyrf1ZindOnf0KI3keck4WcaE2oEeUrSf5L3rOZtS
9c34FZXHCG6wXm/5dHPohFzVu2yazbG33RI8gaXDJ7U46pylpZuUu5KOHQARAQAB
tClvdW5jZXMgKG91bmNlcyBtYWluIGtleSkgPG91bmNlc0Bjb2NrLmxpPokCTgQT
AQoAOBYhBLfySbUKN0XB8ADKisrWjYwungJTBQJaRATYAhsDBQsJCAcDBRUKCQgL
BRYCAwEAAh4BAheAAAoJEMrWjYwungJTcpwP/1QOZ8AKHhh/ZFFbe8paKh5iTq0f
Z/qasX634Ss4vbisXsmIMrMwR94x10AmNTQM97SWcdgfDH1U05Mn6zFmV+dFQjtv
Q7/CkFJZf1ajE4KYAt2XFE3Buvc7vgymwQXakcja84ztZLs+K7heUdiBI/Pvu79r
PAijt+d6FZBu7UY1OJ0BobppX2jm7VDv3naSBXe7Ane5S6GHYcTaHGJfpMZwW56H
1ok3epVafFUyTgx0PtIw6oohTpjYzUapIbyulpWNKE4GXJxA0VPuE7fZMR4mwXPX
4wmYre53orVqmFovQNwWTSZjBQeDpl3lT02LbpKMzv4J8EB9fliGO+/cvBQtj2Rl
s8FJptBvEfRncnZFreZA1E9YqINoravkHX4tbt6ypkXNEUAZuZ5fA1jy/m5ws1YZ
QBkkyyEm6c4zYyWO3ByERYYvqRLfGqSupYLtmBeDNPPNudkKsA1LgfElJdFUmLzE
wz6QZbKlLvDzVU6Hmwtkb/N+sA4TvmF1q3lmQJUzrY0lyKy0TW5ufXVuYx32QaaL
mtiIpO4mSfnAyAVSC1qzHayJQ0qBY+Snuodm6SVHWRUiBBAhS06CYEZU7sKmoRKQ
JtViQb9QmFR7SYmn6BQieRZqI8cMDND9v7DVn/Y/DGKD0Tcs6+gAT20F2qjyRlOE
1weozlt2Ay+UszEduQINBFpEBNgBEADR+kKESlkxuX6WSIQ7wyl+6N8XjBuPgPqq
Lbf7BXGdC/TxfGdiS/cL5JHHTH9f4yktNp5K7vc7BQibWvuOPS4mJJELwEPJuQNS
nu6c8zw52ZFPX9aJAXiTMEEKz1Xk2l/pC1x8vCPKe+7Cp/BMqMUUTj7dyZstbIKL
PzmdCfWnTyLw3oZIC9jvgUk5zw2GlWbFwrPmAQDWD3yGuMNWfT8IySBIp+JqTDYk
g0K4MwJfoFdB3PDmDdW0lldp18cGltYmNjP73jAFOfRGHbHLDd6qq+f3HjWm/lBQ
HPyaZA57QqK72uVKf8ZC3hQJL10vdaCas2C4DQjc6/w7j/yg649BFP4zCCfOSUEa
70rHf7BWVkLwqfzwYzY2bWISRfITi4fQ/BWqGuLf5M26pRjQ34B3o1o1n/e95U2o
wzd2J79MNfugtWTilhTosnVLuE2s1R152Bs0uISzvt7yeMogJ1hVBrJm8mFOBEu3
DaENw4Iiu2CLZKVpcGLjR70GEhtrMneRXDw2NwMRof0ma3znb+YS+NM9Z9Ld8b9/
i/xx3HoqkWxhSJ5VrJ1lNhWJH+tAmnpkOAllGxbvRd74Uz2wFJrd+qjXmDNtcHap
X2gCMG5okcCtFzo7zkJIsxA4oRw3ObRiD03OLeNcI4oKrZKz46O5uWR4R7/ZNzbO
uI7I3jrhmwARAQABiQI2BBgBCgAgFiEEt/JJtQo3RcHwAMqKytaNjC6eAlMFAlpE
BNgCGwwACgkQytaNjC6eAlNOpQ/9H+Fzqwj+06cmqBi44ePew9v/VL9XNzLptoHc
qGkYHQeL9R7AVAL8kv77EsfCrODNHtO5ATpXHdH1HmGfo1iSmGLSr5ftOHQ38ig0
6W0qZvbY1AujikuDnkfPFrtDJx/p5+MnjBT9WFQ76zdol5/h3XTTITZ6R8apPL1d
tqRx9KSHBj/dYlTz6GWy0GOIAAR/kJPUKSUMNB0sd8ZmHKW3RbFk4IHb85efE6Bw
LVIekZ6g58JRSoFC2hVS7L7wQwNLm0nHbloG9QLBryRxAGkqSOqX6kOscXKKG3b6
tZe9I9tsnWRcdGbvcxBFAwE3FH8c3J1pECvFjmXwgzNStzJvvSmQFYasl1auHR65
DuDm3JXvkwS0rAsvFCvP/jWFrHaPYVMvRpgBRCHbqSkpu2HRIrKwVGS/FApBDw7M
f/KcpsfrrYFm8stAx7LQFxX73+db0IHtpkZy+WHxxNywlkF2CQndanTcO9n+zsUN
7oRKNiIoDnralsd45RhjN6l1A98X8WOQbt48etoBPhYQFmT+FKiUUyGykUnofFN4
EvWbOE6J7UIWv6BgO745OSKFqh5TXi/azo1qR4DGz3sZdfYKl23jgdooglR666os
cWXx6l8Aht5KXh5zMIDILYcyXOG0X5ZyEp/l3H74d/bUysZMFtCLQKfpcMHBZUyX
TxHaZwY=
=eQcL
-----END PGP PUBLIC KEY BLOCK-----
interesting, your scraping setup is similar to mine maybe we can join work. I will get in touch with you in the near future.