Evolution forums mirror/scrapes torrent released

EDIT: the old torrent is now deprecated; please use my full DNM archive instead.

To help with the fallout from the Evolution exit scam and let people look up discussions of vendors, their feedback threads, and anything they might've posted to the Evolution forums like contact info, I am releasing my mirrors of the Evolution forums:

This is a 1.1GB XZ-compressed tarball which unpacks to ~290GB, containing mirrors from 2014-01 - 2015-03; as with the Evolution market mirrors, each subfolder is a single wget crawl dated when it finished, of varying levels of completeness. These are the HTML pages displayed by the forum. Not the best browsing or viewing experience, but it's much better than nothing.

Note that the mirrors are heavily redundant since forums are cumulative, so unless you're looking for something you think was deleted, you may only want to extract the last few crawls. A tar option along the lines of --wildcards "*2015-03-*" should work.

For the scrape of the market itself, not the forums, see https://www.reddit.com/r/DarkNetMarkets/comments/2zllmv/evolution_market_mirrorscrapes_torrent_released/

(Hopefully this scrape won't get me shadowbanned too.)


Comments


[58 Points] Therealfed1:

this guy is a fucking DNM god


[26 Points] None:

gwern what you do for the community is awesome, thankyou on behalf of everyone.


[18 Points] Fr0styXT:

LONG LIVE GWERN


[16 Points] sharpshooter789:

(Hopefully this scrape won't get me shadowbanned too.)

So thats why the marketplace post vanished. I thought you decided to pull it after a period of time. Thanks a lot.

Maybe one of these days, I will write a multithreaded scraper that only stores a single copy of a resource.

FYI for those using OSX. Tar doesn't use --wildcard it uses --include.

tar xvf evolution-forums.tar.xz --include "*2015-03-*"

There appears to be 586433 files based on

find evolution-forums -type f -print | wc -l

The size of the contents from March, 2015-present is ~22Gb.

Even with an SSD just using grep will be rather inefficient because of all the overhead since there are so many files. After some googling, I found a related stack exchange post. The solution appears to be a ghetto way of using grep concurrently using find and xargs. The results were tremendous

the time with his trick
real    3m24.358s
user    1m27.654s
sys     9m40.316s


time without trick
real    235m12.823s
user    38m57.763s
sys     38m8.301s

Another possible approach to improving greps performance under these conditions would be to mount a RAM disk and move a portion of the files over there and then use grep. I have a decent amount of RAM so this is actually a viable option for me.

The last option (more of a joke really), would be to buy a HPUX server that implements MemFs. These servers had insane access speeds that far surpass any SSD. The thing is, these are enterprise servers so no ordinary person can afford one.


[14 Points] None:

You are the best bro


[3 Points] awaywethrowwwww:

Good stuff!


[4 Points] PotWillKillYou:

This is beautiful. I actually needed something off the forums. Thanks!


[3 Points] SmokeMethInhalesatan:

Thank's Homie G


[3 Points] Zara02:

Is this without the dox by Debbie? Feel bad for the guy already.


[3 Points] None:

Much appreciated. And you're correct, this is a lot more compressed than the Evo market mirror. This time, however, the redundancies make sense to me. On a forum, each page has the same layout, differing only in the usernames, thread posting dates, and the contents of the thread. The rest of the links remain unaltered.


[3 Points] griffindoodle:

i downloaded a dump ~12hrs ago that was i believe 3.3gb. forgive my noobish question but what is the difference between the two? is it strictly a compression thing? also can somebody recommend a good way of extracting such a large file?

edit: gwern you are the best thank you thank you thank you! since your write up months ago i never kept any decent amount of money in evo wallet. only lost 0.04btc


[2 Points] None:

Thank you again


[2 Points] None:

[deleted]


[1 Points] toothbrush20:

The karma for this is going crazy fast. More upvotes!!! Thanks gwern


[1 Points] Evo-Sacky:

You are a pure motherfucking life-saver mate! I fucking love you in no fucking homo way! You're the man!!! I thought I lost all of my feedback but thank god you exist! :)


[1 Points] theevoinsider:

Do you have the vendor and staff section or just the public sections?


[1 Points] None:

[removed]


[1 Points] numorate:

Anyone else getting tons of this?

tar: evolution-forums/2015-02-28/viewtopic.php?pid=208013: Cannot open: Bad message


[1 Points] TNS_v_Avpd:

Gwern can you help me please? I downloaded and extracted the file and everything was fine but I want to delete it now. It's been deleting for 10+ hours. Do you know of a quicker way? Thanks. (Also I'm using Unix)


[1 Points] charlescharlieq:

fucking amazing thanks!


[1 Points] midnightworlock:

What a dude, thanks man


[1 Points] p00rky:

omg i love you


[0 Points] AllHailTheCATS:

Can someone explain to me whats going on here? does this mean people can recover bitcoing or important information or just view the mirrors?


[0 Points] PrettyTwoFaced:

Neat!


[0 Points] None:

Thanks.


[0 Points] Torcarders:

Useless shit html copied files no php no sql This is shit torrent like market torrent


[-2 Points] thatannoyingcunt:

where can i get the marketplace dump?