Silk Road forums

Discussion => Silk Road discussion => Topic started by: dmtnexus on October 04, 2013, 10:23 pm

Title: Who's making a backup of the forum? Post here
Post by: dmtnexus on October 04, 2013, 10:23 pm
Let's make a list.

1. dmtnexus
2. datalore
3. StExo
4. snailgod
5. peaceloveharmony

Who else?

StExo's backup: https://anonfiles.com/file/2cb4eefb1492f43b9510a6186bc66259
.onion mirror: http://2bewt43pu6ymqutq.onion/
Title: Re: Who's making a backup of the forum? Post here
Post by: Bungee54 on October 04, 2013, 10:56 pm
anyone who has a complete backup let us know and we will put it on our own onion site as well for future traders to learn all these great things !

please let us know!

ponderingthoughts@Safe-mail.net
Title: Re: Who's making a backup of the forum? Post here
Post by: sleeptight on October 05, 2013, 12:07 am
How do you do that?
Title: Re: Who's making a backup of the forum? Post here
Post by: datalore on October 05, 2013, 04:20 am
How do you do that?

The two methods I'm using are:

Code: [Select]
torify httrack http://dkn255hz262ypmii.onion/index.php -W -O "/home/sapient/websites/sr_sec"  -%v -r4 -c5 -C -i
---and/or---

Code: [Select]
torify wget -mk http://dkn255hz262ypmii.onion/index.php

I'm not really sure if one is better than the other, although the httrack command I'm using running 5 parallel connections while wget only fetches one page at a time. I'm pretty sure you could also use curl (lol).
Title: Re: Who's making a backup of the forum? Post here
Post by: datalore on October 05, 2013, 04:29 am
I hope someone is pulling the vendor only forum. There is probably some valuable information on there that we should preserve.I wasn't a vendor or I would do it myself.
Title: Re: Who's making a backup of the forum? Post here
Post by: dmtnexus on October 05, 2013, 04:33 am
We shouldn't really encourage more people to put the strain on the server. Remember yesterday, errors popping up all the time? Those error pages will make their way into our backups should the same thing happen again. I will publish mine once it's done, which by my calculations will be in about 9 days.

Edit: I'm making my backup whilst logged in, so the vendor roundtable will be included.
Title: Re: Who's making a backup of the forum? Post here
Post by: dmtnexus on October 05, 2013, 05:08 am
I'm using 16 simultaneous connections and still the ETA is 9 days. I must be doing something wrong, LOL.
Title: Re: Who's making a backup of the forum? Post here
Post by: dmtnexus on October 06, 2013, 09:39 pm
Does anyone have a full backup yet? Mine is at 25%.
Title: Re: Who's making a backup of the forum? Post here
Post by: midlandsmafia on October 06, 2013, 10:16 pm
If someone could let me know too of any backups once completed..

Also why are backups being created, are the forums going down ?
Title: Re: Who's making a backup of the forum? Post here
Post by: dmtnexus on October 06, 2013, 10:31 pm
http://dkn255hz262ypmii.onion/index.php?action=stats

Total topics: 84488

Actually, I never realised that the reason every topic ID doesn't correspond to an actual topic is because of spammers. The forum software is at fault because you only have to enter the captcha once and you can spam to your heart's content.
Title: Re: Who's making a backup of the forum? Post here
Post by: snailgod on October 06, 2013, 11:06 pm
I'm downloading it but it's taking its time. Apparently wget doesn't support multiple concurrent connections. I'm at 2.3 gigs right now or over 51,000 files.
Title: Re: Who's making a backup of the forum? Post here
Post by: dmtnexus on October 06, 2013, 11:14 pm
I'm at 2.4GB. 160 simultaneous connections.

Are you running wget -m? I approximated it would take me at least 25 days, so that plan was scrapped.
Title: Re: Who's making a backup of the forum? Post here
Post by: Cannaisseur on October 06, 2013, 11:35 pm
can anyone point me to a backup of the silkroad site for the texts of the listings?
I remember a site that was sorting by price before sr could, that was basically a archive of sr lisitngs in text
Title: Re: Who's making a backup of the forum? Post here
Post by: snailgod on October 07, 2013, 08:36 am
I'm at 2.4GB. 160 simultaneous connections.

Are you running wget -m? I approximated it would take me at least 25 days, so that plan was scrapped.

I originally went with -m, but after a couple of days, I decided to restart it with just -rk as I figured five levels were enough. Now I'm thinking it won't make much of a difference. I don't see an option for multiple connections though. And how do you know how far you have to go?
Title: Re: Who's making a backup of the forum? Post here
Post by: ECC_ROT13 on October 07, 2013, 10:42 am
But whenever somebody manages to get a decent recursive crawl of the forums together, posting a copy on Freenet is the way to go.

Before somebody posts the archive somewhere redistributable, they should make sure:

1. That they weren't logged in when they grabbed it (so their username isn't at the head of every HTML file).  Shouldn't be an issue with wget. 

2. It might not be a bad idea to do a 'touch' on each file to get rid of the specific timestamps of each HTML download in the archive before compressing and archiving it. I think something like 'find ./my_archive -exec touch {} \;' should do it.   

3. And a 'chown -R root ./my_archive' wouldn't be a bad idea either, just to strip UIDs of file owners if you're tar'ing it.   

I manually grabbed the threads from Security that I didn't want to lose, but would love the whole archive.
Title: Re: Who's making a backup of the forum? Post here
Post by: dmtnexus on October 07, 2013, 11:37 pm
Phase I is complete: I've downloaded 218879 topic IDs (5.8 GB). Phase II will be to download the remaining pages for topics that contain more than 50 posts. Working on it now!
Title: Re: Who's making a backup of the forum? Post here
Post by: ScoobyDoo on October 08, 2013, 01:17 am
Please keep this forum updated and let us know how we can download a backup copy of the forums!

Many thanks for your hard work.
Title: Re: Who's making a backup of the forum? Post here
Post by: This_is_not_SOCA on October 08, 2013, 01:44 am
Hello - I am on the scrounge.

Do you StExo or you DMTNexus have any desire to make those archives available for download? I have downloaded several threads, mostly from the security forum that I found interesting but would love to have a more inclusive copy.

regards

This is not SOCA

Title: Re: Who's making a backup of the forum? Post here
Post by: dmtnexus on October 08, 2013, 03:09 am
After considerable effort (I'm not a coder), I've whipped up a bash script that's at this very moment generating a list of the remaining URLs to be downloaded. Phase II will begin soon.

Yeah, of course our backups will be available for download; that's the whole point.
Title: Re: Who's making a backup of the forum? Post here
Post by: ECC_ROT13 on October 08, 2013, 03:37 am
Again if somebody can provide me with an easier way of providing this file for people to download, I will happily distribute it! But uploading 300mb over Tor is an extremely slow process.
I've been thinking about it, and I don't have any good ideas other than uploading it over Tor or to Freenet.  Anything else is going to provide a trail back to the last guy on SRF who said he was gonna upload it.  i.e. you.

heh, actually, at this point, the *first* person to upload a tarball of SRF to a clearnet site (unless it was available elsewhere first) is gonna look like StExo to anybody watching these forums.  :)

I say u/l via Tor from a dedicated server somewhere that has a Tor instance running and good bandwidth.  I think that's your best bet.  It's a huge pain in the ass, I know.  But I really hope you find a way to do it, because I have this feeling that in a few months, I'll wish I had a thread about X that I remember, and it won't be one I thought to grab a copy of.
Title: Re: Who's making a backup of the forum? Post here
Post by: dmtnexus on October 08, 2013, 04:03 am
Phase II has begun. 13111 URLs left to download.

Personally, I was thinking of creating an .onion mirror. Who needs the entire forum on their hard drive anyway?
Title: Re: Who's making a backup of the forum? Post here
Post by: kmdma on October 08, 2013, 04:56 am
well.. i'd like to read "drug safety" on my tablet offline, 'cause i don't have adsl at home, just mobile 3g, and can't use tor on tablet.

so i could download anywhere then reading on my bed without wasting my few MB/day

(tnx for the machine) :P
Title: Re: Who's making a backup of the forum? Post here
Post by: kmdma on October 08, 2013, 06:41 am
THANK YOU ! i'm going to my mom just to download it :)
Title: Re: Who's making a backup of the forum? Post here
Post by: Tessellated on October 08, 2013, 06:44 am
Here we are guys, FINALLY got it uploaded:
https://anonfiles.com/file/2cb4eefb1492f43b9510a6186bc66259

Take a copy and pass it along peeps :)

Thanks! Will keep it safe for future generations.
Title: Re: Who's making a backup of the forum? Post here
Post by: snailgod on October 08, 2013, 09:27 am
Received. Cheers.
Title: Re: Who's making a backup of the forum? Post here
Post by: peaceloveharmony on October 08, 2013, 12:13 pm
I also started started downloading the complete forum excluding the user profile pages yesterday.

As i rented a server for downloading the forum, i will probably set up an hidden service on this server  with the cleaned up forum backup running on it. Unfortunately the download is quite slow so this will take some time.
Title: Re: Who's making a backup of the forum? Post here
Post by: Wumbo on October 08, 2013, 12:37 pm
So this is the final copy? Is anything else going to be added to this or should I just download it now?
Title: Re: Who's making a backup of the forum? Post here
Post by: kmdma on October 08, 2013, 02:45 pm
got many errors extracting with WinRAR.. is it normal?




Title: Re: Who's making a backup of the forum? Post here
Post by: This_is_not_SOCA on October 08, 2013, 02:59 pm
StExo,

Again - Thank you for selflessly backing the forums up and making them available.

good luck for the future
Title: Re: Who's making a backup of the forum? Post here
Post by: ScoobyDoo on October 08, 2013, 10:02 pm
Here we are guys, FINALLY got it uploaded:
https://anonfiles.com/file/2cb4eefb1492f43b9510a6186bc66259

Take a copy and pass it along peeps :)

Many thanks, brother!
+1
Title: Re: Who's making a backup of the forum? Post here
Post by: flwrchlds9 on October 08, 2013, 10:27 pm
Again if somebody can provide me with an easier way of providing this file for people to download, I will happily distribute it! But uploading 300mb over Tor is an extremely slow process.

Would this not be great use of usenet?? Encrypt the file with a easy password and post on usenet with innocent subject.



Title: Re: Who's making a backup of the forum? Post here
Post by: ECC_ROT13 on October 08, 2013, 11:39 pm
StExo - thanks so much for posting that! I know it was probably a huge pain in the ass, but you've done everyone a great service by posting it.