So, I have seen a few posts about the timing attacks revealed at Hack in the Box this month, however what is really surprising to me is that no one has mentioned any of the news related to the MEMEX search engines. While there are posts from April when these got first announced, two days ago one named punkSPIDER finished crawling all of the tor hidden services (no that isn't a typo). The surprising news about this? It only took them 7 hours to do so, and there are only around 7000 of them.
This information was received from a Forbes article posted two days ago Link: http://www.forbes.com/sites/thomasbrewster/2015/06/01/dark-web-vulnerability-scan/
PunkSPIDER is one of the many DARPA (aka US government) backed programs under the name Memex which aim to create more intelligent web crawlers, originally intended for the Deep Web. These are the that more obscure websites that get missed by major search engines like Google. (For example NASA is backing one of the Memex crawlers to look for any rocket designs that might be beneficial to them) PunkSPIDER is one of the more interesting and controversial crawlers because of what it does it. PunkSPIDER is automated to search through websites, along with pages on them, 'poking' them and looking for common (and some not so common) vulnerabilities such as SQLi XXS etc. The controversial part about this crawler is that it posts all of the found weaknesses onto a public searchable database. This is to make sure that some organizations (cough cough NSA) don't keep theses holes in security to themselves for their own benefit, as well as limiting personal gains of hackers by making the holes public. Also, by posting them online the websites are motivated to patch and fix them.
This is where the story gets interesting.Two days ago they reworked their engine in order to crawl through the Hidden Services on tor. Now the Forbes article mentions that they crawled through the entirety of the Hidden Services in just 7 hours. However, this isn't accurate as they stated that only 2100 responded to the https requests, so only those were searched ( about a third, but whether these other 5000 are simply offline for good or they just weren't available at the time is unclear). Either way, they ended up finding 50 sites with vulnerabilities and about 100 flaws. As stated in the article the programmer of PunkSpider says that this is much lower than normal clearweb services. While he suspects this is because many of them are simple static HTML pages, I want to give a nice shoutout to OPSec and security measures of the designers of the pages as well, because I believe this is also a reason for the low percentage of vulnerabilities.
The scary part of this however, is while the PunkSPIDER community website gives a searchable database for the vulnerabilities found on the clear web, these security holes found on the Dark Web have been "filtered" for the time being until they decide what to do with the information. This means that they know 50 .onion websites with *significant** vulnerabilities, and they can do whatever they want with them*
The programmer already stated that they know "at least one" [emphasis added] .onion site (related to
kitten orgiessome type of "weird child porn," that they "don't want the website administrator to fix...before someone in law enforcement hacks it" While I assume most of us on here can agree that an intense child porn website should be taken down by LE (unless it's Bailey Jay as a kid) this leaves them in a situation where they can play "Judge" of the Dark Web websites when it comes to the vulnerabilities deciding which ones to alert admins of, which ones to hand over to LE and which ones to post on their community searchable database (putting a target on their site second only to Mr. Nice Guy's at the moment)
While I think that PunkSPIDER is a great start in the right direction for exposing vulnerabilities, I don't believe that they should be able to pick and choose which sites they want to turn in, or turn the hackers onto.
My opinions aside I wanted to post this in order to inform the members of our community about a possible security issue that hasn't been brought up yet.
TL:DR: PunkSPIDER crawls websites to search for vulnerabilities, and two days ago searched the Tor Hidden Services. They found 50 sites (out of the 2100 they searched (out of the ~7000 total .onion sites)) that had about 100 flaws in total. They are not releasing the flaws on their database like they do with clearweb sites, and have already admitted to plans of turning over flaws of at least 1 site (a bad child porn site) to LE.
Edit: Formatting
I just want to set the record straight a little bit. This crawler DID NOT crawl all hidden services. They crawled all the published hidden service addresses they could find online. I run about 20 hidden services that are not published, and not one had any access over the last two weeks. That's right, not one request over the last two weeks.
This was posted in /r/tor in reference to crawling ALL hidden services by /u/Fuck_the_admins:
As with most things, the kids of /r/darknetmarkets are a few days late and just run with the headlines like they're not sensationalized. Get a clue.