I was thinking of having a crawler visit all the pages on the dark web. But there's no broad similarity in the way the websites are named.

2 Answers 2


Actually your suggestion is what I started before. The main problem is the seeds. We do not have all the onion addresses. I recommend you to read my project details named OnionHarvester: OnionHarvester

  • That's a pretty basic approach and unfortunately, I can't seem to find any other way either. TOR is slow and harvesting the details of that number of websites or even to check if their server is up would be tedious and would take a lot of time. Don't understand why someone has downvoted your comment though. If you've run the code yourself, why don't you share the results you might have got?! Mar 5, 2019 at 13:47
  • IN your blog, you had linked your question and there I found out something interesting from the Ethical TOR Research. It reads: *It is not acceptable to run an HSDir, harvest onion addresses, and publish or connect to those onion addresses.* Could you explain this to me? I wanna cross-check my understanding as this seems something serious. Mar 5, 2019 at 13:56
  • Actually TOR does not like to Hidden Services get discovered in any way. And I myself think that real Criminals are not in public sites such as The Hidden Wiki. Therefore a way should be available to getting the addresses: 1) Asking Hidden Service owners such as Ahmia does 2) Running rough HSDirs (which the hit rate of Onion addresses is low) 3) Bruteforcing the address space which is acceptable for now But with Onion v3 (56 chars) it seems to be infeasible!
    – Mir Saman
    Mar 8, 2019 at 6:50
  • @saurabh000345 it means tor hidden service are hidden for a reason, whether illegal or legal. Your search engine break the foundation of them
    – unixandria
    Mar 8, 2019 at 14:30

Please do not attempt this

The entire purpose of hidden services and such is privacy. You suggestions would destroy that. A search engine to find the supposed to be unfindable? A paradox in itself. Some of these services are hidden for a very good reason, such as humanitarian organization under oppressive regime. Your search of the dark web would help the oppressive regime out a lot

If you really need to access a hidden service, contact the server operator. Some services do already offer something similar, like this

Hidden service are hidden and by design of the tor network should remain that way

  • Well, my object is not to really build a search engine. Any website that can be crawled can literally be visited in the first place. I don't see therefore, how this would violate the privacy. My main obstacle is to get the possible seed URLs because a TOR URL can have as many characters as one wants. I have seen variations. Those aren't just a set of 16 alphanumeric characters with the onion registered domain. There are tor pages that do something identical like the fresh onions which I can't seem to find all of a sudden. Mar 5, 2019 at 6:22
  • @saurabh000345 How is it not a privacy concern? Your doing the bad guys work for them! You are handing over addresses that they may not have known before
    – unixandria
    Mar 5, 2019 at 11:52
  • I won't be releasing it. Mar 5, 2019 at 13:44
  • @saurabh000345 doubtful. You would not go through this just for yourself. And even if you did, you now provide a single point of access to the data: your computer
    – unixandria
    Mar 8, 2019 at 14:28

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .