0

Has anyone successfully scraped/mirrored a site which is offered as a Tor Onion Service? If so, how did you manage? I've been researching this for a few days, and come up empty handed.

Improve this question
7
  • 1
    What do you mean by "a Tor site" and why would "scraping" it be any different from any other site?
    – cacahuatl
    Mar 12, 2017 at 22:48
  • Can you please describe what exactly do you want/intend to do? What is exact meaning of a "scraping" in your task?
    – Alexey Vesnin ♦
    Mar 13, 2017 at 0:01
  • By TOR site I am referring to a .onion domain. It's different from scraping a regular site based on the necessity of using a TOR browser to access. This includes I2P (I think that's the acronymn), and any other browser used to access .onion sites. You can't use tools such as Octoparse or Httrack to scrape data from .onion sites.
    – user16668
    Mar 13, 2017 at 1:00
  • By scraping I am referring to the ability to pull data from .onion sites in a manner similar to the surface web.
    – user16668
    Mar 13, 2017 at 1:04
  • Sorry for posting the reply as an answer Alexey... I'm still trying to get used to the stack exchange app
    – user16668
    Mar 13, 2017 at 1:06

1 Answer 1

Reset to default
1

Web sites which are offered Tor Onion Services (.onion domain) can be downloaded/mirrored/scraped like any other websites. You only need to tell your favorite tool to use the Tor software as a proxy. I usually use wget for such purposes. So I added the following lines to my .wgetrc:

use_proxy=yes
http_proxy=127.0.0.1:8118
https_proxy=127.0.0.1:8118

Then I just issue the command wget http://bylu6d6nx3og7shy.onion/. This downloads the index.html. For mirroring I would add the -m option and wget will download the complete site.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .