0

Has anyone successfully scraped/mirrored a site which is offered as a Tor Onion Service? If so, how did you manage? I've been researching this for a few days, and come up empty handed.

7
  • 1
    What do you mean by "a Tor site" and why would "scraping" it be any different from any other site?
    – cacahuatl
    Mar 12 '17 at 22:48
  • Can you please describe what exactly do you want/intend to do? What is exact meaning of a "scraping" in your task?
    – Alexey Vesnin
    Mar 13 '17 at 0:01
  • By TOR site I am referring to a .onion domain. It's different from scraping a regular site based on the necessity of using a TOR browser to access. This includes I2P (I think that's the acronymn), and any other browser used to access .onion sites. You can't use tools such as Octoparse or Httrack to scrape data from .onion sites.
    – user16668
    Mar 13 '17 at 1:00
  • By scraping I am referring to the ability to pull data from .onion sites in a manner similar to the surface web.
    – user16668
    Mar 13 '17 at 1:04
  • Sorry for posting the reply as an answer Alexey... I'm still trying to get used to the stack exchange app
    – user16668
    Mar 13 '17 at 1:06
1

Web sites which are offered Tor Onion Services (.onion domain) can be downloaded/mirrored/scraped like any other websites. You only need to tell your favorite tool to use the Tor software as a proxy. I usually use wget for such purposes. So I added the following lines to my .wgetrc:

use_proxy=yes
http_proxy=127.0.0.1:8118
https_proxy=127.0.0.1:8118

Then I just issue the command wget http://bylu6d6nx3og7shy.onion/. This downloads the index.html. For mirroring I would add the -m option and wget will download the complete site.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.