Has anyone successfully scraped/mirrored a site which is offered as a Tor Onion Service? If so, how did you manage? I've been researching this for a few days, and come up empty handed.
Web sites which are offered Tor Onion Services (.onion domain) can be downloaded/mirrored/scraped like any other websites. You only need to tell your favorite tool to use the Tor software as a proxy. I usually use
wget for such purposes. So I added the following lines to my
use_proxy=yes http_proxy=127.0.0.1:8118 https_proxy=127.0.0.1:8118
Then I just issue the command
wget http://bylu6d6nx3og7shy.onion/. This downloads the
index.html. For mirroring I would add the
-m option and wget will download the complete site.