0

I'm trying to use tor as a proxy for a few HTTP requests as described in Stealthy Crawling using Scrapy, Tor and Privoxy, but instead of privoxy, I'm using tor's built-in HTTP tunnel. This is my tor config:

ControlPort 9051
HTTPTunnelPort 8118

It works locally on my macOS. I install tor as brew install tor and then I run it in one terminal with the configuration above (as tor -f ./tor.cfg). Then I run my Python program in other terminal, and it's able to connect and make requests over tor. It doesn't work on the server though. What I'm doing on the server:

sudo apt-get update
sudo apt-get install tor tor-geoipdb

Then I add logging so I could debug the problems:

touch ./tor.log
chmod 777 ./tor.log
echo "Log info file $(pwd)/tor.log" >> ./tor.cfg

Using info level. I tried debug, but that didn't really uncover anything helpful to me.

Then I copy the config to the global location so the backend service uses it:

sudo cp ./tor.cfg /etc/tor/torrc
sudo service tor start

Here I wait for the tor to start with the following Python script:

import time
from pathlib import Path

while 'Bootstrapped 100%' not in Path('tor.log').read_text():
    time.sleep(1)

It reads the tor.log file and waits until there's Bootstrapped 100% as an indication of the fact tor should be ready to accept requests. When this script ends, I assume tor is ready. I run my Python program which does the HTTP requests over the HTTP tunnel. With sufficient logging level, I get a few

[stem] INFO: Error while receiving a control message (SocketClosed): empty socket content

and then it all fails with an exception

stem.ProtocolError: SIGNAL response contained unrecognized status code: 514

Could someone please explain me what could be wrong or help me to debug this? I don't even know where I should look. Attaching the tor.log just in case someone could understand what's going on.

tor.log

2
  • Hi, please don't use Tor for crawling websites. When the websites that you're crawling decide to block requests from the Tor network, the millions of other Tor users also get blocked, which hurts the network and the other people who use it. – Steve Oct 27 '20 at 13:53
  • This is no extensive unethical crawling what I'm doing. I'm trying to make tens of requests per day with this, but from a free CI machine behind a shared IP, which gets blocked by many websites by default. So what I'm trying to do is to get a different IP and make a few requests, not to abuse the network or the website. – Honza Javorek Oct 27 '20 at 16:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.