I am trying to use two circuits in parallel with stem. In tor browser, two different websites use two different circuits. I am trying to get similar functionality in my python code. Here is what I am doing:

def scan(*args,**kwargs):
    controller = kwargs['controller']
    dest = kwargs['dest']

    circuit_id = controller.new_circuit(await_build = True)

    def attach_stream(stream):
        print(str(dest) + ' ' + str(stream.circ_id))
        if stream.status == 'NEW':
            controller.attach_stream(stream.id, circuit_id)
    controller.add_event_listener(attach_stream, EventType.STREAM)
    controller.set_conf('__LeaveStreamsUnattached', '1')  # leave stream management to us

    proxy = 'socks5://'
    options = Options()
    options.add_argument('--proxy-server=%s' % proxy)
    driver = webdriver.Chrome(executable_path='./chromedriver',options=options)

controller = connect_port(port=9051)

    kw1 = {"controller": controller, "dest":"https://api.ipify.org"}
    kw2 = {"controller": controller, "dest":"https://icanhazip.com/ip"}
    executor = ThreadPoolExecutor(max_workers=2)
    executor.submit(scan, **kw1)

Basically launching two threads, each creates a circuit, hooking event listener, launching chrome and getting IP. Now the problem is that both instances of chrome, accessing two separate APIs have the same IP. The print statement in attach_stream prints output like this:

https://api.ipify.org None
https://icanhazip.com/ip None
https://api.ipify.org 58
https://icanhazip.com/ip 58
https://api.ipify.org 59
https://icanhazip.com/ip 59

Basically both threads end up hooking same streams to both destinations, even though I want them to attach to different destinations. I Tried adding these two lines to attach_stream, in hopes of getting it to work but to no avail.

stream.circ_id = circuit_id
stream.target_address = dest

I believe this isnt impossible as tor browser itself is doing something similar, i.e. using different circuits for different websites, but I cannot figure out how to achieve this. TIA.

  • You're attaching two different event handlers to the same controller, and attempting to attach each stream to two different circuits. The same event handler will always win for both streams. And as always, please don't use Tor for scraping the contents of websites (or other malicious things) as it harms the network for everyone else :) – Steve Jul 27 at 20:57
  • I was not using it using it for scraping, was only using selenium to figure out whats happening with the circuits. From your comment, I am assuming I would need two different instances of controller, connected to different ports. Do I need separate tor instances or would a single tor instance with two controllers on two different ports suffice? – Khizar Amin Jul 28 at 8:16

You definitely don't need two tor instances here. And two controllers wouldn't solve the problem of the event handlers acting on both streams. Instead you may want to add the event handler only once outside of a thread (outside of scan()), and instead use a queue. Each thread can create a circuit and add the circuit ID to the queue, and then the event handler pops one circuit ID off the queue and assigns the stream to that circuit. A possible problem with this approach (depending on your requirements) is that the new stream will use one of any of the circuits built, not specifically the one made in that same thread.

| improve this answer | |
  • Yes, that is a problem. Thanks for the response, cleared up a lot of things. – Khizar Amin Jul 29 at 8:50
  • @KhizarAmin If you need to work around that problem, I usually use getsockname() to get the local ip/port of the socket and store the circuit ID that you want to use for that stream in a dictionary indexed by the stream's ip and port. Then when the event handler runs, you can use stream.source_address and stream.source_port to lookup the circuit ID for the new stream. – Steve Jul 29 at 9:23
  • That makes sense. I`ll try that. Thank you. – Khizar Amin Jul 29 at 9:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.