Using stem with multithreading Python

Question

I am trying to use two circuits in parallel with stem. In tor browser, two different websites use two different circuits. I am trying to get similar functionality in my python code. Here is what I am doing:

def scan(*args,**kwargs):
    controller = kwargs['controller']
    dest = kwargs['dest']

    circuit_id = controller.new_circuit(await_build = True)

    def attach_stream(stream):
        print(str(dest) + ' ' + str(stream.circ_id))
        if stream.status == 'NEW':
            controller.attach_stream(stream.id, circuit_id)
    controller.add_event_listener(attach_stream, EventType.STREAM)
    controller.set_conf('__LeaveStreamsUnattached', '1')  # leave stream management to us

    proxy = 'socks5://127.0.0.1:9050'
    options = Options()
    options.add_argument('--proxy-server=%s' % proxy)
    driver = webdriver.Chrome(executable_path='./chromedriver',options=options)
    driver.get(dest)
    controller.remove_event_listener(attach_stream)

controller = connect_port(port=9051)

try:
    kw1 = {"controller": controller, "dest":"https://api.ipify.org"}
    kw2 = {"controller": controller, "dest":"https://icanhazip.com/ip"}
    executor = ThreadPoolExecutor(max_workers=2)
    executor.submit(scan, **kw1)
    executor.submit(scan,**kw2)
finally:
     controller.reset_conf('__LeaveStreamsUnattached')

Basically launching two threads, each creates a circuit, hooking event listener, launching chrome and getting IP. Now the problem is that both instances of chrome, accessing two separate APIs have the same IP. The print statement in attach_stream prints output like this:

https://api.ipify.org None
https://icanhazip.com/ip None
https://api.ipify.org 58
https://icanhazip.com/ip 58
https://api.ipify.org 59
https://icanhazip.com/ip 59

Basically both threads end up hooking same streams to both destinations, even though I want them to attach to different destinations. I Tried adding these two lines to attach_stream, in hopes of getting it to work but to no avail.

stream.circ_id = circuit_id
stream.target_address = dest

I believe this isnt impossible as tor browser itself is doing something similar, i.e. using different circuits for different websites, but I cannot figure out how to achieve this. TIA.

Answer 1

You definitely don't need two tor instances here. And two controllers wouldn't solve the problem of the event handlers acting on both streams. Instead you may want to add the event handler only once outside of a thread (outside of scan()), and instead use a queue. Each thread can create a circuit and add the circuit ID to the queue, and then the event handler pops one circuit ID off the queue and assigns the stream to that circuit. A possible problem with this approach (depending on your requirements) is that the new stream will use one of any of the circuits built, not specifically the one made in that same thread.