Tor is only assigning circuits from a very limited subset of exit nodes

Question

I am researching Tor, and as part of my research I was curious how many distinct exit relays a real-world client would include in building circuits. For my purposes, I am only interested in US exit nodes.

I found this project: https://github.com/mattes/rotating-proxy, which load balances proxy requests across multiple Tor instances. Specifically, HAProxy load balances over N polipo instances connected to the SOCKS port of N tor instances:

                          <-> Polipo 1 <-socks-> Tor Proxy 1
Client <-http->  HAproxy  <-> Polipo 2 <-socks-> Tor Proxy 2
                          <-> Polipo n <-socks-> Tor Proxy n

I modified the Tor arguments in order to maximize the diversity of exit nodes:

/usr/sbin/tor
    --SocksPort {PORT}

    # Allow creating new circuits after 30 seconds
    --NewCircuitPeriod 30

    # Never reuse a circuit older than 120 seconds
    --MaxCircuitDirtiness 120

    # Only use exit nodes in the US
    --ExitNodes {us}

    # Do not operate an exit relay
    --ExitRelay 0

    # Allow exit nodes not in the consensus
    --RefuseUnknownExits 0

    # Operate in client mode only
    --ClientOnly 1

    # Interpret the above ExitNodes argument strictly (nly US)
    --StrictNodes 1

    # Allow exit nodes supporting single hop circuits (idk if this does anything)
    --AllowSingleHopCircuits 1

    --DataDirectory /var/lib/tor/10011
    --PidFile /var/run/tor/10011.pid
    --Log warn syslog
    --RunAsDaemon 1

The test:

• Start the rotating proxy with 100 tor backends
• Repeatedly request ipinfo.io/json through the rotating proxy, log the IP address
• Sort the resulting file by unique IP addresses

Result after 30 minutes:

• 1581 total requests
• 66 unique IP addresses

Checking the haproxy stats page confirms that requests are being load balanced across all 100 tor instances.

Given that there are 500+ unique exit relays in the USA, I would expect far more unique IP addresses.

Why are there only 66 unique IP addresses? It appears that Tor is not rebuilding circuits as often as it should be, according to the arguments. Or it is rebuilding circuits, but keeping the same exit nodes.

Could it be because polipo is keeping a persistent socks connection?

EDIT: UPDATE

I made some improvements and am now seeing better performance. I am still curious if I'm missing anything.

Improvements I made:

• polipo was caching the ipinfo.io requests for each proxy, which obviously makes it hard to test if the IP is changing... I fixed this by adding ipinfo.io to /etc/polipo/uncachable and doing all further tests on my own test server that explicitly forces clients to NOT cache responses

• I enabled the control port on each tor instance so that the daemonized docker script can periodically force a circuit change by sending the newnym signal to the control port ) via https://www.thesprawl.org/research/tor-control-protocol

Here are the new args:

"--NewCircuitPeriod 15",
"--MaxCircuitDirtiness 15",
"--NumEntryGuards 8",
"--CircuitBuildTimeout 5",
"--ExitNodes {us}",
"--ExitRelay 0",
"--RefuseUnknownExits 0",
"--ClientOnly 1",
"--StrictNodes 1",
"--AllowSingleHopCircuits 1",

So far the test has sent 175 requests, with 61 unique IPs... far better than before, in terms of percentages. We'll see.

Answer 1

It is good to see that you want to do research on the Tor network. First please make sure that you have Tor's ethical research guidelines in mind.

In general you try to create new circuits during short time periods. This puts some load on the nodes and on the network in general. Tor offers the possibility to create an own test network and furthermore there is the Tor Path Simulator. This helps you to answer all kinds of questions regarding circuit building inside Tor.

So please use another library and not the live network for your experiments.

Answer 2

Actually you need to tunnel your Tor instances through a different external IPs, better if even in different AS blocks. Use an IP pool and let's see the result ;)