Tor Button handles the domain islation, which also includes how the "New Tor Circuit for this Site" button works.
IsolateSOCKSAuth is a default enabled isolation flag for a tor
SocksPort. It means that it takes the SOCKS5 username and password authentication credentials and if they're different from others, it will use a new circuit. Tor Browser is patched to allow Tor Button to edit the authentication credentials for the SocksProxy.
When you load up
example.org Tor Button creates a new set of credentials for it, these are
example.org for the login and a random hex string for the password. These will be re-used for each resource associated with that "first party" domain.
When you use Tor Button to choose a new circuit for the site, Tor Button generates a new hex string for the password (it used to just use
2,..etc but this was changed recently, see #19206 but it's possible this is still the current behaviour in versions included in Tor Browser releases), this therefor causes it to use a new circuit.
The code you want to read is in Tor Button at
src/components/domain-isolator.js, I recant my earlier assertion of there not being an API call available, it may be possible if you can call the
tor.newCircuitForDomain() function in Tor Button from another extension (just how bad is browser security anyway?).
However, from your description you probably never actually want to do this and I'm guessing that you're going to be a bit of a bastard and abuse Tor to bypass blacklisting, giving it an even worse reputation and I'd kindly ask you not to, even if that isn't your plan it may be what the effects of your plan look like to website administrators. Thanks.