They're discarded on a per-session basis.
Using the 'New Tor Circuit for this Site' will only cause Tor Browser to use a new circuit for the site, it will not clear any other information. That is what the 'New Identity' option is for, see the Tor Browser Design and Implementation.
N.B. On Firefox's Private Browsing windows, cookies and other tracking data is absolutely not removed when a tab is closed. It is unset when the window is closed. This is trivially testable by visiting a site that sets and displays a cookie, closing the tab (but keeping the window open) and opening a new tab on the Private Browsing window and then checking to see if the cookie is still present.
When tested, it can be seen to be still set. The same is true of Chrom{e,ium}s Inconito Window.