There are search engines for Tor hidden services, analogous to Google, Yahoo, etc. for the greater Internet. How can sites avoid being indexed by these search engines? Does a conventional "robots.txt" work?
robots.txt only works if the search engine provider listens to it, which they are not guaranteed to do.
You will only be indexed if the spider that the search engine is using can not find you, and the only way to have them not find you is to never have a link to your website posted to another website. (which is impossible to guarantee too)
You can try robots.txt and you can try not having any links anywhere like the hidden wiki. If that does not work, contact the search engine and ask them to take you off