After some research, I believe the answer is: yes and no. Here's why:
The meta tag is similar to
robots.txt file in the sense that this is just telling crawlers who visit the site to not crawl/index the website. Both don't actually prevent robots from indexing the site, they only tells robots not to. Whether the crawler actually acknowledges them or not is what determines if a website is indexed. Most (good) crawlers do accept this. So now that's clear, let's answer the original question.
Can hidden service owners actually prevent their site from being indexed using:
<meta name="robots" content="noindex, nofollow">
? Well from what we established above, the meta tag only tells the crawler not to and doesn't actually prevent this. So in theory, yes, a darknet search engine should be able to index a hidden service even with this meta tag. However, darknet search engines might respect what hidden services wish for and may not index the onion service if requested. I know Ahmia.fi follows
robots.txt but I never saw anything about meta tags so I don't exactly know. I don't really know about other darknet search engines. Also, hidden services might be hidden from darknet search engines through other ways:
- Using dynamic content generation
- Password-protecting access to content
- Contacting darknet search engines to not index their site
- Being in the blacklist category of darknet search engines (this may not work for all darknet search engines which don't censor sites like Torch)
So technically, the meta tag may or may not work, it depends on the darknet search engines. Plus, darknet search engines can't find every hidden service. In fact, they probably only have found a small portion of the dark web, since crawling and indexing hidden services can be tricky. All in all, hidden services can try to use the meta tag if they want to hide their site, but there isn't a guarantee so they might use a
robots.txt file or any of the other methods listed above as further ways of hiding/blocking darknet search engines from crawling and/or indexing their site.