Can onion service owners prevent their site from being indexed by darknet search by using a meta tag?

Question

There are many multiple darknet search engines including Ahmia.fi and Torch. These darknet search engines index new hidden services every day. Can darknet site owners use:

    <meta name="robots" content="noindex, nofollow">

to stop darknet search engines from indexing their site? This meta tag works for normal websites with search engines such as Google, Yahoo, Bing, etc. I know that if a darknet search engine honors robots.txt files (such as Ahmia.fi) site owners can use this to hide their website through this:

    User-agent: *
    Disallow: /

However, if the darknet search engine doesn't honor robots.txt files, does the meta tag work in hiding the site?

Answer 1

After some research, I believe the answer is: yes and no. Here's why:

The meta tag is similar to robots.txt file in the sense that this is just telling crawlers who visit the site to not crawl/index the website. Both don't actually prevent robots from indexing the site, they only tells robots not to. Whether the crawler actually acknowledges them or not is what determines if a website is indexed. Most (good) crawlers do accept this. So now that's clear, let's answer the original question.

---

Can hidden service owners actually prevent their site from being indexed using:

<meta name="robots" content="noindex, nofollow">

? Well from what we established above, the meta tag only tells the crawler not to and doesn't actually prevent this. So in theory, yes, a darknet search engine should be able to index a hidden service even with this meta tag. However, darknet search engines might respect what hidden services wish for and may not index the onion service if requested. I know Ahmia.fi follows robots.txt but I never saw anything about meta tags so I don't exactly know. I don't really know about other darknet search engines. Also, hidden services might be hidden from darknet search engines through other ways:

• Using dynamic content generation
• Password-protecting access to content
• Contacting darknet search engines to not index their site
• Being in the blacklist category of darknet search engines (this may not work for all darknet search engines which don't censor sites like Torch)

So technically, the meta tag may or may not work, it depends on the darknet search engines. Plus, darknet search engines can't find every hidden service. In fact, they probably only have found a small portion of the dark web, since crawling and indexing hidden services can be tricky. All in all, hidden services can try to use the meta tag if they want to hide their site, but there isn't a guarantee so they might use a robots.txt file or any of the other methods listed above as further ways of hiding/blocking darknet search engines from crawling and/or indexing their site.