0

does onion sites uses robot.txt files to hide or restrict crawlers? as i know normal webservices uses robot.txt files to stop google crwaler from knowing more about the website or limitaions. so does the onion site uses such files?

1 Answer 1

0

Yes, onion sites are just the same web sites with all the standards and conventions

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .