The hidden services bother with a robots.txt file as much as the EU crawlers. They can just start at an Onion link collection and start crawling.
Don’t you see how this improves law-enforcement? Now the criminals won’t dare to do forbidden stuff there. -- some government, somewhere
The hidden services bother with a robots.txt file as much as the EU crawlers.
They can just start at an Onion link collection and start crawling.