If a site has the file "robots.txt", it displays all the directories and pages on the website that they wish to keep anonymous from the search engine spiders. This may lead to hidden pages.
With Robots.txt Checker you can access the site directly or check if it is indexed in google