The role of the robots.txt file
The robots.txt file is a simple text file placed at the root of your site. It gives instructions to search engine robots: which parts of the site they can crawl, and which they should leave aside.
It's used, for example, to prevent search engines from wasting time on pointless pages such as certain technical or administration pages.
What it doesn't do
Beware of a common misunderstanding: robots.txt is not a security tool. It politely asks robots not to crawl a page, but it doesn't protect it or hide it from the public.
- It doesn't replace a password
- A blocked page can still appear in the results
- Malicious robots can ignore it
The mistakes to avoid
The most serious mistake is accidentally blocking the entire site, which often happens during a poorly configured launch. The result: Google no longer crawls anything and traffic drops. You should also avoid blocking resources needed to display pages correctly.
When in doubt, it's better to block nothing than to block by mistake. Skywebis systematically checks this file at launch to avoid these unpleasant surprises.
Skywebis builds custom, fast, SEO-optimized websites with no forced subscription. You stay the owner of your site.
Discover our solutionsFrequently asked questions
At the root of the site, accessible at yoursite.com/robots.txt.
No, a site works without one. It becomes useful for fine-tuning what search engines crawl.
