What the robots.txt file is for

The role of the robots.txt file

The robots.txt file is a simple text file placed at the root of your site. It gives instructions to search engine robots: which parts of the site they can crawl, and which they should leave aside.

It's used, for example, to prevent search engines from wasting time on pointless pages such as certain technical or administration pages.

What it doesn't do

Beware of a common misunderstanding: robots.txt is not a security tool. It politely asks robots not to crawl a page, but it doesn't protect it or hide it from the public.

• It doesn't replace a password
• A blocked page can still appear in the results
• Malicious robots can ignore it

The mistakes to avoid

The most serious mistake is accidentally blocking the entire site, which often happens during a poorly configured launch. The result: Google no longer crawls anything and traffic drops. You should also avoid blocking resources needed to display pages correctly.

When in doubt, it's better to block nothing than to block by mistake. Skywebis systematically checks this file at launch to avoid these unpleasant surprises.

Frequently asked questions