Robots.txt is a plain text file placed at the root of a website that tells search engine crawlers which pages or folders they can and cannot request. It helps guide crawl behavior, but it does not reliably keep pages out of search results on its own.
What robots.txt does
The file uses simple rules, usually aimed at specific user agents such as Googlebot, to allow or disallow crawling of certain paths. This is useful when you want to reduce unnecessary crawling of low-value areas like internal search results, filtered URLs, staging sections, or duplicate parameter-based pages.
Common directives include User-agent to name the crawler and Disallow to block a path from being crawled. Some setups also use Allow for exceptions and include a sitemap reference to help crawlers find important URLs faster.
Why robots.txt matters for SEO
Robots.txt matters because crawl budget is limited, especially on larger sites. If search engines spend time crawling thin, duplicate, or unimportant URLs, they may discover and refresh key commercial pages less efficiently. A clean robots.txt file helps focus crawler attention on product pages, service pages, category pages, and other content that supports rankings and revenue.
It also prevents avoidable SEO mistakes. For example, teams sometimes launch faceted navigation or site search pages that create thousands of crawlable URLs. Blocking those sections in robots.txt can reduce crawl waste and make reporting cleaner in SEO tools and server logs.
Important: robots.txt is not a security tool. Sensitive pages should be protected with authentication or proper server controls. If you need a page removed from search results, use noindex where appropriate or remove access entirely.
Practical example
Imagine an ecommerce store with URLs for on-site search and filter combinations:
/search/
/products?color=blue&size=large
If these pages create thousands of low-value variations, the robots.txt file might block the search folder while still allowing core product and category pages to be crawled. A simple setup could tell major crawlers not to request the internal search section, while leaving the main catalog open.
Basic checklist
Place the file in the site root, use correct syntax, test rules before publishing, and make sure you are not accidentally blocking CSS, JavaScript, or important landing pages. Review the file after migrations, redesigns, or CMS changes, since an outdated robots.txt can quietly block pages that should rank.