What does the User-agent directive do?

SEOMiddle~1 min read

The User-agent directive in the robots.txt file specifies which search bot the set of rules listed below belongs to. In other words, it sets the addressee: exactly who Allow, Disallow, and other directives apply to.

You can:

set rules for all bots, using User-agent: *
set rules for a specific bot, for example:
- User-agent: Googlebot
- User-agent: Bingbot

Example:

javascript

User-agent: *
Disallow: /tmp/

User-agent: Googlebot
Allow: /

In this example:

all bots cannot crawl /tmp/
Googlebot can crawl the whole site without restrictions

Conclusion: User-agent is a pointer to the bot for which the following rules in the block are written.

Short Answer

Interview ready

Premium

A concise answer to help you respond confidently on this topic during an interview.