What does the User-agent directive do?

SEOMiddle~1 min read

The User-agent directive in the robots.txt file specifies which search crawler the set of rules listed below applies to. In simpler terms, it sets the addressee: exactly who Allow, Disallow, and other directives apply to.

You can:

set rules for all crawlers, using User-agent: *
set rules for a specific crawler, for example:
- User-agent: Googlebot
- User-agent: Bingbot

Example:

javascript

User-agent: *
Disallow: /tmp/

User-agent: Googlebot
Allow: /

In this example:

all crawlers cannot scan /tmp/
Googlebot can scan the entire site without restrictions

Summary: User-agent is a pointer to the crawler for which the following rules in the block are written.

Short Answer

Interview ready

Premium

A concise answer to help you respond confidently on this topic during an interview.