What does a search crawler (bot, spider) do?

A search crawler (also called a bot or spider) is a program that a search engine (for example, Google or Bing) uses to automatically traverse and analyze websites on the internet.

#What the crawler does:

1. Finds pages Starts from known URLs, then follows links inside pages, opens other resources, expanding the map of the internet.
2. Scans the content Reads the HTML, headings, text, meta tags, links, images - everything present in the page's code.
3. Evaluates accessibility Checks whether the page can be indexed (looks at robots.txt, noindex, status codes).
4. Detects updates Periodically comes back to see whether something new has appeared or something old has changed.
5. Passes data on for indexing Whatever it considers useful, it passes to the search index, where it is then processed for ranking.

#What the crawler does not do:

• It does not fill in forms,
• It does not always execute JavaScript (depends on the engine),
• It does not view pages that are closed off or password-protected.

Conclusion: A search crawler is like a robotic archivist that constantly roams the internet, collecting data for the search engine. If it has not come - the page does not exist for search.