Suggest an editImprove this articleRefine the answer for “What does a search crawler (bot, spider) do?”. Your changes go to moderation before they’re published.Approval requiredContentWhat you’re changing🇺🇸EN🇺🇦UAPreviewTitle (EN)Short answer (EN)A **search crawler** (also called a bot or spider) is a program that a search engine (for example, Google or Bing) uses to automatically traverse and analyze websites on the internet. **Key point:** if the crawler has not visited a page, that page does not exist for search.Shown above the full answer for quick recall.Answer (EN)ImageA **search crawler** (also called a bot or spider) is a program that a search engine (for example, Google or Bing) uses to automatically traverse and analyze websites on the internet. ### What the crawler does: 1. **Finds pages** Starts from known URLs, then follows links inside pages, opens other resources, expanding the map of the internet. 2. **Scans the content** Reads the HTML, headings, text, meta tags, links, images - everything present in the page's code. 3. **Evaluates accessibility** Checks whether the page can be indexed (looks at `robots.txt`, `noindex`, status codes). 4. **Detects updates** Periodically comes back to see whether something new has appeared or something old has changed. 5. **Passes data on for indexing** Whatever it considers useful, it passes to the search index, where it is then processed for ranking. ### What the crawler does not do: - It does not fill in forms, - It does not always execute JavaScript (depends on the engine), - It does not view pages that are closed off or password-protected. **Conclusion:** A search crawler is like a robotic archivist that constantly roams the internet, collecting data for the search engine. If it has not come - the page does not exist for search.For the reviewerNote to the moderator (optional)Visible only to the moderator. Helps review go faster.