Suggest an editImprove this articleRefine the answer for “How do search engine crawlers scan a site?”. Your changes go to moderation before they’re published.Approval requiredContentWhat you’re changing🇺🇸EN🇺🇦UAPreviewTitle (EN)Short answer (EN)Search crawlers scan a site step by step, **like smart but limited guests** who have little time and a clear plan. **Key point:** a crawler scans strictly by the rules, quickly and not endlessly, so a clean structure, sitemap, and no obstacles matter.Shown above the full answer for quick recall.Answer (EN)ImageSearch crawlers scan a site step by step, **like smart but limited guests** who have little time and a clear plan. Here is how it works: --- ### 1. **Choosing starting URLs** A crawler starts with: - already known pages; - links from sitemap.xml; - external links to the site; - previously visited URLs. --- ### 2. **Checking** `robots.txt` Before going inside, the bot checks the `site.com/robots.txt` file and looks at **which sections it is forbidden to scan**. If you closed off a section, it will not go there. --- ### 3. **Loading the page** The bot loads the HTML, **not always waiting for a full JS render**. So content that appears dynamically may simply go unnoticed. --- ### 4. **Extracting links** It pulls all internal and external links out of the HTML and **adds them to the crawl queue**, if they are allowed. --- ### 5. **Repeating and updating** A site is not scanned once and for good. The bot returns regularly: - more often to active and popular pages; - less often to static or weakly linked ones. --- ### 6. **Limits (crawl budget)** Every site has its own limit: how many pages, how often, and at what depth to scan. **The more errors, duplicates, and clutter, the faster the crawler "leaves".** --- ### 7. **Passing data to the index** After scanning, analysis begins: - which pages to include in the index, - which content matters, - which is a duplicate, - what to remove. --- **Summary:** A crawler scans **strictly by the rules, quickly, and not endlessly**. If you want your site to rank well, you need to make it convenient for the crawler: a clean structure, a sitemap, no obstacles.For the reviewerNote to the moderator (optional)Visible only to the moderator. Helps review go faster.