Suggest an editImprove this articleRefine the answer for “What is website crawling?”. Your changes go to moderation before they’re published.Approval requiredContentWhat you’re changing🇺🇸EN🇺🇦UAPreviewTitle (EN)Short answer (EN)**Website crawling** is the process by which search bots (crawlers) go through a site's pages to discover what is on them. **Key point:** crawling is not the same as indexing - a bot can crawl a page without including it in the index.Shown above the full answer for quick recall.Answer (EN)Image**Website crawling** is the process by which search bots (crawlers) **go through a site's pages to discover what is on them**. --- ### How it works: 1. **A bot enters the site via a link** For example, Googlebot starts from the homepage or an external link. 2. **It reads the HTML code** It extracts text, headings, links, images, meta tags. 3. **It finds new links** The bot follows them and scans further - this is exactly what "crawling" a site means (crawl). 4. **It saves the data** Everything found is passed on for indexing - to be shown in search later. --- ### What affects crawling: - **robots.txt** - can block access to specific sections. - **Server speed** - if the site is slow, the bot will visit less often. - **Internal linking** - the better the link structure, the deeper the bot goes. - **Crawl budget** - the limit on the number of pages the bot is willing to review on a site. --- ### Important: **Crawling ≠ indexing**. A bot can crawl a page but **not include it in the index** (for example, because of `noindex`, a duplicate, or empty content). --- **Conclusion:** Crawling is like reconnaissance: the bot studies the site to decide what is worth saving and showing to users.For the reviewerNote to the moderator (optional)Visible only to the moderator. Helps review go faster.