By Tia Dang | last updated 7th December 2020


What is Crawling?

The first step of making a search engine identify your page and view it in search results is crawling. Search engines discover and browse web pages using their own web crawlers.

How It Works

Web crawlers go by several names, like spiders, robots, and bots. Before they can produce the correct pages for keywords and phrases, or the words that people use to locate a helpful page, they must be crawled and indexed.

Key point: Web crawlers will periodically need to revisit pages to make sure the latest version of the content is indexed.

Related Terms

Do you want to contribute to this page?