What is Crawling?
The first step of making a search engine identify your page and view it in search results is crawling. Search engines discover and browse web pages using their own web crawlers.
How It Works
Web crawlers go by several names, like spiders, robots, and bots. Before they can produce the correct pages for keywords and phrases, or the words that people use to locate a helpful page, they must be crawled and indexed.
Key point: Web crawlers will periodically need to revisit pages to make sure the latest version of the content is indexed.