How the Google Crawl Works and How to Request It
The whole point of SEO is to get you noticed where people search. Google. But have you ever wondered how the Google algorithm works? How does it crawl your site? Why do SEO tasks manage to help improve the way Google sees your site. Maybe you’ve updated a few pages, how can you help Google to find that new content or request a crawl if you’ve done a website overhaul.
All great questions, and we’ll go over all of them today so you can better understand how Search Engines, like Google, actually work.
How Google Crawls the Web
Straight from Google itself, it follows three basic steps when you enter a search on the web. It crawls for relevant content. It indexes all pages of websites. And then it serves results based on relevancy of the query from what it finds in its index. What we want to focus on here is the crawling part of the process. Google calls it the Googlebot. But essentially it’s part of the algorithm called “spiders” that visits new and updated pages and then adds them to the bigger Google index. Googlebot makes it seem like a single entity. But in reality, huge set of computers crawl and index billions of web pages every single day. The spiders use the Google algorithm to monitor content and web pages.
From Google, “Google’s crawl process begins with a list of web page URLs, generated from previous crawl processes, augmented by Sitemap data provided by website owners. When Googlebot visits a page it finds links on the page and adds them to its list of pages to crawl. New sites, changes to existing sites, and dead links are noted and used to update the Google index.”
There are ways to keep it from crawling a web page. For instance, if a user needed to login to get information, Google would not crawl that page. You can also block it in robots.txt. And previously crawled pages, or duplicate content gets crawled much less frequently than new pages and content.
The Nitty Gritty
There’s different bots that crawl for mobile versus desktop. And sites need mobile optimization in order to rank on either the mobile or desktop index. These bots simulate what it’s like for a user to visit a site using either desktop or a mobile device. It crawls all those pages using either mobile or desktop. Then it uses a secondary crawler to crawl just a couple pages using the opposite option. For example if it primarily crawled using the mobile crawler, the second crawler would be using the desktop one.
Once a page is crawled, it’s added to the Google Index. Google views webpages as a document. The bots process all content, tags, title and alt tags, images, videos and more. Much like an index in a book, Google stores all that information.It takes into account the keywords, the content,the website freshness. But this index contains hundreds of billions of web pages with more added every day. Essentially, Google adds all pages with a certain keyword to the index of that word. So optimizing for that keyword helps Google take note and rank that keyword higher in its index.
How to Request a Google Crawl
There’s several reasons why Google would crawl or re-crawl a web page. New content comes to mind first. If you’ve re-launched your website, created new pages, or made significant changes to your current site, you can request that Google re-crawl, or re-index your page. But this process doesn’t happen instantly or in a matter of hours. Re-crawling a site and the URLs involved in it can take anywhere from days to weeks.
There’s two different things you can do. If you’ve optimized and updated just a few pages, you can submit them via the URL Inspection tool. Once you’ve added in the URL of the page you want it to crawl, just click “Request Indexing”. The tool will provide an instant analysis. And if it doesn’t find any indexing issues, it will add it to the queue. Google still prioritizes helpful and useful content. If it sees the site as irrelevant or spammy, the index might not ever happen.
The other option is for your entire website if you want to re-index the entire thing. You need to submit your sitemap. Your sitemap is crucial and is basically how Google discovers your site. There’s lots of things that go into a good sitemap: metadata, information about alternate languages or video info. If the sitemap hasn’t changed, there’s no point in re-submitting it to Google. This tool is exclusively for new websites or after you have re-created, re-branded, and re-launched an old version of your site with new content and branding.
When you’re ready to resubmit, go to Google’s Sitemaps Report tool. You need to follow several steps to make sure that google receives your request and re-crawls it the next time they visit your site. It’s technical, but not impossible to implement into your site’s code.
If you’re just getting started, Google recommends submitting just your home page for a re-crawl. It’s the most important page, so having a well-optimized and strong home page will help Google with the index of the rist of your site.
How to Improve Your Page for a Crawl
So if you’re ready for Google to see your site in a whole new light, you’ve got to be ready to put in the world. You need to make sure your site has clean, optimized, and helpful content with good keywords specific to your area of expertise. Make sure to optimize your alt and meta tags and descriptions so that Google understands the content like video and images that don’t have words. You need to build up your site authority and relevancy with continuous good content. And you need to always be improving your site and your pages with SEO.
And if you need help with all that, many SEO and website services include re-submitting your site to Google. Do your research and find a company that can help you reach your potential and climb in the rankings. And as a hint, we do that here at Boostability and a price for any budget! Give us a call!
