Found us from search engine?
We rank high, you can too.
SEOPressor helps you to optimize your on-page SEO for higher & improved search ranking.
By jiathong on August 23, 2018
In SEO we always look at ranks and SERP, but we also need to know the process that happens before that. That is crawling and indexing.
Google ranks web pages that are in their index. If your web page is not indexed, or not correctly indexed, your rankings will be affected.
The web has moved from plain HTML – as an SEO you can embrace that. Learn from JS devs & share SEO knowledge with them. JS’s not going away. – John Mueller, Senior Webmaster Trends Analyst
The thing you need to know is this.
Let’s get a closer look at the whole crawling and indexing process.
Who, or in this case, what is involved in the process?
This is the crawler, also called the spider. Whenever there’s a new web page or any new updates on a webpage, Googlebot will be the first point of contact from the search engine.
What it does is it crawls the web pages and follows all the links in a web page. That way, the bot discover more new links and more new web pages to crawl. Crawled web pages are then passed to Caffeine for indexation.
The is the indexer that was launched in back in 2010. Whatever’s crawled by Googlebot will be indexed by Caffeine and that index is where Google choose which web pages to rank.
Links discovered from rendering will also be sent back to Googlebot to queue for crawling which will result in a second indexation. This is a very important point to keep in mind because one important part of SEO is internal linking. Inter-linking your web pages in your website gives Google a strong signal for things like page rank, authority and also crawl frequency. Which all, at the end of the day affects page ranking.
Here we have a straightforward graphic from this year’s Google i/o which shows you the flow from crawling to indexing and rendering.
That is good for getting a general idea of the whole process, but why don’t we zoom a little closer?
Which means, imagine this, you served Google a meal, but because they don’t have the cutlery to eat it, they can only judge how good it is by looking, and the server won’t be back with the cutlery until they’re done taking orders from 3 more other tables, Google then post a review on Yelp saying that your food is crap.
Does that sound fair and beneficial? Hell no.
You can also go to CanIUse to check out what is and is not supported by Chrome 41.
These are all crucial tools that help you understand the whole crawling, rendering, and indexing process. With that, you’ll have a better idea of where and what went wrong.
1. Googlebot crawls, Caffeine index and render.
2. For HTML web pages, Googlebot requests a page and downloads the HTML, contents are then indexed by Caffeine.
4. Rendering is resource heavy and second indexation will be put on queue, which makes it less efficient.
Updated: 11 December 2019