Found us from search engine?
We rank high, you can too.
SEOPressor helps you to optimize your on-page SEO for higher & improved search ranking.
By jiathong on August 23, 2018
In SEO we always look at ranks and SERP, but we also need to know the process that happens before that. That is crawling and indexing.
Google ranks web pages that are in their index. If your web page is not indexed, or not correctly indexed, your rankings will be affected.
The web has moved from plain HTML – as an SEO you can embrace that. Learn from JS devs & share SEO knowledge with them. JS’s not going away. – John Mueller, Senior Webmaster Trends Analyst
The thing you need to know is this.
Let’s get a closer look at the whole crawling and indexing process.
Who, or in this case, what is involved in the process?
This is the crawler, also called the spider. Whenever there’s a new web page or any new updates on a webpage, Googlebot will be the first point of contact from the search engine.
What it does is it crawls the web pages and follows all the links in a web page. That way, the bot discover more new links and more new web pages to crawl. Crawled web pages are then passed to Caffeine for indexation.
The is the indexer that was launched in back in 2010. Whatever’s crawled by Googlebot will be indexed by Caffeine and that index is where Google choose which web pages to rank.
Links discovered from rendering will also be sent back to Googlebot to queue for crawling which will result in a second indexation. This is a very important point to keep in mind because one important part of SEO is internal linking. Inter-linking your web pages in your website gives Google a strong signal for things like page rank, authority and also crawl frequency. Which all, at the end of the day affects page ranking.
Now making sure your site gets crawled and indexed is prerequisite of getting your site show up in the SERPs. One way to check your indexed paged is “site:yourdomain.com”, an advanced search operator.
Head to Google and type “site:yourdomain.com” into the search bar. Google will return results in its index for the site specified like this:
The number of results Google displays (see “About XX results” above) isn’t exact, but it does give you a solid idea of which pages are indexed on your site and how they are currently showing up in search results.
For more accurate results, you can sign up for a free Google Search Console account if you don’t currently have one.
Here we have a straightforward graphic from this year’s Google i/o which shows you the flow from crawling to indexing and rendering.
That is good for getting a general idea of the whole process, but why don’t we zoom a little closer?
Which means, imagine this, you served Google a meal, but because they don’t have the cutlery to eat it, they can only judge how good it is by looking, and the server won’t be back with the cutlery until they’re done taking orders from 3 more other tables, Google then post a review on Yelp saying that your food is crap.
Does that sound fair and beneficial? Hell no.
You can also go to CanIUse to check out what is and is not supported by Chrome 41.
These are all crucial tools that help you understand the whole crawling, rendering, and indexing process. With that, you’ll have a better idea of where and what went wrong.
Then you’ll be losing all opportunities (rank, traffic, leads) to your competitors. You’ll need to have what we call a Rank Intelligence tool to get things working on your side.
Rank Intelligence by BiQ SEO Suite that allows you to leverage data for site analysis and uncover opportunities that’ll allow expansion and control of your search landscape.
The best part of this feature is enabling you to discover rankings that you don’t even know you stand a chance for!
Imagine having over 100+ websites over the years and each of them are left aside without proper care to get it ranked once high. Knowing the right keyword and its position in the SERP will give you a chance to revive your websites for the better!
Better still, BiQ has a Rank Tracking that helps you big time when it comes to tracking and monitoring all your sites once you’re done with optimizing your sites. Talk about efficiency!
These vital information in Rank Intelligence (& Rank Tracking) will provide real-time exploration at your fingertips to unleash the power of your ranking website – exclusive to BiQ Cloud, with the speed of thought.
1. Googlebot crawls, Caffeine index and render.
2. For HTML web pages, Googlebot requests a page and downloads the HTML, contents are then indexed by Caffeine.
4. Rendering is resource heavy and second indexation will be put on queue, which makes it less efficient.
6. Make sure your efforts does not go to waste by making sure your website is also ranked for the right keywords using BiQ”s Rank Intelligence.
Updated: 10 December 2023
Generate 5,000 words in under 5 minutes
Publish 100+ blog posts monthly
Rank in the Top 10 with SEO-optimized content
Drive 500K organic traffic
Rank in the Top 10 now with Longform AI
Save 67% today (As low as $14.69/mo)