John Mueller from Google gave a very detailed and honest explanation of why Google does not crawl and index every URL or link on the web. He explained that crawling is not objective, it is expensive, it can be inefficient, the web changes a lot, there is spam and junk, and all of this has to be taken into account.
John wrote this detailed answer on Reddit, responding to 'Why don't SEO tools show all backlinks?' He answered from the perspective of Google Search:
There is no objective way to crawl the web properly.
Theoretically, it is impossible to crawl the whole web because the number of actual URLs is practically infinite. Since no one can afford to maintain an infinite number of URLs in a database, all web robots make guesses, simplifications, and assumptions about what is realistically worth crawling.
And even then, for practical purposes, you cannot crawl everything all the time; the internet does not have enough connectivity and bandwidth for that, and it costs a lot of money if you want to access many pages regularly (for the robot and for the site owner).
Then some pages change quickly, others have not changed in 10 years - so robots try to save effort by focusing more on the pages they expect to change, rather than those they expect not to change.
Then we get to the part where crawlers try to figure out which pages are actually useful. The web is full of junk that no one cares about, pages that have been marked as spam and are completely useless. These pages may still be updated regularly, they may have reasonable URLs, but they are simply meant for the dump, and any search engine that cares about its users will ignore them. Sometimes these are not obviously and clearly junk. Increasingly, sites are technically sound, but they just do not meet the quality threshold to deserve more frequent crawling.
Therefore, all robots (including SEO tools) work with a very simplified set of URLs. They have to figure out how often to crawl, which URLs to crawl more often, and which parts of the web to ignore. There are no fixed rules for any of this, so every tool has to make its own decisions along the path to indexing. That is why search engines have different indexed content, because SEO tools list different links, and all the metrics built on top of them are completely different.
In case you need help with your site indexing. Click here to get in touch with us.
