Fixing Crawl Errors in Google Search Console

seo cover

A lot has changed in the last few years. Google has unleashed more data that promises to be extremely helpful for SEOs audit. We rely on the Search Console along with Page speed insights more than ever.

The “Links to Your Site” and “Search Analytics” sections are two of the top leaders that did not exist in the ancient Webmaster Tools.

While we may never be delighted with Google’s tools and irregularly call their bluffs, they release some valuable information. To their credit, Google has developed more support resources and help docs to aid Search Console users in locating and fixing errors.

Even though some of this is not as amusing as creating 10x content or watching which of your keywords has jumped in the rankings, this category of SEO is still necessary.

Looking at it through Portent’s epic visualization of how Internet marketing pieces fit together, fixing crawl errors in Search Console fits somewhat into the “infrastructure” piece:

If you can practice preventive maintenance and promote good habits, weekly spot checks on crawl errors will be adequate to keep them under control.

However, if you entirely ignore these errors, things can quickly go from bad to worse.

Table of Contents:

Crawl Errors Layout

One change that has grown over the last few years is the layout of the Crawl Errors view within the Search Console. Search Console is split into two main sections: URL Errors and Site Errors.

Classifying errors in this way is pretty important because there is a definite difference between errors at the page level and errors at the site level.

Site-level problems can be more catastrophic, with the potential to damage your website’s overall usability.

URL errors, on the other hand, are particular to individual pages and are therefore less critical.

The fastest way to access Crawl Errors is from the dashboard. The main dashboard gives you a quick preview of your website, showing you three of the most effective management tools: Search Analytics, Crawl Errors, and Sitemaps.

1. Site Errors

The Site Errors section shows you issues from your website as a whole. These are the errors that affect your site in its entirety, so do not skip these.

In the Crawl Errors dashboard, Google will give you the errors for the last 90 days.

That is the purpose — to get a “Nice!” from Google. As SEOs, we often do not get any validation from Google, so we enjoy this rare second of love.

How Often Should You Check For Site Errors?

In a perfect world, you would log in daily to make sure there are no difficulties here.

It might get tedious since most days everything is accurate, but wouldn’t you kick yourself if you missed some significant site failures?

At the absolute minimum, you should investigate at least every 90 days to look for past errors so you can be more attentive towards them in the future. Regular checkups are the best.

We will talk about automating and setting up alerts for this part later, but know that this section is crucial, and you should be 100% error-free in this section every day. There is no gray area here.

2. DNS Errors

What do they mean?

DNS errors are crucial. If you have extreme versions of these errors, then you definitely have to act now.

DNS errors are the first and most noticeable errors because if the Googlebot is having DNS problems, it means it cannot link to your domain via a DNS lookup issue or DNS timeout issue.

Your domain is likely hosted with a standard domain company, like GoDaddy or Namecheap or your website hosting company.

Most of the time, your domain is hosted separately from your site hosting company, but other times the same company handles both.

Are they important?

A DNS issue is critical, as it is the first step in reaching your website. You should take violent and swift action if you are running into DNS problems that prevent Google from correlating to your site in the first place.

3. Server Errors

What do they mean?

A server error means that your server is taking long to respond, and time is out. The Googlebot that wants to crawl your website can only wait a specific time to load your site before giving up. If it takes too long to respond, the Googlebot will just stop trying.

Server errors are more problematic than DNS errors. A DNS error means the Googlebot cannot even look up your URL because of DNS problems, while server errors mean that although the Googlebot can unite to your site, it cannot load the page because of server issues.

Server errors can happen if your site gets overloaded with too much traffic, which the server cannot handle. To avoid this problem, make sure your hosting provider can scale up to provide website traffic bursts.

 Believe me, everybody wants their website to go viral, but not everybody is ready!

Are they important?

Like DNS errors, a server error is fundamental. It is a real mistake and harms your website overall. You should take urgent action if you see server errors in Search Console for your website.

Making sure Googlebot can attach to the DNS is a crucial first step, but you will not get much further if your website does not show up.

If you are running into server failures, Googlebot will not be able to discover anything to crawl, and it will give up after a while.

4. URL Errors

URL mistakes differ from site mistakes because they only change specific pages on your website, not your site as a whole.

Google Search Console will give you the top URL errors per category — smartphone, desktop, and feature phone. This may not be enough data for large sites to provide all the mistakes, but this will catch all known obstacles for most areas.

Tip from LinkSignal: Going crazy with the number of errors? Mark all as fixed.

Many site owners have run into seeing a large number of URL mistakes and getting freaked out.

Crucial thing to remember:

a) Google ranks the most critical mistakes first, and

b) it may solve some of these mistakes.

If you have made some radical changes to your site to fix mistakes or think many URL mistakes are no longer happening, one tactic to use is marking all mistakes as a set and checking back upon them in a few days.

It will clear your issues out of the dashboard for a given moment, and Google will bring the mistakes back the next time it crawls your site.

If the mistakes still exist, you will know that these are always changing your site.

5. Soft 404

A soft 404 mistake is when a page shows as 200 when it should show as 404.

Is it important?

If the pages that have soft 404 errors are not crucial and don’t spend much money from your budget, then these are not an essential item to fix.

If you have essential pages on your site listed as soft 404s, you will want to fix those. Virtual goods section, or lead gen pages should not be listed as soft 404s if they are live pages.

Pay attention to pages critical to your site’s money-making ability.

If you have a considerable amount of soft 404 mistakes relative to the total number of pages on your site, you should take fast action.

You can be eating up your Googlebot crawl budget by allowing these soft 404 errors to exist.

6. 404

A 404 error simply means that the Googlebot tried to crawl a page that does not exist on your website. Googlebot finds 404 pages when other pages or websites link to that non-existent page.

Are they important?

This is one of the most superficial and trickiest issues of all errors. The massive quantity of 404s that many medium to large sites collect is enough to deter action.

If a page is gone and does not meet Google’s quality criteria above, let it be.

As unpleasant as it might be to see hundreds of mistakes in your Search Console, you just have to ignore them. Unless you get where your problems are coming from, they will continue showing up.

7. Access Denied

Access denied means Googlebot simply cannot crawl the webpage. Unlike a 404, Googlebot is blocked from crawling the page.

What do they mean?

Access denied errors usually block the Googlebot through these methods:

  • Your robots.txt file blocks the Googlebot from whole folders, individual URLs, or your entire website.
  • Your provider is blocking the Googlebot from your website, or the server requires users to authenticate by proxy.
  • You need users to log in to see a URL on your site. Therefore, the Googlebot is blocked.

Are they important?

Like soft 404s and 404 errors, if the pages being blocked are necessary for Google to crawl and index, you should take quick action.

If you do not want this page to be indexed and crawled, you can ignore the access denied errors.

8.  Not Followed

What do they mean?

Not to be mixed with a “no follow” link directive, a “not followed” mistake means that Google could not follow that particular URL.

These mistakes often come about from Google running into problems with JavaScript, Flash, or redirects.

Are they important?

If you are dealing with not following problems on a high-priority URL, yes, these are essential.

If your problems stem from old URLs that are no longer working or from parameters that are not listed and just an extra feature, the priority level on these is lower. I would recommend you examine them anyways.

9.   Server Errors & DNS Errors

Under URL mistakes, Google lists DNS errors and server errors in the same sections in the Site Errors report. Google’s direction is to manage these in the identical way you would take the server’s site DNS errors and mistakes level, so apply to those two sections above.

They would vary in the URL errors section if the mistakes were only concerning individual URLs and not the site as a whole.

If you have separated configurations for unique URLs, such as the different formats for specific URLs on your domain or minibites, they could show up here.

Make SEO Process More Interesting

I get it. Some of this technical SEO stuff can bore you to tears. But you can learn a lot of useful information about SEO too. Nobody wants to inspect useless URL errors personally, or conversely, have a panic attack seeing millions of errors on your website.

However, with practice and repetition, you will gain the mental muscle memory of knowing how to respond to the mistakes: necessary and safely ignored. 

Got something in your mind? Ping us in the comments section.

Leave a Comment