Thought of slaying your website’s user experience? How about also ranking at the same time? Want to know how? Trust me; this is exactly where you are supposed to be then!
The goal of this article is to equip you with all the necessary guides and tools to explore, figure out, test, and analyze distinct ways of finding and fixing broken or dead links on your website.
This will improve your site’s crawlability and user experience.
Your website’s broken links affect the search engine crawler like nothing else! Imagine you have a secret weapon that always keeps you ahead of the pack. We’ll call it search engine crawl. And the way it works is that it sends crawlers/spiders/ almighty creatures (refer to them however you want) to search for the most recent and updated content, including videos, images, pdfs, and more.
Now for you to really stand out and acquire visibility on Google Search, you want to be noticed by the spider. You want attention! Crave to be crawled frequently. That’s the crawl budget, which is directly proportional to your SEO performance.
You see where the dead links can take you? Now, let’s head over to find and fix those broken links together.
What are Broken Links?
Broken links can be described as those links on your website that lead to nonexistent sources (causing 404 errors) or, in other words, are invalid. The term refers to both internal links, leading to another page within your site, and the external ones when the link is supposed to lead to another domain.
Before proceeding with the link-checker tools and identifying the type of links they reach out to, let’s make some http status codes clear first.
1. Status Code: 301 moved permanently redirect
“Where is it taking me?”
This happens either when the URL was entered incorrectly, so the link takes you to another domain against your will and intended reference, or when the linked-to site intentionally redirects you to a different domain, most likely with a very close or similar content.
2. Status Code: 302 temporary redirect
“Is this gonna be here for long?”
They differ from 301 in a way that the user is being redirected to another domain only for a short period while the previous domain is let’s say under construction. However, content-wise, they are a much better alternative to 404s.
3. Status Code: 404 not found
“Did they remove the link from their site too?”
“Oh, man. They renamed the source.”
Let’s get this straight, 404 redirects are the worst. They deindex a page from the search results when searched several times. Note that these are the broken links, while 301 and 302 redirects are solutions to those. 404s occur predominantly when the link you are referring to was deleted or renamed by the connected-to domain.
It becomes explicit how the linked-to domain can easily cause broken or dead links within yours, which only points at your website’s vulnerabilities. I know – not what we want. But now, when we are aware of all these external factors we cannot be fully in control of, one thing is for sure: there is always a need to check and validate your links not to hinder the search engine crawl, and that’s putting the user experience aside.
How do we do that? I was waiting for this!
Finding Broken Links: Tools and Methodologies
There are tons of link checker tools out there that report on the dead or broken links within a domain. But the good news is we’ve tried and tested them and are here to guide you throughout the process with the ones that proved to be most effective.
Ahrefs’ Site Explorer is a good fit for smaller websites, in which case it captures most of the links with increased accuracy. The way it works is you copy your domain into the Site Explorer and look for broken links in the outgoing links section. But be informed that this one finds only external broken links. For scalable research, you will have to set up a live crawl, which we will elaborate on in the coming sections.
Google Search Console is another trusted and well-known link-checker tool. All it takes is logging in to the console and pasting the domain you want to analyze in. You’ll be required to pass a DNS verification first, and once you do, there are various things you can take advantage of. The one we are interested in at this point is letting Google crawl your website and indicate invalid or broken links.
Again Google Search Console gives you whole freedom to play with the settings and filters while offering an in-depth search traffic analysis.
Monitoring: Make it All Systematic
An advanced method to help you succeed at finding broken links also involves implementing time-saving monitoring measures while providing increased productivity. One thing absolutely worth trying is setting up a web crawler at the top of the Google Search Console and Ahrefs Site Explorer.
What is a Web Crawl? Why do I need it?
Web crawler, also referred to as spider, robot, or spiderbot is an automated web bot that periodically browses and scans the World Wide Web. Within the context of SEO, crawlers are used to provide up-to-date information by following link paths and navigating and indexing websites. Since crawlers analyze all the possible links within a website, they are unbeatable at identifying broken links.
The palette of these crawlers is as diverse as SEMrush and Ahrefs Site Audits, Screaming Frog SEO Spider, DeepCrawl, Apifier, DYNO Mapper, OnCrawl, SEO Chat Web Crawler, and many more. But we will focus on three today.
Ahrefs’ Site Audit works for all sorts of sites since it performs a live crawl, making sure no broken link is missed. For this purpose, you have to:
- Start a new project.
- Type your domain.
- On the next page schedule the crawl based on the date and time that works for you.
- Make sure you adjust the crawl frequency with desired URL sources and crawl settings.
- You can add keywords and competitors to track if necessary.
- Finish filtering and start the crawl.
- Once it’s done, look for the dead links in internal pages, and external pages.
- Voila! Your next step is most likely exporting the data as an Excel Sheet and steadily heading over to fixing 404s.
SEMrush’s Site Audit is another validation tool, pretty similar to Ahrefs’ Site Audit. It works the same way, and you can also adjust the crawl settings to meet your digital needs. It checks a website’s so-to-say health, identifying pages with issues, duplicate content, broken links, missing meta descriptions, etc.
Follow the steps below, and your success is guaranteed.
- First, make sure your DNS is configured correctly.
- Start with the domain and limit of pages. The rest is optional.
3. Adjust crawler settings.
4. Adjust allow/disallow URLs.
5. Remove URL parameters if needed.
6. Bypass website restrictions.
7. Finally, schedule the crawl upon required frequency.
Once the crawl is over, find the broken links here:
Overall, both SEMrush and Ahrefs are decent all-in-one tools that provide not only thorough website audits, but also offer keyword research, domain and traffic analysis, crawlability, and content analysis, and many more options. If you are aiming to optimize your SEO and start ranking high, leaving your competitors behind, give it a try with both.
Screaming Frog’s SEO Spider has pretty much the same function as Ahrefs and SEMrush site audits for broken links. Here you also have filtering options and can review 404 broken links once the website crawl is over. Just click on the Response Codes tab and filter it to “Client Error 4xx.”
Although Screaming Frog is relatively underrated, here you can do the Bulk Export as an Excel Sheet, and deal with the broken links all at once, which will save you a handful of time.
Still having difficulties finding broken links? We recommend watching tutorials or following the guidelines published at the tool’s blog. If you are left with questions, feel free to call on us in the comments section.
Fixing Broken Links
Once the links are identified, there are a couple of ways you can go to repair them.
- You may want to replace the dead link with a new location in the code if that works for you.
- Request a fix from the linking website (if that’s their fault).
- Remove the link at all and make content adjustments if need be.
- You may happen to be reached out by a link builder and offered a relevant article to refer to. External broken links are perfect for a link building strategy, and there is a slight chance that you will be offered another reference right off the bat. Don’t rely solely on this though.
Learn more about broken links, as a part of a link building strategy at [the link to the article, see comment #5].
But honestly, the process isn’t as fun as it might seem since you never know what to expect. It can get pretty laborious and time-consuming, but can also take minutes, which totally depends on the number of broken links you get. Especially if you want to replace the links, you will have to find the right one first, and if there is a great number of dead links, it often gets challenging to remember where the right one would come from.
And let me tell you are the lucky one if you are reading this. When things get that complicated, Wayback Machine, Page Freezer, and Stillio are your saviors, as they give the archived copy of the link.
The Wayback Machine, in particular, is golden: gives very specific information as to how the domain used to look like on a particular day, at a particular time, allowing you to choose from a calendar and unraveling the entire domain history.
Keep in mind that finding and fixing broken links should become systematic for you to get your site optimization and user experience off the ground. It is not a single-time practice; otherwise, your efforts are gone down the drain.
Now you realize, this may take a while, and therefore, bigger companies tend to hire VAs to fix 404s for them. But if you think it’s worth your time and resources doing it manually, kudos to you! Have fun and tell us which way you decided to go and how it worked.
What systems, tools, and processes do you have in place to catch and fix technical site issues? Anything we haven’t covered that you want to hear more about? How about other technical issues and outreach strategies? Let us know in the comments below, and we will get back to you shortly!