Tip: Ahrefs Broken Backlinks Report Isn't Accurate - Here's What to Do

[Justin Mosebach]

The "Broken backlinks" report in Ahrefs returns some false positives and misses some broken backlinks altogether.

To get around this, I use a combo of Ahrefs and Screaming Frog (SF) to find broken backlinks.

tl;dr​

1. Export all backlinks from Ahrefs.
2. Run the destination URLs through Screaming Frog using "List Mode". Check for:
   1. No Response
   2. 404s
   3. 5XX errors
   4. Redirect loops
   5. Redirect chains longer than 5 hops

Detailed Steps​

1. Go to Ahrefs, plop the URL into the search bar of Site Explorer and click on "Backlinks" in the left-hand sidebar.
2. Important: make sure that "All backlinks" is selected from this drop-down:

Next:

1. Click "Export".
2. Make sure to change the "Number of rows" to "All". Click Export and open the spreadsheet.

In the spreadsheet:

1. Copy the URLs from the "Target URL" column. Pro Tip: to make the crawling process faster, de-dupe the list. Assuming your site's redirects are properly set up (so that the http/https and www/non-www variations all send the user to the canonical version), do a find/replace in the spreadsheet so that all URLs start with the same "http" or "https" and with or without "www." Then remove duplicates (here's how in Excel).
2. After duplicate URLs have been removed, take the resulting list of URLs and run them through Screaming Frog in "List" mode.
   1. Copy the list of URLs.
   2. In Screaming Frog (SF), select Mode > List from the main menu.
   3. In SF, click Upload > Paste.
   4. Click OK

After the crawl is done, in Screaming Frog:

1. Click "Response Codes" tab.
2. In the "Overview" tab on the right (see screenshot above), click on the following:
   1. No Response
      1. If there's a short list of URLs, check them manually (make sure it's not a false positive) because sometimes the website doesn't handle SF well.
      2. If there's a longer list, click "Export" and re-crawl them in List Mode at the end of this process.
   2. Server Error (5xx)
      1. If there's a short list of URLs, check them manually (make sure it's not a false positive) because sometimes the website doesn't handle SF well.
      2. If there's a longer list, click "Export" and re-crawl them in List Mode at the end of this process.
   3. Client Error (4xx)
      1. Export this list.
      2. Make sure to remove any 403 errors (or manually check them), because those are typically websites just blocking SF from crawling the site.
3. Combine the lists of actual broken links into one spreadsheet.
4. For each URL, figure out the destination URL for where the broken link should redirect to. Ideally, this won't be the homepage. Why? Google considers a redirect to the homepage to essentially be a 404 (soft 404), which won't help with passing link juice.
5. Upload redirects to the website.
6. Back in Screaming Frog, click Reports > Redirects > All Redirects.
   1. Open the exported spreadsheet.
   2. Check for redirect loops("Loop" column)
      1. If there are any, those will need to be addressed by looking into existing redirects (ex. in the site's Redirection plugin). Fixing these loops will fix the broken link.
   3. Check for too many redirect hops.
      1. Look at the "Number of Redirects" column. If there are more than 5, fix the redirects so that there's not more than 5 hops in the redirect chain. Why? Google says, "While Googlebot and browsers can follow a 'chain' of multiple redirects (for example, Page 1 > Page 2 > Page 3), we advise redirecting to the final destination. If this is not possible, keep the number of redirects in the chain low, ideally no more than 3 and fewer than 5. Chaining redirects adds latency for users, and not all browsers support long redirect chains."
7. Clear the website's cache (ex. hosting cache, CDN cache, and/or whatever plugin you're using).
8. Recrawl the list of URLs in Screaming Frog to see if they properly resolve. If they don't, troubleshoot from there.

 

[Conor Treacy]

Very nice!!! That List option speeds things up greatly!

Nice set of HOW-TO!!