How Scanfully detects broken links

Scanfully finds broken links by crawling your site from the outside, the way a visitor or a search engine sees it. It loads your pages, collects the links on them, requests each link and records the HTTP status code the server answers with. A link that answers with a 4xx or 5xx status code is reported as broken.

Scanfully does not change your content, add scripts to your pages or store anything in your WordPress database. The results live in Scanfully, on the Content screen.

How Scanfully finds your pages

A crawl starts from:

your home page,
every page in your sitemap, if one is set,
every page on your extra pages list.

From there, Scanfully follows the links that stay on your site’s domain, up to the maximum depth you set (1, 2 or 3 clicks). Links to other domains, subdomains included, are checked but not followed. Pages that match one of your exclusion rules are not checked. You set all of these in the crawl settings.

Scanfully only checks the links on pages that load. A page that itself answers with an error, or that redirects to another domain, is skipped. If it is linked from your other pages, it shows up as a broken link there.

When the site has custom request headers, Scanfully sends them with its requests to your own site, and to nothing else.

Which links are checked

On every page it crawls, Scanfully checks every link (<a href>), internal and external. Internal links point at your own domain; external links point at other sites. A link that appears on many pages, such as one in your menu or footer, is requested once per crawl and reported for every page it is on.

Some links are always left out, because they are not web pages or because checking them only produces false alarms:

mailto: and tel: links, javascript: links, and links to an anchor on the same page (such as #top),
links to Facebook, Instagram, LinkedIn, Bluesky and Google Maps,
Cloudflare’s /cdn-cgi/ addresses on your own site.

You can leave out more with Links to ignore in the crawl settings. See How wildcard ignore patterns work.

When a link counts as broken

Scanfully first asks the server for just the headers of the target (a HEAD request). When the server answers 403, 405 or 501, it tries again with a normal GET request, as some servers refuse header-only requests. Redirects are followed, and the status of the final address counts.

The link is reported as broken when that status is a 4xx or 5xx code, such as 404 Not Found or 500 Internal Server Error. A 429 Too Many Requests is reported as a warning: the site turned the crawler away, which says little about the link itself. A link that gives no answer at all is reported as No response: the request timed out, the domain no longer exists, the server refused the connection, or its TLS certificate could not be used. Before it reports this, Scanfully tries once more with a GET request and more time, so a slow server or a short network problem does not count.

External sites can behave differently for a crawler than for a browser. A link that works in your browser may still show as broken, often with a 403. See A link works in my browser.

What happens between crawls

Each crawl compares what it finds with the last report. New broken links are added, links that still fail get a new last checked date, and links that work again, or are no longer on the page, leave the report. When each reference was first found and last checked is explained in Where a broken target was found and when.

Broken media (images, scripts, video and other files) are found by a second crawl that works the same way. See What broken media means in Scanfully.