Move Over Xenu: Make Way for Screaming Frog’s SEO Spider [Review]

For a long time, Xenu’s Link Sleuth has been a key secret weapon in the SEO’s arsenal. Originally designed for finding bad links in a site, the simple crawler, especially combined with a bit of Excel wizardry, turned out to be useful for all sorts of site audit problems.

However, Xenu’s always had its limitations. That’s fine, it was designed as a simple tool after all. But there’s a new contender on the block, and that contender is Screaming Frog’s SEO Spider. Xenu on crack.

Overview

At its simplest level, SEO Spider is similar to Xenu. It’s a desktop application that will crawl any site you give to it. However, it’s a lot more configurable to Xenu, and returns a lot more information.

Critically, unlike Xenu’s static reports, SEO Spider’s report format is effectively a spreadsheet itself, allowing you to filter and reorder results at will. The standard export options are all there, of course.

In a small touch of genius, although SEO Spider has a lot of options, all the defaults are set up to do what you most often want to do: simply crawl a site. Loading up the package, giving it a URL and hitting “Start” will do pretty much the same as what Xenu does out of the box, but returning a lot more information. It also seems to crawl a lot faster than Xenu, although this can be bottle-necked by the server itself.

Data Returned

SEO Spider returns information in 12 separate tabs:

• Internal pages: an overview of all your pages and other resources (images, etc)
• External links: any external links on the site
• Response codes: the response code and redirected URL (if appropriate) for every resource
• URI: details on your site’s URLs
• Page titles: detail on all the page titles on the site
• Meta description: detail on all the meta descriptions on the site
• Meta keywords: detail on all the meta keyword tags on the site (although they’re not used any more)
• H1: detail on your site’s H1 tags
• H2: detail on your site’s H2 tags
• Images: information on all the images found
• Directives: information on your site’s meta tag directives, such as robots and refresh
• Custom: your own custom source code searches

There is some duplication between the different reports (“Internal pages” especially is really just a summary of all the rest). However, importantly, each report has its own set of filters, carefully designed for the kind of work an SEO would want to do with that data.

For each asset on your site, you can drill down into an even deeper level of data, including the number of external links on the page, and the number of internal links pointing to it.

Dynamic Results

These filters are one of the key differences between Xenu’s Link Sleuth and SEO Spider. The latter is specifically designed for SEOs, and allows you to work with the data directly and dynamically.

For example, some use cases using filters are:

• The “Titles” report allows you to find duplicate tags, pages where the title and H1 tag are the same and overly long or missing titles.
• The “URI” report allows you to find duplicate pages (they have to be exact duplicates, but this is still very useful), dynamic URLs, URLs containing non-ASCII characters, etc.
• The “Directives” report allows you to find all the pages set to noindex, etc.
• The “Custom” report could allow you to find all pages missing Google Analytics code (hands up everyone who has had a developer break the code on some, but not all, of a client’s site).
• The “Images” report lets you find images missing alt text or with overly long alt text.
• The “External links” report is great for finding broken, malformed or redirected external links, helping you to be a good net citizen.

Being a desktop applicable, it’s fast and responsive. It feels almost magical having all this data at one’s fingertips.

You can also export filtered reports to CSV, and commands for checking things such as Google’s cache or index info, backlinks from all the usual providers or the Wayback Machine are all just a click away for every indexed URL.

Limitations

The only real limitation is that SEO Spider comes in both free and premium versions.

The free version is limited to crawling 500 URLs on a given domain, and disables many of the advanced options. In this form, it does everything that Xenu does, just with the URL limit, so if you only ever work with smaller sites, this will suit you fine.

If you want to get into more advanced scanning (the custom searches are only available in the premium version, for example) or work with larger sites, the £100 per year price tag is pretty trivial.