For a long time, Xenu’s link sleuth has been a key secret weapon in the SEO’s arsenal. But there’s a new player on the block, Screaming Frog’s SEO Spider. Here’s a look at how the tools compare, and an overview of SEO Spider’s reports and features.
For a long time, Xenu’s Link Sleuth has been a key secret weapon in the SEO’s arsenal. Originally designed for finding bad links in a site, the simple crawler, especially combined with a bit of Excel wizardry, turned out to be useful for all sorts of site audit problems.
However, Xenu’s always had its limitations. That’s fine, it was designed as a simple tool after all. But there’s a new contender on the block, and that contender is Screaming Frog’s SEO Spider. Xenu on crack.
At its simplest level, SEO Spider is similar to Xenu. It’s a desktop application that will crawl any site you give to it. However, it’s a lot more configurable to Xenu, and returns a lot more information.
Critically, unlike Xenu’s static reports, SEO Spider’s report format is effectively a spreadsheet itself, allowing you to filter and reorder results at will. The standard export options are all there, of course.
In a small touch of genius, although SEO Spider has a lot of options, all the defaults are set up to do what you most often want to do: simply crawl a site. Loading up the package, giving it a URL and hitting “Start” will do pretty much the same as what Xenu does out of the box, but returning a lot more information. It also seems to crawl a lot faster than Xenu, although this can be bottle-necked by the server itself.
SEO Spider returns information in 12 separate tabs:
There is some duplication between the different reports (“Internal pages” especially is really just a summary of all the rest). However, importantly, each report has its own set of filters, carefully designed for the kind of work an SEO would want to do with that data.
For each asset on your site, you can drill down into an even deeper level of data, including the number of external links on the page, and the number of internal links pointing to it.
These filters are one of the key differences between Xenu’s Link Sleuth and SEO Spider. The latter is specifically designed for SEOs, and allows you to work with the data directly and dynamically.
For example, some use cases using filters are:
Being a desktop applicable, it’s fast and responsive. It feels almost magical having all this data at one’s fingertips.
You can also export filtered reports to CSV, and commands for checking things such as Google’s cache or index info, backlinks from all the usual providers or the Wayback Machine are all just a click away for every indexed URL.
The only real limitation is that SEO Spider comes in both free and premium versions.
The free version is limited to crawling 500 URLs on a given domain, and disables many of the advanced options. In this form, it does everything that Xenu does, just with the URL limit, so if you only ever work with smaller sites, this will suit you fine.
If you want to get into more advanced scanning (the custom searches are only available in the premium version, for example) or work with larger sites, the £100 per year price tag is pretty trivial.