Screaming Frog announces SEO Spider 10.0 update

The popular crawling tool has been updated, and now boasts a host of new features including scheduled crawls, XML Sitemap crawl integration and Internal Link Score.

The Screaming Frog SEO Spider - one of the most popular technical SEO tools around - has updated to version 10.0, and introduced a range of new features.

See a quick run-down of the new features below, and read our Screaming Frog SEO Spider review for more information.

Scheduled crawls

One of the most useful new features is the ability to schedule crawls to run automatically - and you can choose to run them at chosen intervals or as a one-off crawl.

When scheduling a crawl you are able to choose between the two different modes - spider or list - and select the different APIs you also want to pull data from (Google Analytics, Search Console, Ahrefs, Moz, Majestic etc.)

You can also now automatically save the crawl file and export it whole or choose any of the tabs, filters or reports to export to you chosen location.

For anyone who runs regular crawls on Screaming Frog, this new feature promises to be a massive time-saver.

Indexability status

Every URL within a Screaming Frog crawl report is now classed as either 'indexable' or 'non-indexable'

While the concept of 'indexable' can be a loose one - search engines may index URLs that can't be crawled if there are links pointing to them - this new feature simplifies things a bit, showing users all pages that return a 200 status code and is permitted to be indexed as 'indexable'.

This should make it easier to spot problems at a glance, rather than having to run formulas in a spreadsheet and look at multiple columns when reviewing the 'indexability' of a long list of URLs.

XML Sitemap Crawl Integration

Screaming Frog has always allowed users to crawl XML Sitemaps within the list mode of the SEO Spider, but with this update you can now crawl and integrate your XML Sitemap as part of a site crawl.

Under 'Configuration > Spider' you can select to crawl XML Sitemaps and either tell Screaming Frog where the file is, or it will fetch the location from the robots.txt file.

There is also a new Sitemaps tab in the crawl report which allows you to filter through any issues with your XML Sitemap, such as non-indexable URLs, orphan pages etc.

To save you from having to switch to 'list mode; when you want to crawl the XML Sitemap, you can also now typed the location of the Sitemap in the top bar and the SEO Spider will crawl it.

Internal Link Score

Internal linking is an important, but often overlooked, aspect of SEO. Screaming Frog already had several features to help with internal linking such as crawl depth, number of internal links etc...

With the new update, Screaming Frog now gives each URL a 'Link Score' out of 100 which shows the relative value of a page based upon its internal links.

This gives you a quick and clear indication of where your internal linking structure could be improved, and it's worthing noting the metric takes into account things like canonical tags, redirects, nofollow tags and so on.

Crawl Visualisations

Screaming Frog has always been focused on providing solid, actionable crawl data that is presented in a text format and can be easily exported to Excel. Now, perhaps with an eye on competitors like Sitebulb, Screaming Frog has introduced a new crawl visualisation feature.

Explaining their thinking behind crawl visualisations, Screaming Frog founder and director Dan Sharp said:

"We wanted to build a better way of visually understanding a site, its architecture, internal link structure and issues. We wanted to make them scalable, and we didn’t want to have to hide data from users to make them work."

The new crawl graph visualisations are very useful when analysing internal linking, as they provide a clear view of how Screaming Frog - and by extension, search engines - crawl the site. Below is a view of how the crawl of the Screaming Frog website looks: