One our favourite SEO tools, Screaming Frog, recently got a big new update (with subsequent .1 and .2 updates/bug fixes). Here’s the key details.
“We use Screaming Frog on a regular basis to help us crawl sites and provide key data and insights in to client’s website structure and performance. It’s an essential tool for anyone working in web analytics. A number of weeks ago, Screaming Frog announced an update, moving to version 7.0 – here’s some of the key points from the update.
- Blocked Resources. You can now get details of each pages blocked resources. This can be seen on the ‘Rendered Page’ tab or under Response Codes > Blocked Resource, and can be exported in bulk via the Bulk Export menu.
- Custom robots.txt. You now have the ability to download, edit and test a site’s robotos.txt. You can add multiple robots.txt at subdomain level, test directives in the SEO Spider and view URLs which are either blocked or allowed. This offers an alternative to using the regex exclude feature.
- hreflang Attributes. The SEO Spider can now extract, crawl, and report on hreflang attributes delivered by HTML link element and HTTP Header.
- rel=”next” and rel=”prev” Errors. The report now highlights errors and issues with rel=”next” and rel=”prev” attributes, which are used to indicate paginated content.
- Maintain List Order Export. URLs that you’ve uploaded in list mode can now be exported in the same order as the original upload, meaning you can now easily match it up vs the original data.
- Web Form Authentication (Crawl behind a login). A new authentication configuration allows users to log in to any web form within the SEO Spider Chromium browser and then crawl it, meaning nearly all password-protected areas, intranets and web forms with logins, can now be crawled.
There are several other smaller feature updates and bug fixes, as well as some subsequent (.1 and .2) updates containing bug fixes, that were also introduced as v7. The full list of features and changelists can be found here.”