« Changelog History


simplecrawler v1.1.6 Release Notes

Release Date: 2017-10-06 // over 6 years ago

  • 🆕 New features

    • Sitemap directives in /robots.txt are now added to the queue if Crawler#respectRobotsTxt is truthy.

    🐛 Bug fixes

    • #398 - fix issue where multiple cookies weren't properly serialized for outbound requests
    • 📜 #400 - fix issue where <meta name="robots"> tags weren't properly parsed