SmartCrawl – crawl timing out, URLs missing from sitemap

SmartCrawl wasn’t working on the site. Once that was solved, first crawl was reporting timeouts due to “too many server errors”, then it completed finally but is still reporting missing URLs.

Currently, it states that 128 URLs where detected and 55 are hidden (missing from sitemap). Those 55 URL’s listed in SmartCrawl are accessible (they can be opened in browser) but

– crawl times out again

– the “update sitemap” button in SmartCrawl metabox in dashboard doesn’t do anything

– if you try to add any of “missing urls” to sitemap manually (via the option in smartcrawl), it’s not added as well

  • Dimitris Kalliris
    • Support Team Lead

    Hello there Kevin ,

    hope you're doing good today! :slight_smile:

    I just tried to "add to sitemap" the reported missing URLs and these were added to the "Extra URLs" in SmartCrawl -> Sitemap -> Sitemap.

    After doing so, I've run a new sitemap crawl which completed successfully without any timeout errors and it now reports that there's no issue at all.

    [attachments are only viewable by logged-in members]

    Warm regards,

    Dimitris