New to the community? Start here

URL Crawler fails due to timeout

I updated my sitemap, but now I can’t run a new URL Crawler, it says:

‘Your SEO crawl has timed out due to an unknown error. Please try again or contact support.’

Wed, 14 Mar 2018 13:38:58
More SmartCrawl Pro discussions
SmartCrawl Pro
  • Adam
    • Support Gorilla

    Hello Wayne,

    I hope you’re well today and thank you for your question!

    I checked your site and I can see the issue there. I have, however, noticed two additional issues that might be related:

    1. There’s an error in debug.log that shows up today, saying:

    Unfortunately, it doesn’t point exactly to what’s causing it but I think it might be related to the second one below.

    2. There’s a message in site’s back-end “Warning! WP Redis object cache cannot connect to Redis server.”.

    I see that the site’s hosted with Cloudways. From all the opinions that I came through, they are quite fine and trustworthy so that’s a good thing. However, these two errors together bother me a bit and I’m wondering whether either some changes were made to the site or to the server configuration before SmartCrawl stopped working.

    Are you aware of any of such changes? Could you also get in touch with your host and ask them if they are able to help or advice on these two errors?

    I think their opinion on that might be helpful here and if its something they could fix, we’d at least know after that whether this was causing the issue or if we need to “drill down” more.

    Looking forward to you response,

    Adam

  • Wayne
    • Design Lord, Child of Thor

    Hi,

    Thanks for looking at this issue as quick as you did. Unfortunately I’ve let this slide when I shouldn’t (it’s a big issue).

    Anyway regarding the Redis issues they’re to do with my quick removal of the plugin due to a bug in woocommerce which was hiding products from view. There was a file left throwing the error. The file has since been removed and the issue remains.

    As a note even before installing the redis plugin I’ve had an issue, it’s just the sitemap was accurate so I didn’t think about it. Recent work to change the main domain and fix the SEO issues brought it to my attention.

    As for the first error in the debug.log, while I can’t see it in your response my memory of the issue was that the problem was related to the redis plugin.

    All the issues regarding redis have been resolved (stale database records, etc.) and the site should be in a good state to generate the sitemap.xml

    That said I just ran another test and it is stuck at 78%.

    As it is the site should be relatively basic (no caching) but has a cdn for images. Please take another look and I’ll actively help fix the issue.

  • Predrag Dubajic
    • Support

    Hi Wayne,

    Thanks for the additional information about the issue.

    I see that you’re still on SmartCrawl version 2.1.1 and we recently released 2.2 version with quite a few fixes and improvements.

    We did have some reports on newest version causing higher memory usage on certain installations. It’s something already being worked on and we’re going to see a new release during this week, if not even today.

    So could you wait for the 2.2.1 release, run the new crawl with that version and see if it works for you after that?

    Best regards,

    Predrag

  • Wayne
    • Design Lord, Child of Thor

    Well updated to 2.2.1 and the extra issues I had with 2.2 have gone and it’s working as before. It currently is at 77% and seems to be stuck (at 77% for an hour).

    Normally it would fail with a timeout but this time it still seems to be trying.

  • Predrag Dubajic
    • Support

    Hi Wayne,

    I wanted to enable debug log on your site in order to see if we can get some further info from there but your host seems to be using some custom wp-config setup and in order to enable it the “classic” way I would need to remove their debug code from there, so I stepped back.

    Can you ask them to enable debug log for you and then start a new scan and see if there’s anything added to debug.log in your wp-content folder after it fails?

    Best regards,

    Predrag

  • Predrag Dubajic
    • Support

    Hi Wayne,

    Hmm, strange, I was expecting that when the crawl fails it will at least give us some info about what’s going on.

    I’m afraid that I couldn’t find any further info about it on your site so I have forwarded your ticket to our developers so they can investigate it further and shed some light on this.

    Best regards,

    Predrag

  • Adam
    • Support Gorilla

    Hello Wayne

    I hope you’re well today!

    I’m afraid that we don’t have any new information for you yet. I’m sorry for keeping you waiting but the issue seems a bit “unusual” and our developers are still looking into it. We’ll update you as soon as we get some feedback from them.

    Kind regards,

    Adam

  • Ari
    • Recruit

    Hey Wayne

    Hope you are fine today!

    I just checked the site, but I didn’t find any issue now, is this fixed? or you changed the sitemap?

    I logged in to your site & run the SEO Checkup task, and the task finished like a charm without any error. Did I miss something?

    Please check it now, If you still need my assistance please give me the replication steps.

    Regards,

    Ari