New to the community? Start here

SEO: We encountered too many server errors, and won't process any further.

Hi, on the SEO tab i always have the "We encountered too many server errors, and won't process any further." error. But it didn't got away since i have WPMUDEV. Can you guys help me?

King Regards

Cedric

Mon, 9 Apr 2018 10:33:18
  • Adam
    • Support Gorilla

    Hello The Media Company

    I hope you're well today and thank you for your question!

    When crawler (SmartCrawl's SEO crawler) scans the site, it goes through the URLs and may encounter some errors, for example: a broken (not working/existing) link, a connection can be rejected for some reasons by the server, connection might time out etc. If there's too many of such errors, crawler "gives up" at some point because it doesn't quite make sense to perform further scan.

    In that case, we need to find out why this is happening. One of the most common reasons is aforementioned timing out but not only one. I'd like to start with checking your site so could you please enable support access to it so I could take a closer look?

    To do this, please go to the "WPMU DEV -> Support" page in your site's back-end, click on "Grant support access" button there and let me know here when it's done.

    [attachments are only viewable by logged-in members]

    Best regards,

    Adam

  • Adam
    • Support Gorilla

    Hello The Media Company

    Thanks for enabling support access.

    I checked the site and I can see the issue. What’s bothering me here is that it looks like the crawl doesn’t even start. When I try to run a new crawl, it immediately redirects back to the “Sitemap” tab (SmartCrawl -> Sitemap -> Sitemap), the crawl is not running in background but also – there’s no trace of it in the logs on our end.

    That is alarming because the plugin in fact only triggers the crawler on on our end, that then crawls the site from our server. However, in this case it doesn’t look like the “start crawl command” even reached our API.

    I don’t see any firewall/additional security plugins on site that could be causing this and there’s no errors in a browser console. The site can be access via cURL from outside (so that’s fine) so there must be something that’s interfering with this communication. Just in case, I double-checked if the IP of the site is not black-listed on our end, but it doesn’t seem so.

    In that case, I’d like to start further investigation with some additional simple step so please:

    1. enable WP debugging by adding following lines to the “wp-config.php” file of your site (via FTP or cPanel “File Manager” usually):

    define('WP_DEBUG', true);
    
define('WP_DEBUG_LOG', true);
    
define('WP_DEBUG_DISPLAY', false);

    2. If there’s any kind of CDN implemented for the site and it can be switched off or switched to “dev mode” – set it so, please

    3. Check if there is any kind of caching tool on a server and if can be cleared, if so, do it too please

    4. Temporarily disable all the plugins except WPMU DEV Dashboard and SmartCrawl

    5. Go to the “SmartCrawl -> Sitemap” page and click on “New Crawl”

    See if that immediately takes you back to “Sitemap” tab and if you switch to “URL Crawl” tab if there are still those errors or you can see the progress bar running.

    If there’s no progress bar running or there is but it terminates at some point and doesn’t complete the scan (returning similar errors again), please:

    1. look into the “/wp-content/” folder of your site’s WP install for the file “debug.log”; if it’s there, download it, rename to “debug.txt” and attach to your response here

    2. See if you have access to any additional server logs – error logs, access logs and if so, share them with me as well please (you might need to put them on some file storage cloud like Google Drive or Dropbox and share a download link with me as they may not come through as file attachments here); if you don’t have access to such logs, please get in touch with your host and ask them if they could provide such logs from around the time when you were conducting that test.

    Hopefully these logs will give me some more information on what is happening on the site.

    Best regards,

    Adam