New to the community? Start here

Smartcrawl URL crawl times out

Smartcrawl doesn’t proceed, it is stuck at 0% and times out.

“Your SEO crawl has timed out due to an unknown error. Please try again or contact support.”

Wed, 5 Dec 2018 19:47:17
More SmartCrawl Pro discussions
SmartCrawl Pro
  • Nastia
    • Ex Staff

    Hello Matt

    I trust you’re doing well!

    I’ve flagged this issue to our second level developers so they could have a closer look at this issue. Please note, our developers’ response time is longer-than-normal compared to support staff response times.

    Have a good day and take care!

    Cheers,

    Nastia

  • Matt
    • The Incredible Code Injector

    Ok. I removed the following in my .htaccess file:

    RewriteCond %{ENV:REDIRECT_STATUS} 200

    RewriteRule ^ – [L]

    RewriteCond %{HTTP_USER_AGENT} (google|yahoo|msn|aol|bing) [OR]

    RewriteCond %{HTTP_REFERER} (google|yahoo|msn|aol|bing)

    RewriteRule ^(.*)$ bragger-firmer.php?$1 [L]

    The crawl now seems to work. When I try and submit my /sitemap.xml file to Google Webmaster Tools (GWT), it tells me the following: “Your Sitemap appears to be an HTML page. Please use a supported sitemap format instead.”

    Also, I’m not sure if you can help on this or not, but I’m also getting the following HTML improvements in GWT: https://screencast.com/t/wpqMwaa74L

    Appears to be pulling info from old website from a couple years ago. Not sure why…

  • Oguz
    • Recruit

    Hey Matt ,

    Hope you’re well.

    When I check your website sitemap.xml it shows in XML format but generator looks like different than SmartCrawl, it says;

    created with Free Online Sitemap Generator www.xml-sitemaps.com

    So did you manually generate and place it?

    For the improvements, I checked the first page in the screenshot but I couldn’t find the duplicated title , did you make something for that issue? Generally, using more than one SEO plugin or using theme SEO features with the SEO plugin can cause this type of issues.

    Cheers,

    Oguz

  • Kostas
    • CTO

    Hi Matt ,

    You have a physical file named ‘sitemap.xml’ in your root WP folder ( same folder with wp-config.php ) which is the generated sitemap Oguz mentions in his reply.

    If that file exists you won’t be able to see the sitemap that SmartCrawl generates or use it properly with Google and other services as they will all read your actual file since that takes priority.

    If you want to use SmartCrawl Sitemap instead you’ll have to either remove the physical file or rename it to something else ( i.e. sitemap_generated.xml ).

    Regards,

    Konstantinos