New to the community? Start here

Troubleshooting: High Load Issue

Where is a good place to start to troubleshoot high load spikes? We have a specific blog that seems to trigger loads of 2 to 6% or more when the page is re-loaded a couple of times… this still occurs will all normal plugins (not mu-plugin folder) disabled and occurs with different themes as well.

This blog isn’t huge, 70 or so posts, but was recently imported into MU from a single intall.. is there a chance that the import may have been corrupted or something similar? Grapsing for straws here….

WHM is showing the following processes seem to be involved:

Top Process %CPU 54.0 /usr/bin/php /home/tracedef/public_html/index.php

Top Process %CPU 53.0 /usr/bin/php /home/tracedef/public_html/index.php

Top Process %CPU 45.0 /usr/bin/php /home/tracedef/public_html/wp-cron.php

Wed, 21 Oct 2009 1:50:42
  • Tracy
    • The Incredible Code Injector

    Thanks, I’ve got that, but still need to resolve root issue….

    I’m noticing that even navigating around my dashboard will rock my CPU load to 5 or 6 percent… which isn’t normally humanly possible, not even when gzip backups are running… the highest natural load I’ve EVER seen is around 1 percent and gzips might hit 1.5 to 2…. I’ve only seen anything over 2.5 if something is wrong….

    Is it possible its time to set up multi-dbs based on these numbers?:

    Total: 331 Tables 24,908 Records

  • Tracy
    • The Incredible Code Injector

    Andrew, good to know… I haven’t heard of the slow query log, any tips on where to find it? I’m trying to learn ssh now so that I can get in and see more real time data on what processes are eating cpu… WHM only gives me average process stats…. have been avoiding ssh for a long time….

  • Tracy
    • The Incredible Code Injector

    Thanks! Turns out slow queries log was not enabled, but is now.

    Also noticing that accessing specific blog dashboards brings server to a crawl, even by simply accessing first page of the specific blog’s dashboard via blogs > backend from main blog dashboard….. other blogs don’t have the issue… for a second it seemed as though changing privacy options from only allowing admins to see the blog to public helped, but could have been coincidence.

    I also noticed the following in my error log from earlier today, first time this series of errors appears that I’ve ever seen and it pretty much occurs over and over. Not sure if time syncs up exactly with when the issue first started…but cpu usage seems to be consistently related to the following (last two hours average) :

    Top Process %CPU 39.0 /usr/bin/php /home/tracedef/public_html/wp-cron.php

    Top Process %CPU 36.0 /usr/bin/php /home/tracedef/public_html/wp-login.php

    Error Log (sorry for posting all of this, hoping something might stick out or be helpful at that specific time):

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT meta_value FROM wp_sitemeta WHERE meta_key = ‘active-member-theme’ AND site_id = 1 made by require, require_once, require_once, get_template_directory, get_template, apply_filters, call_user_func_array, bp_core_force_buddypress_theme, get_site_option

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT meta_value FROM wp_sitemeta WHERE meta_key = ‘active-member-theme’ AND site_id = 1 made by require, require_once, require_once, get_stylesheet_directory, get_stylesheet, apply_filters, call_user_func_array, bp_core_force_buddypress_stylesheet, get_site_option

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT meta_value FROM wp_sitemeta WHERE meta_key = ‘automessage_installed’ AND site_id = 1 made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, get_site_option

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query show tables like ‘wp_am_actions’ made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SHOW TABLES; made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, dbDelta

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query CREATE TABLE wp_am_actions (

    id bigint(20) NOT NULL auto_increment,

    level varchar(10) default NULL,

    action varchar(150) default NULL,

    title varchar(250) default NULL,

    description text,

    PRIMARY KEY (id),

    KEY level (level),

    KEY action (action)

    ) made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, dbDelta

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SHOW TABLES; made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, dbDelta

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query CREATE TABLE wp_am_schedule (

    id bigint(20) NOT NULL auto_increment,

    system_id bigint(20) NOT NULL default ‘0’,

    site_id bigint(20) NOT NULL default ‘0’,

    blog_id bigint(20) NOT NULL default ‘0’,

    action_id bigint(20) default NULL,

    subject varchar(250) default NULL,

    message text,

    period int(11) default ‘0’,

    timeperiod varchar(50) default ‘day’,

    pause tinyint(4) default ‘0’,

    PRIMARY KEY (id),

    KEY system_id (system_id),

    KEY site_id (site_id),

    KEY blog_id (blog_id),

    KEY action_id (action_id)

    ) made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, dbDelta

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SHOW TABLES; made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, dbDelta

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query CREATE TABLE wp_am_queue (

    id bigint(20) NOT NULL auto_increment,

    schedule_id bigint(20) default NULL,

    runon bigint(20) default NULL,

    sendtoemail varchar(150) default NULL,

    user_id bigint(20) NOT NULL default ‘0’,

    blog_id bigint(20) default ‘0’,

    site_id bigint(20) default ‘0’,

    PRIMARY KEY (id),

    KEY action_id (schedule_id),

    KEY runon (runon),

    KEY user_id (user_id),

    KEY blog_id (blog_id),

    KEY site_id (site_id)

    ) made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, dbDelta

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query INSERT INTO wp_am_actions (level,action,title) VALUES (‘site’,’wpmu_new_blog’,’Create new blog’:wink: made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query INSERT INTO wp_am_actions (level,action,title) VALUES (‘blog’,’wpmu_new_user’,’Create new user’:wink: made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT meta_value FROM wp_sitemeta WHERE meta_key = ‘automessage_installed’ AND site_id = 1 made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, update_site_option

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT meta_value FROM wp_sitemeta WHERE meta_key = ‘automessage_installed’ AND site_id = 1 made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, update_site_option, add_site_option

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query INSERT INTO wp_sitemeta (site_id,meta_key,meta_value) VALUES (‘1′,’automessage_installed’,’yes’:wink: made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, update_site_option, add_site_option

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT ID, post_name, post_parent FROM wp_160_posts WHERE post_type = ‘page’ made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->install, automessageadmin->flush_rewrite, WP_Rewrite->flush_rules, WP_Rewrite->wp_rewrite_rules, WP_Rewrite->rewrite_rules, WP_Rewrite->page_rewrite_rules, WP_Rewrite->page_uri_index

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT ID, post_name, post_parent FROM wp_160_posts WHERE post_type = ‘page’ made by require, require_once, require_once, do_action, call_user_func_array, automessageadmin->setup_listeners, automessageadmin->flush_rewrite, WP_Rewrite->flush_rules, WP_Rewrite->wp_rewrite_rules, WP_Rewrite->rewrite_rules, WP_Rewrite->page_rewrite_rules, WP_Rewrite->page_uri_index

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT meta_value FROM wp_sitemeta WHERE meta_key = ‘map_supporteronly’ AND site_id = 1 made by require, require_once, require_once, do_action, call_user_func_array, domain_map->setup_plugin, get_site_option

    [20-Oct-2009 19:30:21] WordPress database error MySQL server has gone away for query SELECT option_value FROM wp_160_options WHERE option_name = ‘rich_editing’ LIMIT 1 made by require, require_once, require_once, do_action, call_user_func_array, cforms_addbuttons, get_user_option, get_option

    [20-Oct-2009 19:30:22] Wor

  • Tracy
    • The Incredible Code Injector

    Thanks Barry, I will definitely check it out. I’m wondering if the mysql has gone away is more an after effect of server resources not being available due to other runaway services or processes….

    I think the privacy-settings plugin is a suspect at some role in what is going on. One thing I’ve noticed that is consistent: If I log into the dashboard of a blog that has the privacy settings set to "I would like only administrators of this blog, and network to see my blog." then the server starts to lock up. If I change the privacy settings to "I would like blog to be visible to everyone…" then the CPU load immediately goes back to normal and the problem goes away. Then I am unable to change back to the previous setting of "I would like only administrators of this blog, and network to see my blog." and the CPU issue does not occur…. at least for a while….

    I looked at privacy settings as a cause because only two blogs have this issue and they are both blogs we recently imported that will replace single installs. Since we haven’t mapped them yet and are doing work on them, they are set to the strictest privacy setting until we map them……

  • drmike
    • DEV MAN’s Mascot

    Top Process %CPU 39.0 /usr/bin/php /home/tracedef/public_html/wp-cron.php

    I’m looking at that and wondering. Andrew and James, when you folks were having problems with end users wrapping around rss feeds back to their own blogs, did it show up like this?

  • Andrew
    • Champion of Loops

    Hmm, I wonder if it’s the redirect causing the problem. Try changing it to the troublesome privacy option and comment out line 68 in the plugin. That line handles the redirect for that privacy option and is really the only bit of code that could cause any problems.

    What’s weird is that a ton of people use that plugin and we’ve never had a report of any problems. Your site is actually fairly small too and it’s in use on much larger sites. We’re not experiencing any problems on Edublogs/Campus either.

    Thanks,

    Andrew

  • Tracy
    • The Incredible Code Injector

    Andrew, I have commented that line out and will follow up if I see the issue occur again…. it’s a little hard to test as the issue is a bit inconsistent as once we change privacy settings from "I would like only administrators of this blog, and network to see my blog." to "Anyone can see blog" and back, the issue goes away for a while… There seems to be some window after this change where the issue doesn’t occur …

    Thank you so much for the feedback and help guys!

  • Tracy
    • The Incredible Code Injector

    Quick update: I just logged into one of the two blogs affected and it very apparent when the load problem exists because logging in takes 10 plus seconds…. I checked WHM to confirm CPU issue was occurring and it was at 5%, so I commented out the line suggested above and that in fact solved the problem, similar to what occurs when the privacy setting is manually changed to "anyone can see blog…." ….

    So… as best as I can tell it definitely seems to be some sort of redirect loop scenario.. in any case, we seem to be an isolated case, so we’ll just be sure to have line 68 commented out in the future. Really appreciate the help and feedback we got here, its always nice to get to the bottom of some weird behavior and get it sorted out. Thanks!

  • Hiranthi
    • Recruit

    That’s all I needed to know.

    Could you give me a link to it? I can’t find it on the WPMUDEV site and there isn’t a searchform to look for it..

    With the google custom search on the homepage I’m getting mostly WPMUDEV Premium search results.