New to the community? Start here

WPMU Performance assistance w cluster/load balancer

We run WordPressMU 2.6.1 in a cluster of Solaris servers behind a load balancer. At present, we’ve got 700+ users and ~470 blogs (~350 active–others are test/abandoned).

I’ve made a plea for tips running WPMU in cluster behind a load balancer on the WPMU Forums, but haven’t gotten any assistance as of yet (we’re still hoping :wink:…

Here’re our specs:

  • Apache httpd 2.2.8

  • MySQL 5.0.51a

  • PHP 5.2.5

MU-Plugins:

Plugins:

We’re considering running WP Widget Cache, as well as Multi-DB.

We’re running 3 custom themes, which use a proprietary plugin to provide configuration values for analytics, ads, etc.

Wed, 26 Nov 2008 0:04:10
  • andrea_r
    • The Incredible Code Injector

    What’s the traffic like?

    Unless it’s rather heavy wicked traffic, load balancing for that amount of blogs seems pretty heavy-handed.

    multi-db (16) woudl be okay, but it should all be able to be handled by one server.

  • trent
    • Site Builder, Child of Zeus

    There is no doubt the setup that James put forth would deal with incredible amounts of blogs and traffic. Another thing to consider would be to have the web server with a mysql install on it as well as a slave to the main DB server. That allows you to use the replication (that hopefully Andrew can fix soon) that is included with the Multi-DB setup for some extra redundancy on the mySQL side.

    I have seen many documents on apache load balancing on howtoforge.com and thought about testing some of it out myself for fun, but without huge traffic, it is purely academic for myself.

    Trent

  • Andrew
    • Champion of Loops

    Are the hits spread pretty evenly over the 24 hour period or do you have one or more peak periods? If you have peak periods how many hits would you say you get during a peak?

    I ask because unless you have a small window of time when you’re absolutely pounded then you should be more than fine with a solution like the one James mentioned. 300K hits in a four hour window is only about 21 requests per second.

    Thanks,

    Andrew

  • James Farmer
    • Founder & Chair (honest)

    Because you’re only talking 350 blogs, I’d be tempted to see if the peaks could be managed effectibely with super cache.

    Alternatively, you could cache the most popular blogs (that’s something we’ve experimented with at edublogs with some success) – am not sure how complex modifying the plugin to cache just those blogs was though, and Andrew is pretty busy this week so we might need to wait until next to get that down for ya.

  • webmaestro
    • WPMU DEV Initiate

    We use WP Super Cache and are considering WP Widget Cache…

    We’ve got a cluster of servers and Master/Slave DB servers behind a load balancer (not just for blogs), and use httpd virtual CONF files instead of .htaccess files (AllowOverrides None) with the hope of eeking out more performance.

  • James Farmer
    • Founder & Chair (honest)

    What kind of performance are you getting at the moment? If it’s anything like this (a wordpress.com blog we use for benchmarking) then you are doing just fine and really couldn’t ask for much more:

    http://www.pingdom.com/reports/tdmmfmbt03rn/check_overview/?name=edubenchmarking.wordpress.com

    We’ve got ourselves at Edublogs into the 800MS average range which we’re absolutely delighted with, and that’s with very limited caching so we’re hopeful that with some more down the line we’ll be knocking on 600.

    Like this guy says:

    http://mashraqi.com/2008/09/startonomics-startup-scalability.html

    “4. Scalable startups worry about the right things:

    When building your startup, you should worry more about the foundation than sweating over performance increases.”

    I’d agree! As long as it works and works pretty well that’s 90% of your job done.

  • zealventures
    • WPMU DEV Initiate

    It is a very good topic! My company is getting into WPMU/Buddypress development. I would like to hire a consultant to assist us with scalability issue. Please contact me immediately. My email is [email protected].

    I am not sure if Incsub could spare their time to assist us right now. We will be paying very well for this project. Hope to hear from WPMU gurus soon!

  • webmaestro
    • WPMU DEV Initiate

    Thanks for the links… Interesting content…

    Our users have three ‘corporate’ themes available to them (each include ads and analytics tying the blogs to their corporate ‘parent’ sites), and the response time(s) tend to vary with each:

    Theme1: ~300ms

    Theme2: ~1700ms

    Theme3: ~5000-8000ms

    Those are fairly consistent. The ones with the higher rates, are frequently the more ‘savvy’ and configure their ‘themes’ with ‘custom’ sidebars (I’m hoping WP Widget Cache can help there…:wink:.

    Things I suspect increase the Response Time:

    – RSS feeds (it’d be great to offload the content via a Javascript)

    – category listing (with count of articles per cat)

    – tagcloud (usually using CTC, although some use default Tag Cloud)

  • ron_r (aka Mr. Andrea)
    • Design Lord, Child of Thor

    Theme1: ~300ms

    Theme3: ~5000-8000ms

    The disparity in these are a clear indication that you don’t have a server performance issue you have a theme performance issue. Instead of trying to boost performance, spend some time trying to find what is causing the themes to be slow.

  • andrea_r
    • The Incredible Code Injector

    “The ones with the higher rates, are frequently the more ‘savvy’ and configure their ‘themes’ with ‘custom’ sidebars”

    So, if this theme has users customizing it a lot with widgets, it’s quite possibly a widget as well.

    I’d go over the theme first with a fine-toothed comb, see how it manages with no customizations first, then I’d test out those widgets ones by one.

    I was having some page load issues, and it turns out it was the Astronomy pic of the Day widget, which was getting the *full-sized* image over 1 meg and just displaying it smaller.

  • webmaestro
    • WPMU DEV Initiate

    Yeah… I’ve spent time on that…

    I suspect the issue is Widgets (RSS latency, Categories, Tag Cloud, etc.). I understand wpmu-2.6.5 has improvements to RSS caching, so that should help, and if I add WP Widget Cache I’m hoping for *serious* improvements! (It couldn’t possibly be *my* code! :wink:

    The numbers for Theme2 & Theme3 are somewhat skewed, as all of the users with Theme3 tend to be more savvy, and put more widgets into their sidebar… If I remove them, the response time moves below 1000ms (at times as low as 55ms–w WP Super Cache, but logged in, so limited caching)…

    Theme2 & Theme3 cache the parent site’s pages/template locally (in /tmp/). There could be latency in the rendering of the page’s blog-specific values (entered via a proprietary plugin), but Theme1, Theme2 & Theme3 all share the same plugin. There are differences in how they’re implemented, but the DB calls are identical:

    // Build SQL query to get myStuff Options values
    
$myStuff_opts_query = "SELECT option_name, option_value FROM $wpdb->options where option_name like 'myStuff_%'";
    

    
// Run Query
    
$myStuff_opts = $wpdb->get_results($myStuff_opts_query);
    

    
$my_opts = array();
    
$myDebug='';
    
foreach ($myStuff_opts as $key) {
    
if ($_GET['my_debug'])
    
$myDebug.='<dt>'.$key->option_name.'</dt><dd>'.$key->option_value.'</dd>n';
    
str_replace('myStuff_','',$key->option_name)]=$key->option_value;
    
}

    I global $my_opts in my themes’ files (footer, header, single, etc.) then call each value via $my_opts['key'].

  • webmaestro
    • WPMU DEV Initiate

    Lately, we’ve been getting spikes in usage for the RSS feeds. It occurs once or twice a day, and then dies down.

    A result of the spikes is that our number of connections goes through the roof, and then response time increases as HTTP requests queue up.

    Anyone else see similar issues?

  • webmaestro
    • WPMU DEV Initiate

    Many of them do. And I’ve done some research which shows me that those links aren’t very well trafficked.

    FWIW, many have links to other blog/categories/tags RSS feeds within the same WPMU install…

    Do you think it behooves us to send those RSS requests to Yahoo Pipes or feedburner, or some other aggregator?

    Doesn’t WPMU cache RSS feed results?

  • webmaestro
    • WPMU DEV Initiate

    Many of them do. And I’ve done some research which shows me that those links aren’t very well trafficked.

    IOW, in spite of the fact that recursive feed appear to provide little actual benefit, some complain WPMU (or at least our installation) isn’t robust enough to handle the RSS feeds.

    I suspect the problem is due to WPMU running in a cluster behind a load balancer. I suspect that if we could somehow access WordPressMU feeds and other functions without going making HTTP requests, we could get around those issues related to the load balancer.

    IMPORTANT: WP Widget Cache 0.25.1 incompatible with WPMU

    BTW, WP Widget Cache 0.25.1 is not yet compatible with WordPressMU. I tested on WPMU 2.6.1 and it doesn’t provide separate a widget cache for each blog. IOW, if you’re using WP you’re fine since you only have one set of widgets. If you’re using WordPressMU, the blogs will share widget cache, so they’ll see each others’ cache… I’ve been in contact with the developer, who sent me a potential fix, but I’m still testing his changes and he has not yet made a release.

  • drmike
    • DEV MAN’s Mascot

    How did I miss this thread? :slight_smile:

    Ticket made since I’ve seen this issue as well: http://trac.mu.wordpress.org/ticket/852

    There seems to be confusion among end users that think the rss widget is used to get people to subscribe to the feeds instead of displaying outside feeds.

    Yes, wpmu does cache RSS data but it still has to pull the data out of the db, put it into proper form and output it. Not sure if wp-super-cache actually caches the feed pages though and I’m not in a position to check.

    We recommend to my own clients that they send the sitewide feeds to a service like feedburner. That’s more for problems where the entire feed goes down thanks to a single post from an end user that breaks the feed.

  • webmaestro
    • WPMU DEV Initiate

    andrea_r says:

    Oooo yeah, the default RSS widget can create some serious lag time. IME.

    I haven’t found sidebar RSS feed alternatives to the default RSS widget. Are you aware of any alternatives?

    I’d love to have a widget which loaded RSS feeds asynchronously via JavaScript, and parsed the feed for inclusion, instead of embedding the feed links & such within the blog’s HTML. That would minimize any lag (perceived or actual) related to fetching, parsing and rendering RSS feeds.

  • drmike
    • DEV MAN’s Mascot

    I’m not in a position to check but I believe the time check is within the function fetch_rss. I *believe* that function is within wp-includes/functions.php. I’m off to help a church right now but I can check afterwards if you don’t find it. Maybe bumping up the time check (I believe it;s an hour) to something sooner would work for you.

    You could remove the time check all together but that would probably annoy the sites on the other end if they noticed it.

  • webmaestro
    • WPMU DEV Initiate

    Will this type of fix also apply to feeds from other blog feeds in the same WPMU instance? For example assume http://blog1.domain.com/ grabs a feed from http://blog2.domain.com/feed/, or http://blog1.domain.com/category/coolcat/feed/ or http://blog2.domain.com/author/felix/feed/

    I suspect this might cause similar issues.

    Then again, if WordPress MU Domain Mapping or Multi-Site is active, other domains might also throw a wrench into the mix, but I suspect documentation could be provided with the hope of limiting ‘recursive’ feeds.