Seeing double headers due to cache

I have a site hosted with WPMU DEV, since after we have moved we are facing a weird issue where you see a double header at times on the site. It goes away when we clear the cache and comes back after a few days.