Last week was a good one for XenServer.  Within just a few days of each other:

  • Virtualization Review published benchmark performance test results that showed XenServer to be “the porsche of hypervisors“ while acknowledging the excellent runner-up performance of Hyper-V (no surprise there – same architecture, smart chaps, large team) and the rather pokey performance of ESX
  • An announcement that SAP has selected XenServer for virtualization of its XenApp farms, in a deployment over 500 servers
  • An announcement thatTesco PLC is using XenServer to virtualize its mission critical point of sales transaction software in a deployment of over 500 HP servers.

(Oh, and XenServer Virtual Infrastructure, including all management is 100% free. Get it here)

Combined with the detailed and thorough independent benchmarks of Project Virtual Reality Check, these recent market validations place XenServer at the very forefront of virtualization, and are a tremendous validation of the hard work of the XenServer crew, and the incredible commitment by the powerful Xen community, which develops the engine of this Porsche. 

The Tesco announcement is personally important to me:  Some 40% of food in the UK is purchased at Tesco, and all those point of sale transactions cross XenServer in real time. So when my mum swipes her card at the check-out, XenServer needs to do its thing!  Personal too, in that this deployment, targeted at 1,500 servers, plays a key role in enabling Tesco to reach its commitments for a reduced carbon footprint.  Sure, XenServer does that every day, in countless enterprises and clouds, but this one is a little more personal.

The stand taken by Keith Ward and team at Virtualization Review is also personal.  Though Keith has been quick to point out that his team did not violate the VMware EULA in its performance benchmarking of XenServer, Hyper-V and ESX, FUD from the VMware side appears to allude to the fact that this may be so.  Hats off to this courageous team which attempted to provide a thoroughly unbiased comparison of the performance of the three hypervisors, and continually consulted VMware for guidance to ensure that they were using valid use cases:”The Porsche of hypervisors? XenServer. Raise your hand if you saw that coming. It outperformed Hyper-V and ESX in most categories. The pokiest? ESX. Again, not at all what I expected. In fact, even in the few tests ESX came out on top, it barely edged out the competition. Microsoft did well across the board, and is definitely a fine product.”

Of course, I immediately wondered what my friends at VMware would do to spoil the party.   Remember, this is the organization that challenged IDC and Yankee group on their empirical research that showed VMware was vulnerable to Microsoft Hyper-V and Xen, and losing share to both.   

The VMware benchmarking team are smart folks.  They appear eminently reasonable:”Benchmarking is a difficult process fraught with error and complexity at every turn. It’s important for those attempting to analyze performance of systems to understand what they’re doing to avoid drawing the wrong conclusions or allowing their readers to do so. For those that would like help from VMware, we invite you to obtain engineering assistance from benchmark@vmware.com. And everyone can benefit from the recommendations in the Performance Best Practices and Benchmarking Guidelines paper.  Certainly the writers at Virtualization Review can.”

However Eric Horschmann slaps Virtualization Review with the conclusion that “We’re Not Against Benchmarking – We’re Only Against Bad Benchmarking” implying that VMware had no input into or approval over the results, and moreover asserting that benchmarking is such a deep science that it clearly is only accessible to a small VMware clique whose methodology is not to be shared with mere mortals.

In Horschmann’s rather lengthy but shallow critique of the work done by the Virtualization Review team, he only mildly alludes to the fact that VMware engineers were in full approval of the methodology and tests used.  Keith Ward confirms that”We talked extensively with VMware during the process, and an engineer in the benchmarking department approved our methodology before we went to press.”

Moreover”To ensure the validity of our test results and testing environment, we enlisted the help of Stuart Yarost to formulate and validate the test plan. Yarost is an ASQ Certified Software Quality Engineer and Certified Quality Engineer with more than 22 years’ experience in the software and quality fields. Yarost currently holds the position of Vice Chair of Programs for the ASQ Software Division. A special thanks to Yarost for his help.”


Now, given that XenServer is doing so well, I’m inclined to be really positive to the folks in the corner with the black eye.  So let’s assume Eric Horschmann and the lads at the VMware Ministry of Truth are right.  That means

  1. Keith Ward, Rick Vanover, Stuart Yarost and team (including engineer from VMware who approved of the methodology), you’re all idiots.
  2. Benchmarking hypervisors requires deep science, and ESX as the industry leader is profoundly deep, requiring such profoundly, super deep expertise to tune and benchmark that it is just not possible for ordinary humans (hence the VMware EULA that forbids publications of comparative benchmarks – and Horschmann clearly states that this is why the EULA is so restrictive). Virtualization Review was foolish to think that even with a staff of trained VMware engineers, an independent consultant, and advice and approval from VMware, that it would be possible to reproduce the fine art that is uniquely owned by the VMware benchmarking team. (I’ve commented on this previously in “VMware Wins! (Bad Science Required)”. What they didn’t know of course is that results are meaningless – this is just spin.
  3. Since rational, well disposed folks trying to make their ESX installation work as well as possible failed utterly in the attempt (that is, XenServer and Hyper-V clobbered ESX on performance), we conclude that probably no normal user could get ESX to perform either.  It really is a super complex (expensive) hypervisor, and it’s so difficult to get to work that if you’re merely a well intended VMware user, you have no hope.
  4. Therefore, probably most VMware installations run extremely badly, since they are run by mere mortals who could never understand how deep and fickle the beast called ESX is.  And even if you’ve spoken to a VMware engineer about how to get it right, you’re probably still doing it wrong.

Wake up VMware.  Your response rings hollow, and we are afforded yet another laugh at your expense.