Ever seen one of those shows where they expose how magic tricks work? First they show you the trick where you are astounded until it is explained to you how you are being deceived? Well that is a pretty good analogy for the F5 performance report. The report appears to show F5 are the fastest Application Delivery Controller (ADC), but hiding in plain sight is a sleight of hand. F5 go into a lot of detail on how their testing is set up so you are led to believe that the configuration of the systems and the response sizes they test with are what you would use in the real world. In fact they even reference google statistics on response sizes – but then here is the crafty bit – they then test response sizes that do not reflect the google statistics they are referencing! They also neglect to point you to their own documentation where it explains that the setup they are using is not typical for a modern datacenter ADC and probably not even viable for the vast majority of load balancing use cases either.

So what do they do? They use a special performance mode that does not support most of the functions that customers use today in an ADC or load balancer but happens to give F5 great results.  It is a bit like the days when car manufacturers claimed completely unrealistic gas/petrol mileage by driving their test car in a way that pretty much no one would ever do in real life. F5’s own technical site admits that “the FastHTTP profile is a scaled down version of the HTTP profile that is optimized for speed under ideal traffic conditions.” It goes on to explain how it shouldn’t be used for out of order packets which of course excludes traffic from the internet or a mobile device or a Wi-Fi device. In other parts of the documentation it gives a high level list of functions that are not supported such as:

  • Secondary Persistence
  • Basic L7 Rewrite
  • DNS LB
  • Content Rewrite
  • Out Of Order Packets
  • SSL
  • Source IP preservation
  • Connection Logging
  • IPv6

For example, once a webserver setup requires more than basic cookie insertion and URL load balancing, the F5 appliance has to be configured using a standard virtual server and not the Performance (HTTP) virtual server and of course they won’t show you that because, as we found out,  their performance is massively lower when you use real world setups. Worse still, if you do start with a fast profile, it is not just a case of checking a box to make it a standard profile, you have to start again.

On the response sizes, the google statistics showed that 50% of response sizes are below 2KB, the average is 7.19KB  and 80% are below 8KB. So why did F5 only test with one size below 2Kb and then so heavily skew there tests to significantly larger and unrealistic response sizes? – It made them seem so much faster that way.

Below is a section of the table showing the KB per get for all web sites collected from a sample of several billions of pages that are processed as part of Google’s crawl and indexing pipeline.

Metric Mean Min 10% 20% 30% 40% Median 60% 70% 80% 90% Max
KB Per Get 7.19 0.00 0.43 0.63 0.92 1.31 1.93 2.90 4.38 7.96 18.46 35,932.92

 

OK, so I hear you saying where is the proof that NetScaler is any better? I knew you’d ask that question, so we had Tolly come in and verify our findings and publish them. You can get access to the report and the setups from our NetScaler web Page. Here you will see that NetScaler outperforms the equivalent F5 platform by up to 4.8 times.

If you have questions or need more detail on the testing, contact your local friendly Citrix sales team who will be happy to help.