Taguchi Sucks for Landing Page Testing

I recently spoke on the multivariate testing panel at eMetrics in San Francisco. You would think that I dropped a hand grenade into the room when I opined that the Taguchi Method was a bad fit for landing page testing. This is a well understood fact to anyone with a solid understanding of basic statistics. Unfortunately this seems to leave out most landing page testers…

In the world of landing page testing there are two common mathematical approaches: A-B Split testing, and parametric Multivariate testing. A subset of Multivariate testing is known as “Design of Experiments” (DoE) and is also called “fractional factorial”. A common fractional factorial approach is called the “Taguchi Method”.

Some online marketers consider A-B Split testing to be kind of wimpy, and endow fractional factorial methods with an almost mythical quality.

I spend way too much of my time explaining to people that at least when applied to landing page optimization fractional factorial methods are a really bad idea. Despite this, the illusion persists that this kind of testing is somehow state-of-the-art, when in fact, nothing could be further from the truth.

For a lot more detail (30 pages worth), download the whitepaper – “The Truth About Taguchi”.

Testing is composed of two important activities:

– Deciding what to test and coming up with good ideas
– Finding the best solution among your tested alternatives

People claim to get really good results with fractional factorial multivariate testing, and they credit this to the method that they use to analyze the data.

In reality, the improved conversion rates are the results of the great ideas for new landing page elements that go into the test. If all of your alternative landing pages designs are better then the original, it does not really matter what method you use to confirm that. Fractional factorial approaches may actually miss the best version of the landing page in your test and often lead you to a sub-optimal answer.

There is a huge mismatch between the original environment in which fractional factorial testing was developed and how it is usually applied to landing page optimization. It was basically transplanted to online marketing because it is relatively easy for a non-mathematical audience to understand, and not because of its appropriateness or fitness for the task.

The principal drawbacks of fractional factorial methods are:

 

• Very small test sizes
• Restrictive & inflexible test designs
• Less accurate estimation of individual variable contributions
• Drawing the wrong conclusions
• Inability to consider context and variable interactions

Despite misinformation to the contrary, fractional factorial methods do not offer any speed advantage over full factorial data-collection approaches (such as those available in the free Google Website Optimizer tool) if you are simply planning to understand the impact of the individual variables in the test (a so-called “main effects” analysis).

If you plan on using parametric (i.e. “model building” )approaches for landing page testing you should always use full factorial data collection regardless of the subsequent analysis you plan to do. It greatly simplifies your test design, and produces better estimates of the main effects.

All parametric methods (including fractional factorial) are also outclassed by newer non-parametric testing methods such as the SiteTuners TuningEngine, which can be licensed to run your own tests in-house and have the following advantages:

 

• Very large test sizes (1,000-10,000 times larger with the same data rate)
• Much faster data collection (on the same data rate)
• More accurate results (consider variable interactions)
• Flexible test construction
• No knowledge of statistics required

Hopefully this will set the record straight. If you still have an issue with this, and insist on proclaiming the superiority of fractional factorial methods, tell your statistician to call us and I will have my Chief Scientist beat them up properly.