Andrew Chen posted this interesting piece on A/B split testing: http://andrewchenblog.com/2009/03/02/does-ab-testing-lead-to-crappy-products/ which in turn caused someone in my twitterverse, to comment that metrics are not a substitute for strategy. And he is correct, however metrics can tell you whether your strategy worked or not. They can also help identify gaps in your strategy.
A/B split testing is just one of the tools in the web analysts arsenal to aid in improving the website. However there are some non or less measurable reasons for things. I work at the CBC which is Canada’s broadcasting company and we have a mandate to serve the public. I can guarantee there are shows that only have a snall audience and from a metrics only standpoint should be jettisoned for something more popular. However, we have a mandate to serve all Canadians and so these shows survive.
Here strategy trumps metrics, and so it should every time. This of course assumes you have a strategy. Building a website without a clear and defined purpose or goal is a sure road to failure; and the best analytics team in the world can’t save you. It should also be noted that design is not a substitute for strategy. Having a flashy cool looking website will not guarantee success. Design, like metrics is a tool nothing more.
In conclusion I agree with Andrew. I think it is important to remember where metrics fit into the larger scheme of things, and that metrics must work hand in hand with design to produce the best possible outcomes.