Are Algorithms the Magic Bullet?
Are Algorithms The Magic Bullet? by Caleb Elston
(I am trying something new, I have recorded an MP3 version of this post. I hope you find it useful as another way to stay connected. You can even subscribe as an iTunes podcast here. Let me know if you like this idea.)
Algorithms are in the news a great deal recently. The Netflix prize has finally been seized, Microsoft and Yahoo are joining forces to hone their ad targeting and search algorithms, hoping against all hopes to out Google Google, not to mention the countless social news filtering services cropping up to help us all cope with information overload. All this attention on algorithms begs the question, how important are algorithms to the success of these companies? Is a killer algorithm the magic bullet? I think not. Chris Dixon has a fantastic post about the topic which can be found here. Chris posits it's about the data, not the algorithm. I agree, but would take it a step further, it is about unique insight. The mainstream press loves talking about algorithms because they sound "technological" and complicated; futuristic in some way. Many companies believe algorithms are the magic bullet, they aren't...insight is. Algorithms are simply one way of executing on insight, a way that is particularly well suited to large data sets, no doubt. The fact remains, you still have to ask the right questions of the right data. If you are trying to beat Google or Netflix, both algorithmic bastions in their own right, then the pursuit of a slightly better algorithm is foolhardy. There are very few instances in business when doing the exact same thing as a competitor, only 5% better, is a winning long term strategy, yet many companies seem to operate as if it were, putting their faith in besting a competitor's algorithm with their slightly better version. It doesn't make sense. Google, Netflix, and Amazon are all known for their algorithmic excellence and as a result are often portrayed as owing their success to their unbelievably complicated and deep understanding of how to write them. Hogwash! If you look back, you realize their initial algorithms were rough and incomplete. But they all sought to illuminate a killer insight: links as authority, collective ratings as an indicator of movie preference, and browsing/purchase behavior as fuel for targeted cross promotion. The algorithm facilitated their insight. It was only after their insight was validated did these companies commit massive resources to advancing them. What's so fascinating to me is that the raw data that makes Google Google already existed in plain sight, it wasn't held under lock and key, nor was it immensely difficult to query. It was the human leap of logic, the synthesis of one world (academic citations) to the web world (links as authority) that cried out for the creation of PageRank. When some company eventually usurps Google it won't be because they do PageRank better, or because they can crawl the web a bit faster or deeper. No, this company will have discovered a new bridge into surfacing information, some new insight into the way it can be discovered, filtered, stored and made more useful. Companies need to stop focusing on building better algorithms and instead focus on coming up with unique insights.
