The answer lies in the Power Law :-)

As a kid I was always too tall for my age and had difficulty explaining this monstrosity to people (as if I was responsible for it!!), that is, until the knowledge of Normal Distribution came to my rescue. Every tangible thing around us follows normal distribution, well almost everything…and if something doesn’t, there’s a theory that says, the distribution of those distributions will follow normal distribution. So, you see, Normal Distribution is like God’s law - you cannot defy it. Once, you get this point, it’s so easy to use it in arguments. For instance, if you are blessed with less than average height, you could say something like this – well you know what, there are way too many people with average height and there are quite a few monsters like yours truly… and since normal distribution cannot be defied, some people had to be short to prevent the world from God’s wrath, so here I’m.  If you are a stats genius, you’ll quickly see the fallacy in the logic, but I can tell you that it works most-of-the-time (because normal distribution prevents too many people from becoming stats geniusJ).

Jokes apart, here’s the thing. We have been led to believe that Normal distribution is the best way to describe all sorts of common phenomenon that happens around us. It appeals to our common sense. Isn’t bell-curve the most intuitive thing in the world? Even a 10-year old can understand it. In fact, Mirowski says – “Linear thinking is engrained in our mentality. Scientific and mathematical models are based on the concepts of equilibrium and linearity. Economics, for instance, is almost theistic in its assumption that economic phenomena trend toward equilibrium.” We have been trained to look for averages and variances (the two criteria for describing any ND…What’s the average age of Indian cricket team?, average attendance record of students in a class, average height of Dutch people etc.) Imagine a world where averages didn’t have any meaning, where knowing variances did not add anything to your knowledge. That’s the world we are heading towards. More on this later!!

When we view the world using Normal distributions lens, we run the risk of ignoring important future trends as unimportant outliers in the data set. These future trends will only appear outside of the Bell curve until it is too late to take any action. (Nassim Nicholas Taleb has written two books – Fooled by Randomness and The Black Swan to discuss this very issue. His words seem almost prophetic in the backdrop of recent global economic meltdown, an extreme event). There’s a growing realization among statisticians and business strategists that ND doesn’t reflect the true nature of our new world. In fact, people who are still using normal distribution to explain contemporary business models are quickly losing ground. Check this blog post by John Hagel (the author of Net Worth, Net Gain and more recently The Power of Pull) for an in-depth analysis of Gaussian world (Normal Distribution) v/s Paretian World (Power Law) debate. John says – “We’re shifting from a Gaussian world to a Paretian world, with profound implications for business” and then goes on to explain how Guassian (ND) distribution fails to model the workings of post-modern world. It’s an extremely interesting read and I cannot recommend it any more strongly. John talks about two conditions under which a Gaussian system can morph into a Paretian system – when tension increases and cost of connections decreases.  The increasing usage of digital media and particularly social networks has brought down the barriers for meeting these two criteria. In fact, business models in digital media cannot be satisfactorily understood if one doesn’t understand the Power Law. If you had been using ND to assess the nature of digital world, you probably arrived at wrong conclusions. The virtual world is not governed by the “real-world” normal distribution (you see, we don’t call it virtual world for nothing!!). 


So, what is Power Law?
Power law distribution is just one of many probability distributions, but it is considered a valuable tool to assess uncertainty issues that normal distribution cannot handle when they occur at a certain probability. Pareto distributions have long and fat tails with potentially infinite variance, unstable means, and unstable confidence intervals. Such distributions often go by the names of power laws, Zipf distributions, Pareto distributions (80/20 principle) or the long tail. It is a common observation that virtually everything measurable on the Internet — site popularity, site traffic, ad revenues, tag frequencies on del.icio.us, open source downloads by title, Web sites chosen to be digg‘ed, Google search terms — follows such power laws or curves. The fundamental difference between normal distribution and Pareto distribution lies in assumptions about the correlation among events. In a normal distribution events are assumed to be independent whereas in Pareto distribution, events are interdependent. In social networks, a disproportionate number of events are interdependent – and so Gaussian distribution cannot be applied to explain the workings of social networks.


Implications of Power law:
§  In understanding social networks (source – Clay Shirky): The average is meaningless in power law distribution. Assuming that social network is held together by its average members leads us to underestimate seriously the likelihood of sharing a link with someone we meet. In fact, social networks are held together not by the bulk of people with hundreds of connections but by the few people with tens of thousands. It is the presence of these highly connected people that forms the backbone of the social networks. When you list the participants in a Small World network in rank order by number of connections, the resulting graph approximates a power law distribution: a few people account for a wildly disproportionate amount of the overall connectivity. Malcolm Gladwell, in his book “The Tipping Point”, calls these people Connectors.
§  Implication for managers: This from a paper by Bill McKelvey (check refrences) – “Given that the world in which organizations live is frequently Paretian, what types of changes in thinking and practices are required of managers to successfully interact and prosper in a Paretian world? How to transform the new understanding of scalability and scalefree theories in tools useful to anticipate and govern the transformation of small initiating events into extreme events, either to favourably shape the emergence of new business market and/or organizational structures or to avoid their potentially lethal consequences? Ignoring emergent properties allows the radical simplification of reality. This assumption and the consequent methodology and methods have permeated entire disciplines within management and business studies, from decision-making to marketing, from logistic and supply chain management to strategy.”
§  Implications for Web analytics: “In a world where most bloggers get below average traffic, audience size can't be the only metric for success. This is the reason why most web analytics software now come with advanced segmentation features that can be used to segment long tail traffic. Avinash Kaushik says that the long tail provides an average of seven times the data of short tail metrics. For any blog post, the small number of residual daily visits and subscriptions eventually match or surpass the initial surge of visits and subscriptions when the article is first written and posted. This is called the long-tail of ROI and it cannot be ignored anymore.

Looking at world with Pareto’s lens is difficult. Think about it – if 90% of data points fall below average, what’s the point in knowing averages and variances. However, if you don’t look for averages and variances, what will you look for? The answer is you’ll look for outliers in the fat-tail. You’ll place sophisticated filters to capture the early sings of an extreme event from fat-tail. That’s where you’ll find Facebook, Twitter, sings of global economic recession…all kinds of extreme events.

Once you understand Power Law, you also understand why every brand should not create its own community. A, because making a community popular is not in your hands (there are way too many choices for people) and B, not every organization can sustain a community, in fact organizations are not capable of sustaining communities, communities are built around social-objects, not brands. Power law is precisely the reason why brands are (and should) go to places where people already are.  You cannot plan to move from the fat-tail to the head; you can only try and watch for signs. A better and more reliable alternative is attaching oneself to someone who’s already in the head. That’s the reason why having your company’s home page on Facebook makes so much sense. Facebook is not a planned success, how and why it moved from fat-tail to head is a mystery. Since, you cannot control such extreme events, the best you can do is look for them and make use of them.

Going back to the height distribution example. I think it’d be an interesting sci-fi story to see how a planet would look like in which heights of people followed power law. Probably a topic for another post!! 

Links:

No comments:

Post a Comment