Big Data

It's Correlation Baby!


From Some to All... 
From Clean to Messy...
From Causation to Correlation...


Everyone knows that the Internet has changed how businesses operate, governments function, and people live. But a new, less visible technological trend is just as transformative: "big data." Big data starts with the fact that there is a lot more information floating around these days than ever before, and it is being put to extraordinary new uses. Bid data is distinct from the Internet, although the Web makes it easier to collect and share data. Bid data is about more than just communication: the idea is that we can learn from a large body of information things that we could not comprehend when we used only smaller amounts.


From Causation to Correlation:

These two shifts in how we think about data - from some to all and clean to messy - give rise to a third change: from causation to correlation. This represents a move away from always trying to understand the deeper reasons behind how the world works to simply learning about an association among phenomena and using that to get things done.

     Of course, knowing the causes behind things is desirable. The problem is that causes are often extremely hard to figure out, and many times, when we think we have identified them, it is nothing more than a self-congratulatory illusion. Behavioral economics has shown that humans are conditioned to see causes even when none exist. So we need to be particularly on guard to prevent our cognitive biases from deluding us; sometimes we just have to let the data speak.


Big data will have implications far beyond medicine and consumer goods; it will profoundly change how governments work and alter the nature of politics. When it comes to generating economic growth, providing public services, or fighting wars, those that can harness big data effectively will enjoy a significant edge over others.

The Internet has reshaped how humanity communicates. Big data is different: it marks a transformation in how society processes information. In time, big data might change our way of thinking about the world. As we tap ever more data to understand events and make decisions, we are likely to discover that many aspects of life are probabilistic, rather than certain.


Hat tip ~ Kenneth Cukier and Vikyor Mayer-Scoenberger
The Rise of Big Data ~ Foreign Affairs, May/June 2013
Authors of Big Data: A Revolution That Will Transform How We Live, Work, and Think (Houghton Miffin Harcourt, 2013)