Messy

Nowadays, everyone is talking about big data. Many hope that the proliferation of information will help us better predict future events like the rise of the Dow and the next torrential hurricane.

But the reality is, more data doesn’t always mean more accurate predictions. In fact, whenever you quantify something, you’re bound to pick up some random noise, such as errors in measurements. This poses problems for the big data approach since a model that takes into account all available information will suck up that noise, resulting in worse predictions.

Imagine you’re comparing the past lows and highs of the stock prices for two oil companies. Your predictions regarding the future prices of both stocks will probably be better if you cut out data from rare or unpredictable events, like a massive oil spill that sent one company’s stock plummeting.

Beyond that, the very act of measuring can distort the thing you’re attempting to measure. Take heart surgeons, for example. If they’re ranked by the number of successful surgeries they perform, some of them will naturally begin embellishing their numbers by hand-picking patients with the best prognoses.

Another human tendency is to prefer order over disorder. We like to see the world arranged into clear, predictable patterns. But actually, imposing order is not always beneficial.

Consider a 1990 Harvard study, where AnnaLee Saxenian compared two hubs of high-tech development, Route 128 in Massachusetts and California’s Silicon Valley.

Silicon Valley came out on top, and the reason was its disorganized nature. Companies there let employees jump around from job to job and even allowed ideas and knowledge to be exchanged between firms. The result was a climate of cooperation that fostered innovation and made it easier to recruit experts, helping the entire area to prosper.

Meanwhile, the companies of Route 128 kept their businesses in neat little silos, walled off by legally binding non-compete contracts, which made it difficult for companies to react to new developments or to handle crises. As a result, the whole area suffered.