Everybody Lies summary

Data science is a useful tool. But what makes it special is not the amount of collected data, but rather that the data is useful; in other words, it’s the kind of data that can reveal patterns or make predictions.

Google is a case in point. Larry Page and Sergey Brin’s search engine, founded in 1998, became such a giant not simply because they were able to collect lots of data. Rather, what set Google apart was that the collected data could be used efficiently.

Recently, graduates at the University of Maryland were surveyed about their grade point averages, or GPA. Among the respondents, two percent admitted they had graduated with a GPA lower than 2.5 on a four-point scale. However, according to official records, the number was much higher, at 11 percent.

While this is just one example, it demonstrates a universal truth about surveying: people lie.

It’s difficult to get your head around just how big big data is. Each day, staggering amounts of data fly through Google alone, not to mention other search engines or other websites in general. The volume of this data means we can now do things we could never do before.

This is the third great power of big data: the size of the datasets means we can zoom in on a subset and reliably extract information from it.

Every day, we’re bombarded with stories about correlations. A food is linked with a disease. A habit is linked with success. These correlations seem credible at first, but correlation doesn’t necessarily imply a cause-and-effect relationship.

In reality, to learn about the causal effect of something, you’d need to establish causality by using randomized, controlled experiments, commonly called A/B tests. For instance, a study might report that people who drink alcohol only moderately are usually healthier. But does that imply that drinking moderately causes our health to improve? Of course not.

Although there are definite advantages to big data, it isn’t flawless. Its biggest limitation becomes patently clear in datasets with many variables: it’s difficult to extract reliable answers because the number of variables obscures possible findings.

Let’s consider the work of the behavioral geneticist Robert Plomin. In 1998, he thought he had found a gene, IGF2r, that was indicative of people’s IQ. He had obtained a dataset compiled from several hundred students containing information about DNA and IQ levels. Plomin compared their DNA with low and high IQs, and found that IGF2r was two times as likely to occur in students with high IQs.

Every time you type in a Google search or shop for a product online, you’re contributing to big data – what about the ethical considerations? What if the government had access to this data? What could they do with it?

Say someone typed “I want to kill myself” into a search engine. Should the local police be notified?