Web Science: How is it different? Daniel Tunkelang Head of Query Understanding

If you like this presentation – show it...

Slide 0

Daniel Web Science: How is it different? Daniel Tunkelang Head of Query Understanding

Slide 1

Slide 2

tl;dr: The scientific method is alive and well. Big data has just changed the economics.

Slide 3

How have the web and big data changed science? Let’s ask some of the experts.

Slide 4

“You have to kiss a lot of frogs to find one prince. So how can you find your prince faster? By finding more frogs and kissing them faster and faster.” Mike Moran Do It Wrong Quickly: How the Web Changes the Old Marketing Rules, 2007 Cited by Kohavi in Online Controlled Experiments at Large Scale, 2013

Slide 5

Web Science = faster, cheaper experiments.

Slide 6

“The cost of experimentation is now the same or less than the cost of analysis. You can get more value…by doing a quick experiment than from doing a sophisticated analysis.” Michael Schrage Value-Creation, Experiments, and Why IT Does Matter, 2010

Slide 7

Web Science = more experiments, less analysis?

Slide 8

“with massive data, this approach to science — hypothesize, model, test — is becoming obsolete… Petabytes allow us to say: "Correlation is enough." We can stop looking for models…analyze the data without hypotheses…throw the numbers into the biggest computing clusters the world…and let…algorithms find patterns where science cannot.” Chris Anderson The End of Theory, 2008

Slide 9


Slide 10


Slide 11

Let’s rewind.

Slide 12

What makes it science?

Slide 13


Slide 14


Slide 15


Slide 16

The scientific method still works today. What’s changed is the economics.

Slide 17

Scientific Method 1747

Slide 18

Scientific Method Today

Slide 19

It’s the economy, science. Yesterday Experiments are expensive, choose hypotheses wisely. Today Experiments are cheap, do as many as you can!

Slide 20

What about Web Science?

Slide 21

A/B testing: everybody’s doing it.

Slide 22

Google: 20k search experiments per year

Slide 23


Slide 24

The Myth of Insight

Slide 25

Scientists gain insight by staring at data.

Slide 26

Big data tools improve data exploration.

Slide 27

In hypothesis generation, quantity trumps quality.

Slide 28

Except when it doesn’t.

Slide 29

Slide 30

Easier to analyze data than research humans.

Slide 31

But we pay the price. Example: search engine improvements in batch evaluations don’t always predict real user benefits. [Hersh et al, 2000] Do Batch and User Evaluations Give the Same Results? [Turpin & Hersh, 2001] Why Batch and User Evaluations do not Give the Same Results [Turpin, Scholer, 2006] User Performance versus Precision Measures for Simple Search Tasks But also see… [Smucker & Jethani, 2010] Human Performance and Retrieval Precision Revisited

Slide 32

When local optimization is cheap, you neglect the rest.

Slide 33

To summarize: how is web science different? Online testing is cheaper and scalable. Data exploration tools make hypothesis generation cheaper and easier. But the experiments that are easy and cheap aren’t always the most valuable. Easy to forget our biases as scientists.

Slide 34

Take-Aways The scientific method is alive and well. Big data has just changes the economics. Cheaper hypothesis testing and generation has already been transformative. That’s why big data matters. But we neglect the human side of scientific experimentation at our peril.

Slide 35

Daniel Tunkelang dtunkelang@linkedin.com https://linkedin.com/in/dtunkelang