Humans By The Hundred

If you like this presentation – show it...

Slide 0

Humans By The Hundred Scaling Big Data for Big Team Growth

Slide 1

$ whoami SRE Manager at Yelp CWRU Alum Pittsburgh native <3 Web Operations Just a dude

Slide 2

Yelp’s Mission: Connecting people with great local businesses.

Slide 3

Yelp Stats: As of Q2 2015

Slide 4

What is Yelp? Many sites: www, m, biz, api Mobile apps Partner platform Hundreds of developers Thousands of servers

Slide 5

Why Am I Here?

Slide 6

Slide 7


Slide 8

This talk is about people

Slide 9

Slide 10

Slide 11

Slide 12

Slide 13

Slide 14

Slide 15

Slide 16

The Goal

Slide 17

Iterate as fast as possible

Slide 18

Regardless of how many people are participating

Slide 19


Slide 20

How It Starts

Slide 21

Deployment: the early days Get a few people together in slack/irc/etc. Merge up the code Run the tests Manually test it in stage Cross your fingers

Slide 22

Slide 23

Slide 24

Things get slower... Tests take longer to run More hosts = longer downloads More developers = more eyeballs More features = more code

Slide 25

The Problem: Humans Are Fallible

Slide 26

The Problem: Humans Are Fallible “…oh @$#&”

Slide 27

Slide 28

The Problem, With Math Assume: Every change has a chance of success: 98% That means no test failures, no reverts, etc. Every deploy has a number of changes: n Any failure in the pipeline invalidates the deploy Let’s figure out the probability of a successful deployment: p

Slide 29

The Problem, With Math Only you p = .98 (98%) You and a friend p = .98 * .98 = .96 (96%) You and nine co-workers p = .98 * .98 * .98 * … * .98 = .82 (82%)

Slide 30

The Problem, With Math p = (.98)n

Slide 31

The Problem, With Math p = (.98)n exponential decay!

Slide 32

Slide 33

This doesn’t scale! More developers = more changes More changes = longer deploys Longer deploys = less time to develop Less time to develop = slower to iterate Slower to iterate != the goal

Slide 34

Mitigating Exponential Decay p = (.98)n

Slide 35

Mitigating Exponential Decay p = (.98)n

Slide 36

Slide 37

Making it harder to screw up Write more tests Write better tests Get better code reviews Get better infrastructure Switch programming languages Use better tools

Slide 38

Just write better software and stop making mistakes!

Slide 39


Slide 40

Slide 41

The Real World Testing builds confidence in our changes Testing does not protect you from failure Better tools, tests, and infrastructure can raise our success rates

Slide 42

Mitigating Exponential Decay p = (.98)n

Slide 43

Mitigating Exponential Decay p = (.98)n

Slide 44

Service-Oriented Architecture Large monolith > smaller services Services communicate over network Usually HTTP, but you can do RPC, SOAP, etc. Service = independent code base Independent deployments

Slide 45

Service-Oriented Architecture Benefits Smaller code bases = upper bound to n Failure domains become isolated Technology independence Federated responsibility

Slide 46

Service-Oriented Architecture Drawbacks everything becomes decoupled function calls start looking like HTTP requests versioning can be a nightmare tracking dependencies is hard data consistency becomes challenging end-to-end testing becomes hard(er), if not impossible

Slide 47

SOA scales people, not code.

Slide 48

Conquering SOA With the monolith, it’s easy to focus on mean time between failures (MTBF)

Slide 49

Conquering SOA In a SOA, focus on mean time to recovery (MTTR)

Slide 50

Conquering SOA Fail fast Anticipate failure Leverage iteration speed to recover fast

Slide 51

Conquering SOA Treat everything as distributed That means everything will fail Use timeouts, retries Find ways to degrade gracefully Fail fast & isolated Don’t rely on synchronous processes Prepare for eventual consistency

Slide 52

Reaping the Benefits Smaller failure domains Fewer people & changes to manage Deploys get smaller Deploys get faster Deploys become continuous

Slide 53

Reaping the Benefits Smaller changes means smaller code reviews means faster validation means smaller blast radius means faster iteration

Slide 54

Continuous Delivery Everyone works against master branch Master is deployed when commits added Deployment gated by tests Monitoring knows something is wrong before you do!

Slide 55


Slide 56


Slide 57

Tests are hard to get right.

Slide 58

Slide 59

Slide 60

Slide 61

Slide 62

Slide 63

Slide 64

How can we do better?

Slide 65

Slide 66

“Not Recommended” Tests

Slide 67

“Not Recommended” Tests If a test fails on master: a feature is broken on the live website, or your test sucks and you should ditch it In either case, we disable it Ticket is created Developers can fix it later or just bin it and start fresh

Slide 68

Reliable tests >> test coverage.

Slide 69

Don’t always run all the tests!

Slide 70

Tests of external services should be monitoring

Slide 71

Define your boundaries.

Slide 72

yelp.com / dataset_challenge 61K businesses 61K checkin-sets 481K business attributes 1.6M reviews 366K users 2.8M edge social-graph 495K tips Your academic project, research or visualizations, submitted by Dec 31, 2015 = $5,000 prize + $1,000 for publication + $500 for presenting* *See full terms on website Academic dataset from 10 cities in 4 countries!

Slide 73

@YelpEngineering YelpEngineers engineeringblog.yelp.com github.com/yelp

Slide 74


Slide 75