'

Beyond DevOps: How Netflix Bridges the Gap

Понравилась презентация – покажи это...





Слайд 0

Josh Evans - Director of Operations Engineering November 16, 2015 Beyond DevOps: How Netflix Bridges the Gap


Слайд 1

Technical Debt Java 6 Perforce Single Master Jenkins Ant CentOS Asgard/Mimir Fall 2013


Слайд 2

How do we drive broad-based change?


Слайд 3

The Paved Road Java 7 Stash Jenkins Shards Gradle Ubuntu


Слайд 4

Some said You’re overloading us Too many projects Poor targeting Others said What took you so long? We’ve moved on Now we need to migrate That’s great but… We’re paying a high tax


Слайд 5

Expectations gap Division of labor Timing of solutions Leadership Affects Reputation Relationships Lost opportunities Organizational Debt


Слайд 6

How do we bridge the gap?


Слайд 7

“Remember that TIME is money…”


Слайд 8

Time is a form of currency


Слайд 9

Product Engineering Operations Engineering Challenges & Strategies Our time today…


Слайд 10

Product Engineering Operations Engineering Challenges & Strategies Our time today…


Слайд 11

Product Innovation winning moments of truth


Слайд 12


Слайд 13


Слайд 14

Every facet of the product 1400 AB tests in the last year & accelerating Continuous Innovation


Слайд 15

But wait, there’s more…


Слайд 16

Build It design code build bake test deploy Run It configure monitor triage fix …at scale, globally You build it, you run it


Слайд 17

Internet 1000s of starts per second 100,000s of requests per second 100,000,000 hours of content / day 3 AWS Regions, 3 AZs per region


Слайд 18

Relentless product innovation Building & running micro-services at scale, globally


Слайд 19

Product Engineering Operations Engineering Challenges & Strategies Our time today…


Слайд 20

DevOps is a software development method that emphasizes the roles of both software developers and other information-technology (IT) professionals with an emphasis on IT Operations. - Wikipedia The Gap


Слайд 21

Why? How?


Слайд 22

Quality Velocity Operational Excellence


Слайд 23

Operational Excellence is the continuous improvement of the management, design, and function of operational environments to achieve greater quality, velocity, and competitive advantage.


Слайд 24

Engineering Tools Insight & Real-time Analytics Performance & Reliability Operations Engineering is the application of software engineering practices to achieve and sustain operational excellence.


Слайд 25

Operations Engineering Service provider Operational excellence driver Cross-cutting solutions Undifferentiated heavy lifting


Слайд 26

Product Engineering Operations Engineering Challenges & Strategies Our time today…


Слайд 27

You’re overloading us What took you so long? Remember that feedback? We made assumptions Requirements – what & when Time for non-product work


Слайд 28

Move from assumptions to knowledge Affect change without imposing a tax? Achieve and sustain operational excellence? How do we…


Слайд 29

Time is a form of currency


Слайд 30

5 strategies for success in time-based economies software & organizational engineering


Слайд 31

1. Reach out


Слайд 32

What are your biggest operational pain points? How can we help? How well are we meeting your needs today? What would you like to see from us in the future? Listen Shower, rinse, repeat Talk to your engineering customers


Слайд 33

Grease the Squeaky Wheels low tolerance for tax more vocal than most


Слайд 34

High impact solutions Clarity on deliverables Lower operational tax Leadership, innovation, and partnership What they wanted


Слайд 35

Deliver on solutions Better road map definition & communication A more aggressive stance on automation Deeper investment into leadership, innovation, planning Our commitments


Слайд 36

2. Make an impact Apply what you’ve learned Deliver what matters


Слайд 37

global cloud console end to end delivery automation platform velocity with confidence


Слайд 38


Слайд 39

Pipelines - Automated Global Delivery


Слайд 40


Слайд 41

3. Make it easy to do the right thing


Слайд 42

Engineering time is scarce We must do more heavy lifting Supply & Demand


Слайд 43

Spinnaker manual step Automated migrations – Mimir Provide on-ramps


Слайд 44

Automate proven practices


Слайд 45

Alerting and Monitoring Apache & Tomcat Hardening Automated Canary Analysis Autoscaling Chaos Participation Consistent Naming ELB Configuration Healthcheck Configured Red-Black Pipeline Squeeze Testing Timeout & Fallback Tuning Workload Reliability Production Ready?


Слайд 46

Alerting and Monitoring Apache & Tomcat Hardening Automated Canary Analysis Autoscaling Chaos Participation Consistent Naming ELB Configuration Healthcheck Configured Red-Black Pipeline Squeeze Testing Timeout & Fallback Tuning Workload Reliability Production Ready?


Слайд 47

Old Version (v1.0) New Version (v1.1) Load Balancer Customers 100 Servers 5 Servers 95% 5% Metrics Canaries


Слайд 48

Old Version (v1.0) New Version (v1.1) Load Balancer Customers 0 Servers 100 Servers 100% Metrics Canaries


Слайд 49

Define Metrics A threshold Every n minutes Classify metrics Compute score Make a decision Automated Canary Analysis


Слайд 50

Canary Analysis Performance Integration Tests Chaos Conformity Static Unit Tests Make it easy to do the right thing Static & Functional Testing


Слайд 51

4. Reduce the cost of change


Слайд 52

Ongoing migrations Library propagation 100s of micro-services Complex dependencies Continuous, Broad-based Change


Слайд 53

Change Engineering Locate Communicate Facilitate


Слайд 54

Automated forensics Who last touched x? What team? Who was their manager? Who owns this artifact, repository, service?


Слайд 55

Whitepages Workday wrapper App & REST API Organization hierarchy Metadata Change log (###) ###-####


Слайд 56

Krieger REST-based service Sources Whitepages Stash Edda Jenkins Spinnaker Etc… { "content": {}, "_links": { "employees": { "href": "/api/employees/" }, "projects": { "href": "/api/projects/" }, "teams": { "href": "/api/teams/" }, "applications": { "href": "/api/applications/" }, "jobs": { "href": "/api/build/jobs" }, "masters": { "href": "/api/build/masters" }, "projectDistribution": { "href": "/api/teams/projectDistribution" } } }


Слайд 57

/api/employees?q=jevans "employees": [ { "id": "241", "firstName": "Josh", "lastName": "Evans", "username": "jevans", "email": "jevans@netflix.com", "jobTitle": "Director of Operations Engineering", "isManager": true, "isCurrent": true, "title": "Josh Evans (jevans) - Operations Engineering", "_links": { "self": { "href": "/api/employees/241" }, "manager": { "href": "/api/employees/117890" }, "team": { "href": "/api/teams/f9134a81" }, "projects": { "href": "/api/teams/f9134a81/projects" } } } ] }


Слайд 58

Security vulnerabilities Who owns this service? Platform updates Who is using this version of this library? Today – Targeted Coordination


Слайд 59

Automated, efficient technical project management Communication Guidance Tracking Low tax for TPMs & engineers Security Fix Java 9 Guava Future – Change Campaigns


Слайд 60

5. Develop Partnerships Beyond supply & demand


Слайд 61

Nearing completion Aggressive schedule Unexpected delays Commitment to June delivery Spinnaker 1.0 – 1H 2015


Слайд 62

Built their own continuous delivery solution Not positioned for engineering-wide support Believes common solutions Edge Engineering


Слайд 63

Partnership in Action Strong relationship Open discussions about concerns Decision - leaned forward +2 engineers on Spinnaker Successful 1.0 launch


Слайд 64

Moving Forward Together Containers? Achieving alignment Collaborative exploration Edge, Platform, Operations A new paved road?


Слайд 65

Paved Road adopted Adding new ones Production Ready ongoing Migrations easier Reputation improving Improved Service uptime Rate of change Payoffs


Слайд 66

Putting it to the test in 2016 Streaming production & test - EC2 Classic to VPC Highly cross-functional Complex dependencies Zero downtime Stay tuned…


Слайд 67

Five Strategies Reach out Make an impact Make it easy to do the right thing Reduce the cost of change Develop partnerships


Слайд 68

Open Sourced! https://netflix.github.io/


Слайд 69

Josh Evans jevans@netflix.com @ops_engineering Questions?


×

HTML:





Ссылка: