'

Challenges, Benefits and Best Practices of Performance Focused DevOps 

Понравилась презентация – покажи это...





Слайд 0

Wolfgang Gottesheim @gottesheim Challenges, Benefits and Best Practices of Performance Focused DevOps 


Слайд 1


Слайд 2

? ? ? Unit/Integration Tests Acceptance Tests Capacity Tests Release Developers When do we find performance problems?


Слайд 3


Слайд 4


Слайд 5


Слайд 6


Слайд 7

Test


Слайд 8

Production


Слайд 9

Dev Ops


Слайд 10

What Operations tells Developers…


Слайд 11

What Developers would like to know


Слайд 12

What Developers would like to know Top contributor is save of Mage_Core_Model_Abstract 70% of that time comes from Sales_Model_Quote Where are those calls coming from?


Слайд 13


Слайд 14


Слайд 15

Because NOBODY wants this …


Слайд 16


Слайд 17

~80% of problems caused by ~20% patterns YES we know this 80% Dev Time in Bug Fixing $60B Defect Costs BUT


Слайд 18

Define metrics that are understood across teams Share measurement methods and tools Make performance part of agile stories Broaden the View


Слайд 19


Слайд 20

But…


Слайд 21

It’s a culture thing


Слайд 22

Culture Measure Share Automation


Слайд 23

Starting from… Production Environment Developers CI Server Testing Environment Release


Слайд 24

…or maybe… Production Environment Developers CI Server Testing Environment Release


Слайд 25

We want to get to… Commit Stage Automated Acceptance Testing Automated Capacity Testing Release Developers


Слайд 26

METRICS to look at, WHY we want them, HOW they help


Слайд 27

Queues and Pools


Слайд 28

Online Banking: Slow Balance Check 1.69min (=101s!) To check balance! 87% spent in IIS 600! SQL Executions


Слайд 29

#1: Time Spent in IIS? Thread 32 in IIS took 87s to pass control to Thread 30 in ASP.NET


Слайд 30

#2: SQL Executions! A new connection for every statement…


Слайд 31

#2: SQL Executions! continued … #1: Same SQL is executed 67! times #2: NO PREPARATION because everything executed on new Connection


Слайд 32

Lessons Learned! ASP.NET Worker Thread Pool Sizing! DB Connection Pools More Efficient SQL


Слайд 33

Helpful Metrics Idle vs. Busy Threads # SQLs / Request # Opened connections


Слайд 34

Push without Plan


Слайд 35

Mobile Landing Page of Super Bowl Ad 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB http://apmblog.dynatrace.com/2014/01/31 /technical-and-business-web-performance-tips-for-super-bowl-ad-landing-pages/


Слайд 36

m.store.com redirects to www.store.com ALL CSS and JS files are redirected to the www domain This is a lot of time “wasted” especially on high latency mobile connections http://apmblog.dynatrace.com/2013/12/02/the-terrible-website-performance-mistakes-of-mobile-shopping-sites-in-2013/


Слайд 37

Fifa.com during Worldcup http://apmblog.dynatrace.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/


Слайд 38

Helpful Metrics Load Time # Images # Redirects # HTTP 3xx, 4xx, 5xx Size of resources


Слайд 39

Reusing Components


Слайд 40

Requirement: We need a report


Слайд 41

Using Hibernate results in 4k+ SQL Statements to display 3 items! Hibernate Executes 4k+ Statements Individual Execution VERY FAST But Total SUM takes 6s http://apmblog.dynatrace.com/2014/04/23/database-access-quality-metrics-for-your-continuous-delivery-pipeline/


Слайд 42

Helpful Metrics # SQL Statement Executions # of same SQLs Result Set Size


Слайд 43

Tools and Frameworks


Слайд 44

Online Bank: Transaction History CSV Download! Building CSV output in memory… Problem: Takes 207s! To download. 87% of Time spent in Garbage Collection


Слайд 45

Online Store: Rendering Search Result Problem: 4.4s to render result page Root Cause: Custom RegEx Library with performance issues on large strings


Слайд 46

Helpful Metrics Memory Usage Time Spent in APIs


Слайд 47

Architectural Decisions


Слайд 48

Symptoms HTML takes between 60 and 120s to render High GC Time Assumptions Bad GC Tuning Probably bad Database Performance as rendering was simple Project: Online Room Reservation System


Слайд 49

Developers built own monitoring void roomreservationReport(int roomid) { long startTime = System.currentTimeMillis(); Object data = loadDataForRoom(roomid); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, roomid); } Result: Avg. Data Load Time: 45s! DB Tool says: Avg. SQL Query: <1ms!


Слайд 50

#1: Loading too much data 24889! Calls to the Database API! High CPU and High Memory Usage to keep all data in Memory


Слайд 51

#2: On individual connections 12444! individual connections Classical N+1 Query Problem Individual SQL really <1ms


Слайд 52

#3: Putting all data in temp Hashtable Lots of time spent in Hashtable.get Called from their Entity Objects


Слайд 53

Custom Measuring Was impacted by Garbage Collection Just measured overall time but not # SQL Executions Learn SQL and don’t use Hashtables as Workaround Lesson Learned void roomreservationReport(int roomid) { long startTime = System.currentTimeMillis(); Object data = loadDataForRoom(roomid); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, roomid); }


Слайд 54

Helpful Metrics # SQL Executions # of SAME SQLs Connection Acquisition Time


Слайд 55

Performance as a Quality Gate Automated collection of performance metrics in test runs Comparison of performance metrics across builds Automated analysis of performance metrics to identify outliers Automated notifications on performance issues in tests Measurements accessible and shareable across teams Actionable data through deep transactional insight Integration with build automation tools and practices


Слайд 56

PERFORMANCE as part of our Continuous Delivery Process Commit Stage Automated Acceptance Testing Automated Capacity Testing Release Developers


Слайд 57

# Images # Redirects Size of Resources # SQL Executions # of SAME SQLs # of Connections Time Spent in API Remember: Use Tools to measure… # Calls into API # Functional Errors 3rd Party calls # of Domains Total Page Size # Items per Page # AJAX per Page


Слайд 58


Слайд 59

Collaborate Verify Measure


Слайд 60


Слайд 61


Слайд 62

If we do all that …


Слайд 63

Unit/Integration Tests Acceptance Tests Capacity Tests Release Developers


Слайд 64

CANCELLED


Слайд 65

… so there is more time for …


Слайд 66


Слайд 67

blog.dynatrace.com bit.ly/dttrial


Слайд 68

A FINAL THOUGHT!


Слайд 69


Слайд 70

Putting it into Test Automation We identified a regresesion Problem solved Exceptions probably reason for failed tests Now we have the functional and architectural confidence Let’s look behind the scenes


Слайд 71

And in your Pipeline Commit Stage Compile Execute Unit Test Code Analysis Build installers Automated Acceptance Testing Automated Capacity Testing Manual testing Key showcases Exploratory testing Unit & Integration Tests Functional Tests Performance Tests Production Monitoring Functional Tests


Слайд 72

Wolfgang Gottesheim Free Tools: http://bit.ly/dttrial Follow me @gottesheim Email me wolfgang.Gottesheim@dynatrace.com http://blog.dynatrace.com


×

HTML:





Ссылка: