'

Challenges, Benefits and Best Practices of Performance Focused DevOps 

Понравилась презентация – покажи это...





Слайд 1

Wolfgang Gottesheim @gottesheim Challenges, Benefits and Best Practices of Performance Focused DevOps 


Слайд 2


Слайд 3

? ? ? Unit/Integration Tests Acceptance Tests Capacity Tests Release Developers When do we find performance problems?


Слайд 4


Слайд 5


Слайд 6


Слайд 7


Слайд 8

Test


Слайд 9

Production


Слайд 10

Dev Ops


Слайд 11

What Operations tells Developers…


Слайд 12

What Developers would like to know


Слайд 13

What Developers would like to know Top contributor is save of Mage_Core_Model_Abstract 70% of that time comes from Sales_Model_Quote Where are those calls coming from?


Слайд 14


Слайд 15


Слайд 16

Because NOBODY wants this …


Слайд 17


Слайд 18

~80% of problems caused by ~20% patterns YES we know this 80% Dev Time in Bug Fixing $60B Defect Costs BUT


Слайд 19

Define metrics that are understood across teams Share measurement methods and tools Make performance part of agile stories Broaden the View


Слайд 20


Слайд 21

But…


Слайд 22

It’s a culture thing


Слайд 23

Culture Measure Share Automation


Слайд 24

Starting from… Production Environment Developers CI Server Testing Environment Release


Слайд 25

…or maybe… Production Environment Developers CI Server Testing Environment Release


Слайд 26

We want to get to… Commit Stage Automated Acceptance Testing Automated Capacity Testing Release Developers


Слайд 27

METRICS to look at, WHY we want them, HOW they help


Слайд 28

Queues and Pools


Слайд 29

Online Banking: Slow Balance Check 1.69min (=101s!) To check balance! 87% spent in IIS 600! SQL Executions


Слайд 30

#1: Time Spent in IIS? Thread 32 in IIS took 87s to pass control to Thread 30 in ASP.NET


Слайд 31

#2: SQL Executions! A new connection for every statement…


Слайд 32

#2: SQL Executions! continued … #1: Same SQL is executed 67! times #2: NO PREPARATION because everything executed on new Connection


Слайд 33

Lessons Learned! ASP.NET Worker Thread Pool Sizing! DB Connection Pools More Efficient SQL


Слайд 34

Helpful Metrics Idle vs. Busy Threads # SQLs / Request # Opened connections


Слайд 35

Push without Plan


Слайд 36

Mobile Landing Page of Super Bowl Ad 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB http://apmblog.dynatrace.com/2014/01/31 /technical-and-business-web-performance-tips-for-super-bowl-ad-landing-pages/


Слайд 37

m.store.com redirects to www.store.com ALL CSS and JS files are redirected to the www domain This is a lot of time “wasted” especially on high latency mobile connections http://apmblog.dynatrace.com/2013/12/02/the-terrible-website-performance-mistakes-of-mobile-shopping-sites-in-2013/


Слайд 38

Fifa.com during Worldcup http://apmblog.dynatrace.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/


Слайд 39

Helpful Metrics Load Time # Images # Redirects # HTTP 3xx, 4xx, 5xx Size of resources


Слайд 40

Reusing Components


Слайд 41

Requirement: We need a report


Слайд 42

Using Hibernate results in 4k+ SQL Statements to display 3 items! Hibernate Executes 4k+ Statements Individual Execution VERY FAST But Total SUM takes 6s http://apmblog.dynatrace.com/2014/04/23/database-access-quality-metrics-for-your-continuous-delivery-pipeline/


Слайд 43

Helpful Metrics # SQL Statement Executions # of same SQLs Result Set Size


Слайд 44

Tools and Frameworks


Слайд 45

Online Bank: Transaction History CSV Download! Building CSV output in memory… Problem: Takes 207s! To download. 87% of Time spent in Garbage Collection


Слайд 46

Online Store: Rendering Search Result Problem: 4.4s to render result page Root Cause: Custom RegEx Library with performance issues on large strings


Слайд 47

Helpful Metrics Memory Usage Time Spent in APIs


Слайд 48

Architectural Decisions


Слайд 49

Symptoms HTML takes between 60 and 120s to render High GC Time Assumptions Bad GC Tuning Probably bad Database Performance as rendering was simple Project: Online Room Reservation System


Слайд 50

Developers built own monitoring void roomreservationReport(int roomid) { long startTime = System.currentTimeMillis(); Object data = loadDataForRoom(roomid); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, roomid); } Result: Avg. Data Load Time: 45s! DB Tool says: Avg. SQL Query: <1ms!


Слайд 51

#1: Loading too much data 24889! Calls to the Database API! High CPU and High Memory Usage to keep all data in Memory


Слайд 52

#2: On individual connections 12444! individual connections Classical N+1 Query Problem Individual SQL really <1ms


Слайд 53

#3: Putting all data in temp Hashtable Lots of time spent in Hashtable.get Called from their Entity Objects


Слайд 54

Custom Measuring Was impacted by Garbage Collection Just measured overall time but not # SQL Executions Learn SQL and don’t use Hashtables as Workaround Lesson Learned void roomreservationReport(int roomid) { long startTime = System.currentTimeMillis(); Object data = loadDataForRoom(roomid); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, roomid); }


Слайд 55

Helpful Metrics # SQL Executions # of SAME SQLs Connection Acquisition Time


Слайд 56

Performance as a Quality Gate Automated collection of performance metrics in test runs Comparison of performance metrics across builds Automated analysis of performance metrics to identify outliers Automated notifications on performance issues in tests Measurements accessible and shareable across teams Actionable data through deep transactional insight Integration with build automation tools and practices


Слайд 57

PERFORMANCE as part of our Continuous Delivery Process Commit Stage Automated Acceptance Testing Automated Capacity Testing Release Developers


Слайд 58

# Images # Redirects Size of Resources # SQL Executions # of SAME SQLs # of Connections Time Spent in API Remember: Use Tools to measure… # Calls into API # Functional Errors 3rd Party calls # of Domains Total Page Size # Items per Page # AJAX per Page


Слайд 59


Слайд 60

Collaborate Verify Measure


Слайд 61


Слайд 62


Слайд 63

If we do all that …


Слайд 64

Unit/Integration Tests Acceptance Tests Capacity Tests Release Developers


Слайд 65

CANCELLED


Слайд 66

… so there is more time for …


Слайд 67


Слайд 68

blog.dynatrace.com bit.ly/dttrial


Слайд 69

A FINAL THOUGHT!


Слайд 70


Слайд 71

Putting it into Test Automation We identified a regresesion Problem solved Exceptions probably reason for failed tests Now we have the functional and architectural confidence Let’s look behind the scenes


Слайд 72

And in your Pipeline Commit Stage Compile Execute Unit Test Code Analysis Build installers Automated Acceptance Testing Automated Capacity Testing Manual testing Key showcases Exploratory testing Unit & Integration Tests Functional Tests Performance Tests Production Monitoring Functional Tests


Слайд 73

Wolfgang Gottesheim Free Tools: http://bit.ly/dttrial Follow me @gottesheim Email me wolfgang.Gottesheim@dynatrace.com http://blog.dynatrace.com


×

HTML:





Ссылка: