Java Performance: Biggest Mistakes

If you like this presentation – show it...

Slide 0

More on http://blog.dynatrace.com Dynatrace Free Trial: http://bit.ly/dttrial Andreas Grabner - @grabnerandi Java Performance: Biggest Mistakes

Slide 1

And this is where I am from

Slide 2

Slide 3

Slide 4

Can you tell my age by looking at my first computer?

Slide 5

That’s why I ended up talking about performance

Slide 6

Nobody wants this …

Slide 7

Slide 8

Nor this …

Slide 9

Slide 10

As it leads to this …

Slide 11

The “War Room” Facebook – December 2012

Slide 12

And potentially to this …

Slide 13

Slide 14

And this …

Slide 15

Slide 16

And that’s why Business doesn’t like it either …

Slide 17

~80% of problems caused by ~20% patterns YES we know this 80% Dev Time in Bug Fixing $60B Defect Costs BUT

Slide 18

6 Situations on WHY this happened, HOW to avoid it

Slide 19

Slide 20

“Blindly” (Re)use Existing Components

Slide 21

Requirement: We need a report

Slide 22

Using Hibernate results in 4k+ SQL Statements to display 3 items! Hibernate Executes 4k+ Statements Individual Execution VERY FAST But Total SUM takes 6s

Slide 23

Slide 24

Not every Architect makes good decisions

Slide 25

Symptoms HTML takes between 60 and 120s to render High GC Time Assumptions Bad GC Tuning Probably bad Database Performance as rendering was simple Project: Online Room Reservation System

Slide 26

Developers built own monitoring void roomreservationReport(int roomid) { long startTime = System.currentTimeMillis(); Object data = loadDataForRoom(roomid); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, roomid); } Result: Avg. Data Load Time: 45s! DB Tool says: Avg. SQL Query: <1ms!

Slide 27

#1: Loading too much data 24889! Calls to the Database API! High CPU and High Memory Usage to keep all data in Memory

Slide 28

#2: On individual connections 12444! individual connections Classical N+1 Query Problem Individual SQL really <1ms

Slide 29

#3: Putting all data in temp Hashtable Lots of time spent in Hashtable.get Called from their Entity Objects

Slide 30

Custom Measuring Was impacted by Garbage Collection Just measured overall time but not # SQL Executions Learn SQL and don’t use Hashtables as Workaround Lesson Learned void roomreservationReport(int roomid) { long startTime = System.currentTimeMillis(); Object data = loadDataForRoom(roomid); long dataLoadTime = System.currentTimeMillis() - startTime; generateReport(data, roomid); }

Slide 31

Slide 32

Implementation Flaws

Slide 33

Business Impact requires Action!

Slide 34

Solution: Cache to the RESCUE!!

Slide 35

Implementation and Rollout Implemented InMemory Cache Worked well in Load Testing

Slide 36

Result: Out of Memory Crashes!! Still crashes Problem fixed! Fixed Version Deployed

Slide 37

Slide 38

Disconnected Teams

Slide 39

“Teamwork” between Dev and Ops SEV1 Problem in Production Need access to log files Where are they? Can’t get them Need to increase log level Can’t do! Can’t change config files in prod!

Slide 40

Solution: Implement a Custom “On Demand” Remote Logger

Slide 41

Implementation and Rollout Implemented Custom Logger Worked well in Load Testing

Slide 42

What happened? ~ 1Mio Lock Exceptions in 30 mins

Slide 43

Root Cause: A special WebSphere Setting!

Slide 44

Slide 45

“Deployment” Gone Bad!

Slide 46

Test Environment Production Environment 8x slower 3x more SQL

Slide 47

Test Environment Production Environment Hibernate, Classloading, XML – The Key Hotspots Hibernate, Classloading, XML – The Key Hotspots I/O for Web Requests doesn’t even show up! That’s Normal: Having I/O for Web Request as main contributor

Slide 48

Top Contributor Class.getInterfaces Called from Hibernates FieldInterceptionHelper These calls all originate form thousands of calls to find item by code

Slide 49

Top Methods related to XML Processing Classloading is triggered through CustomMonnkey and the Xalan Parser Classloading is triggered through CustomMonkey and the Xalan Parser

Slide 50

Slide 51

#Push without a Plan

Slide 52

Mobile Landing Page of Super Bowl Ad 434 Resources in total on that page: 230 JPEGs, 75 PNGs, 50 GIFs, … Total size of ~ 20MB

Slide 53

Fifa.com during Worldcup http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/

Slide 54

Waiting for Bug Reports

Slide 55

Look behind the scenes

Slide 56

Slide 57

Performance Clinic: Bring Your Own App When: Thursday, December 18th Lunch Session: NOON – 3PM After Work Session: 4PM – 7PM Where: Dynatrace Office 404 Wyman Street, Waltham Register: http://bit.ly/onlineperfclinic

Slide 58

Quick Demo

Slide 59

Get it here: http://bit.ly/dttrial YouTube Tutorials: http://bit.ly/dttutorials Live Q&A Sessions: http://bit.ly/onlineperfclinic Contact me: agrabner@dynatrace.com - @grabnerandi Special Offer: SaaS VIP Program: http://bit.ly/dtaasvip 30 Days Dynatrace Free Trial

Slide 60

Andreas Grabner Dynatrace Developer Advocate @grabnerandi http://blog.dynatrace.com