'

Learn the Best Practices in Mobile Game Testing

Понравилась презентация – покажи это...





Слайд 0

2 December 2014 Ville-Veikko Helppi Technical Product Manager ville-veikko.helppi@bitbar.com Learn the Best Practices in Mobile Game Testing W E B I N A R


Слайд 1

Agenda Different Types of Testing and How Game Testing Differs from Regular App Testing? Frameworks, Tools and Technologies – What is Important in Mobile Game Testing Today? Top Priorities for Mobile Game Testing Infrastructure for Successful Game Testing Hands-on Example (Image Recognition) Testdroid Update Q&A


Слайд 2

Let's Define 'Testing'...


Слайд 3

Different Types of Testing? Black-box testing vs. White-box testing Methods used in Mobile Game Testing Functional Compatibility Performance Localization Regression Load Smoke Acceptance


Слайд 4

Defining Some Ways to Test Feature-based testing Performance testing End-user testing


Слайд 5

The Old-Fashion Thinking Deterministic systems Identifiable objects Tool support(s) User-driven actions Simple inputs/outputs Virtualization Random setups/results No handle on objects No standards User reacts to input Complex inputs/outputs Real hardware <> <> <> <> <> <> High-level of automation Low-level of automation X


Слайд 6

Test Automation as a Solution Random setups/results -> Easy to generate random events No handle on objects -> Variety of options (e.g. image recognition) No standards -> Variety of frameworks (cross-platform etc.) User reacts to input -> Test scripts can handle random events Complex inputs/outputs -> Test scripts can generate input/content Real hardware -> Availability of real mobile hardware Test Automation brings real value with variety of options to mobile game testing


Слайд 7

Challenges that Test Automation Tackles Proliferation of Devices, Brands, Operators, Platforms Coupling the Testing and Development process (Continuous Integration/Deployment/Delivery) Different Types of Testing (as automated) Stress, Load, Performance, Functionality etc. Test script & test asset usability Cons of Manual testing Tedious effort, Expensive human resources, Error-prone (as done by humans) & Not reusable (done once)


Слайд 8

OS versions Chipsets CPU + GPU Tens of OEMs Memory Displays (resolutions, physical hw) OEM mods Other hardware (connectivity, calibration) Relation to other software Significant Benefits of Using Automation


Слайд 9

Correct behaviour across platforms and browsers Integration with web back-ends Typically need to fully utilize HW (CPU+GPU) Resource (e.g. battery) consumption OpenGL ES 2/3 Functionality and usability Screen orientations, connectivity, user profiles Robustness Robustness and security! Brand Compliances, verification with back-ends and data Different 'Verticals' Require Different Practices


Слайд 10

Importance of Mobile Game Testing


Слайд 11

Important Aspects in Game Testing User Interface and Functionality UI layouts/elements, Menus, Resolution, Orientation, Overall visuality Graphics Performance Usability and User Experience Responsiveness, Real Occuring Events Multi-player/User Social Integrations Security and Liabilities


Слайд 12

User Interface and Functional Compared to regular app testing, mobile game testing may seem to be more challenging e.g. OpenGL ES content not action-identifiable Graphics and UI – Identical look&feel across all devices and variants Usability controls in the context of game Navigation/game-play, Progress, Delays between "game-stages" Animations, Fonts, Other graphics assets Screen Resolution and Orientation


Слайд 13

Performance Testing Performance, Load, Stress, Spike, Soak Measure the real user experience Execute tests with all possible devices (SW&HW) Discover the impact of server loads, app performance (CPU/GPU utilization)


Слайд 14

Studies: Majority of users leave game due sluggish, general slowness or other bad performance behavior It's very important to understand how game behaves on different devices, under different circumstances Performance Makes +/- UX


Слайд 15

Building the User Experience Understanding How quickly gamer assimilates what the game is about, how to fully play with it, what are the functionalities. Usability The main User Interface, menus, design and how usable those are Playing Experience Intuition to follow the game and go forward with its instructions and game-play Monetization Are gamers willing to buy – invest – in game Graphics performance capability is the most important thing for mobile games when forming UX Too many games fail to work properly with low-end/mid-range devices due hardware performance The real understanding of game's performance capability can be only figured out by testing on real devices


Слайд 16

End-to-end between tests, AI or more simple implementations Optimize the data communication between server and clients Test simultaneously on several (different) devices Check what happens if game gets interrupted (platform) Location-based features – how to properly mock up those? Hermetic environment – what happens? Multi-Player/User Features


Слайд 17

Important part of the after-engagement (how gamers share their results) Many of OEM branded devices provide certain SoMe apps pre-installed (version?) Connectivity, login and data transferring are tested in context of game Hermetic environment – what happens? Social Integrations


Слайд 18

Security and Liabilities


Слайд 19

4 Top Priorities in Mobile Game Testing


Слайд 20

Localization


Слайд 21

Device Diversity By OpenSignal, Inc. 2014 - http://opensignal.com/reports/2014/android-fragmentation/


Слайд 22

Real Devices vs. Emulators Emulators/simulators won’t help you to test… User Experience and Usability Hardware Software Infrastructure 0 % = the percentage of your app users that use emulator to run your app! Why Real Devices are Must-to-Have?


Слайд 23

Manual vs. Automation Smaller coverage, More money burnt & time wasted, Error-prone Large coverage, quickly completed, Less money & time wasted, Exact results. Manual Automation


Слайд 24

Infrastructure for Mobile Game Testing


Слайд 25

High-Level Infrastructure


Слайд 26

Architecture / Infrastructure GAME Game Engine Android iOS ... API & services Server Database App Middleware Platform Connectivity/Network Back-end


Слайд 27

Isolated, Hermetic Testing GAME Game Engine Android iOS ... API & services Server Database Hermetic Environment App Middleware Platform Connectivity/Network Back-end


Слайд 28

Compatibility & Porting Testing GAME Game Engine Android iOS ... API & services Server Database Real Devices App Middleware Platform Connectivity/Network Back-end


Слайд 29

Smoke/Reference Testing GAME Game Engine Android iOS ... API & services Server Database Reference Environment App Middleware Platform Connectivity/Network Back-end


Слайд 30

Back-End Performance Testing GAME Game Engine Android iOS ... API & services Server Database Massive no. of users and significant load App Middleware Platform Connectivity/Network Back-end


Слайд 31

Continuous Integration


Слайд 32

Game Testing Process


Слайд 33

Hands-on Example: Integrating Development with Testing Environment


Слайд 34

Continuous Integration (e.g. Jenkins) Appium framework (for Android and iOS) No need for devices – we'll use devices on Testdroid Cloud service Test scripts / test cases for game Setup


Слайд 35

Continuous Integration


Слайд 36

Appium Intro Family Tree: Android Test Automation Frameworks Appium works for both, Android and iOS


Слайд 37

© Copyrights by Bitbar Technologies Ltd. 2013 All rights reserved. 38 Uses Selenium Webdriver (W3C standard) as a scripting framework Supports native Android, native iOS and mobile web: Android via uiautomator (API level >=16) and Selendroid (API level <16) iOS via UI Automation Mobile web as Selenium driver for Android and iOS You can write your Appium scripts on almost any programming language (Haskell/Go/Clojure/Java/Ruby) Appium Basics


Слайд 38

Appium @ Localhost © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 39 Test Script Test Case Desired Capabilities Desired Capabilities / Android Example


Слайд 39

Appium @ Localhost © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 40 Test Script Test Case Desired Capabilities Example Desired Capabilities / iOS


Слайд 40

Appium @ Localhost © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 41 Test Script Test Case Desired Capabilities Appium @ Localhost WebDriver http://localhost_4723/wd/hub Appium Server 4723 Device Example


Слайд 41

Appium @ Localhost © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. Test Script Test Case Desired Capabilities From Localhost to Testdroid Cloud WebDriver http://localhost_4723/wd/hub *Testdroid Caps http://appium.testdroid.com/wd/hub Example


Слайд 42

Appium @ Localhost © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. Test Script Test Case Desired Capabilities From Localhost to Testdroid Cloud WebDriver { “testdroid_username”: “user@domain.com”, “testdroid_password”: “p4s$w0rd”, “testdroid_project”: “My First Project”, “testdroid_testrun”: “Test 1”, “testdroid_device”: “iPad Mini 7.0.4 A1432”, “testdroid_app”: “http://domain.com/app_v1.ipa” . . “app”: “com.bitbar.testdroid.BitbarIOSSample” } Example


Слайд 43

Projects and Test Runs © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 44 Example


Слайд 44

350+ Android & 20+ iOS Devices


Слайд 45

Appium @ Testdroid © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 46 Example


Слайд 46

Appium @ Testdroid © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 47 Example


Слайд 47

Appium @ Testdroid © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 48 Example


Слайд 48

Appium @ Testdroid © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 49 Example


Слайд 49

Appium @ Testdroid © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 50 Example


Слайд 50

Running tests on multiple devices © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 51 Example


Слайд 51

Example Projects and Test Runs


Слайд 52

Appium Broker Appium @ Localhost © Copyrights by Bitbar Technologies Ltd. 2014 All rights reserved. 53 Test Script Behind the Scene Device Cluster Device 1 Device 2 Device 3 Session Map Proxy Appium Device 1 Example


Слайд 53

Image Recognition The basic example of Image Recognition in Mobile Game Testing: Template Matching


Слайд 54

Code Example


Слайд 55

Code Example


Слайд 56

Code Example


Слайд 57

Code Example


Слайд 58

Test Script contains the following 'steps': Identify the used platform (Android / iOS) Search for 'Goldmine' graphical element (using find_image to search for screen) Script runs test to go to shop Script runs test to buy a cannon Script runs test to place the cannon (all these three are defined in .png files) Script runs test to start a battle! Game is brought down and test ends. Example: Clash of Clans


Слайд 59

VIDEO Image Recognition with Real Android and iOS Devices in Local Environment


Слайд 60

Testdroid Update


Слайд 61

New UI in Testdroid Cloud 2.0


Слайд 62

Testdroid Cloud's Interactive


Слайд 63

Testdroid PrivateCloud - Running user-selected devices at any of our hosting facilities


Слайд 64

NEW HOSTING CENTER! Mountain View, CA San Francisco and Mountain View datacenter


Слайд 65

Option 1: Public Device Cloud on-demand devices (multitenant) Mobile app testing on over 400+ real Android and iOS devices hosted by Bitbar Option 2: Private Device Cloud reserved devices Hosted by Bitbar in the US and/or Europe Devices chosen by and reserved only for the Customer Option 3: On-premise Device Cloud Automated mobile app testing on real Android and iOS de- vices hosted by the customer, usually 30-500 devices Testdroid Deployment Options


Слайд 66


Слайд 67

Video recording & slides of this webinar will be soon available at Vimeo and Slideshare! Please visit at www.testdroid.com for more information. THANK YOU! W E B I N A R


Слайд 68

We operate the largest global device cloud with over 400 devices instantly available for developers Devices from the US, Europe, China, Japan and Korea – global devices for global customers


×

HTML:





Ссылка: