SWE 637: Here! Test this! - George Mason University

SWE 637: Here! Test this! - George Mason University

Generating Automated Tests from Behavior Models Jeff Offutt SWE 737 Advanced Software Testing OUTLINE 1. 2. 3. 4. SWE 737 Overview of Model-Based Testing The Mapping Problem The Test Oracle Problem Summary & Conclusions Jeff Offutt 2 Benefits of Model-Based Testing Better tests Lower cost Early detection of requirements defects Traceabilit Software y evolution

Less overlap among tests SWE 737 Jeff Offutt 3 Model-Based Testing Process Mod el Criterio n Lots Test criteria Creating models Executing tests Test Requireme nts Abstrac t Tests Concret e Tests Test Execution SWE 737

Test Reports of research and tools for: Extra Info Test Oracle Not much known about: Converting abstract tests to concrete tests Writing effective test oracles These are normally done by hand Jeff Offutt 4 OUTLINE 1. 2. 3. 4. SWE 737

Overview of Model-Based Testing The Mapping Problem The Test Oracle Problem Summary & Conclusions Jeff Offutt 5 The Mapping Problem Transform abstract tests to concrete tests An abstract test: [1, 3, 4, 1, 2, 4] AddChoc, Coin, GetChoc, Coin, AddChoc Nine edge coverage abstract tests use : 15 AddChoc transitions 16 Coin transitions 6 getChoc transitions Coin 1 Coin

2 GetChoc AddChoc AddChoc Coin 3 GetChoc 4 AddChoc / Coin AddChoc Testers usually convert abstract 1: credit = 0 & #stock = 0 to concrete tests by hand 2: credit > 0 & #stock = 0 Solution? 3: credit = 0 & #stock > 0 4: credit > 0 & #stock > 0 Semi-automate the SWE 737 transformation Jeff Offutt

6 Model-Based Test Components Test components Test 1 Test 2 C1 C2 C3 C4 C5 C6 C7 SWE 737 ct a r t bs nent a h o Eac omp e t c st b s al e e

t u r m to ete d ppe concr a m in ts e d co tes Test 3 many times Jeff Offutt 7 Solving the Mapping Problem A language to define mappings for individual test components Structured Test Automation Language (STAL) A tool to choose model-level test components

and compose mappings to create concrete tests Behavioral Models SWE 737 Identifiable Elements 1. Transitions 2. Objects 3. Constraints Jeff Offutt 8 Empirical Evaluation of STALE Can STALE help programmers efficiently design tests ? Subject Programs 17 open source and example programs 52 to 9486 LOC 7 GUIs, 4 web apps, 1 CL, 5 others Subject Models Participants 9 professional developers

/ graduate students SWE 737 Drawn using Eclipse / Papyrus 722 states 12116 transitions Jeff Offutt 9 STALE vs. Hand Test Generation Generating Tests by Hand 1. Write tests by hand 2. Ensure that constraints in models were satisfied 3. Compile and run the tests Generating Test with STALE 1. Import model and program into STALE 2. Create test component mappings by hand 3. STALE creates concrete tests to satisfy transition (edge) coverage Measurements Time Number of errors in tests SWE 737 Jeff Offutt

10 Mapping Problem Results Time Ratio = Time using STALE / Time with manual generation 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 M ack dar ess gic age rser age nes age age oly ake Toe ree gle ine T i er er P Sn ac r T ian ch A ckJ len Ch sLo er Pa e m v v v v a J o r s o c

T a Ca i l o o T c o i M r B T C DFC am phC icC IMC ng i g n a Lo M nd Dy Gr e V 48 errors with manual approach, 0 with STALE SWE 737 Jeff Offutt

11 OUTLINE 1. 2. 3. 4. SWE 737 Overview of Model-Based Testing The Mapping Problem The Test Oracle Problem Summary & Conclusions Jeff Offutt 12 RIPR Model Reachabilit y Infection Propagation Revealabilit y SWE 737 Test Final Program State

Reache s Observed Final Program IncorrectState Incorrect Final Final State State Fault Infect s Incorrec t Program State Propagat es Reveal s Test Oracles Jeff Offutt 13

Test Oracle Problem An automated test must check whether the behavior was correct (assertions in Junit) How much of the program state should be checked ? (That is, which variables ?) More checking adds more cost Method return values ? Method parameters ? Global variables ? How often should state be checked ? SWE 737 Once at end of test ? After every transition Jeff ? Offutt 14 Test Oracle Strategies f r After each transition e q u

Requires a nalysis by e Once at end of test tester n c y precision NOS null (crash) SIOS state invariant (SI) Free SWE 737 OS1 SI member objects OS2 SI returns OS3 SI objects

returns OS4 SI params OS5 SI objects returns params Can be created from mappings Jeff Offutt 15 Empirical Evaluation of Test Oracle Do more precise OSes reveal more Strategies failures ? Do more frequent checks reveal more failures ? Which OS balances cost and Subjects ? statecharts effectiveness 16 Java programs & UML 24 test sets for each OS for edge coverage 9627 faults (mutants) generated using

muJava Experimental Variables Independent : 13 oracle strategies Dependent : 1. Cost (number of assertions) 2. Faults revealed SWE 737 Jeff Offutt 16 Precision% Faults Revealed 8,881,000 tests executed (12 OSes * 250 tests * 9627 faults) Edge Coverage Tests 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 NOS

SIOS OS1 OS2 OS3 OS5 OS6 More precise OSes are not always more effective One-tailed Wilcoxon signed-rank test SWE 737 Jeff Offutt 17 Frequency% Faults Revealed Checking after each transition vs. checking at the end of the test Multiple Once 1 2

3 4 5 Multiple checks is not significantly more effective SWE 737 Jeff Offutt 18 Test Oracle Strategy Findings NOS is uselesswastes much of testing effort The most cost-effective choice is to check the state invariants once at the end of the tests (SIOS) SWE 737 Only one check needed Can be partially derived automatically from the model Jeff Offutt

19 OUTLINE 1. 2. 3. 4. SWE 737 Overview of Model-Based Testing The Mapping Problem The Test Oracle Problem Summary & Conclusions Jeff Offutt 20 Contributions in Context Mod el Criteri a Test Requireme nts Abstrac t Tests Concret

e Tests Test Execution SWE 737 Test Reports Extra Info Test Oracle A language like STAL can significantly ease the transformation of abstract tests to concrete tests Test oracles are effective with modest, but not zero, cost Jeff Offutt 21 Summary of Contributions Algorithms for computing efficient test paths on models

Language to partially automate mappings from abstract tests to concrete tests Based on assembling test components Demonstrated value of STAL over manual test automation Determined cost-effective test oracle strategy Extended classic RIP model to RIPR SWE 737 Jeff Offutt 22 References 1. 2. 3.

4. SWE 737 Software Test Automation Practices in Agile Development Environment: An Industry Experience Report, Eliane Figueiredo Collins and Vicente Ferreira de Lucena, Jr., 7th International Workshop on Automation of Software Test, Pages 57-63, Zurich, Switzerland, June 2012 (Eric) Test Automation in the Real World2016, (results from a survey monkey poll) New survey reveals test automation trends, Cecilia Rehn, Software Testing News August 2015 (summary of a longer article behind a pay wall) Test Oracle Strategies for Model-based Testing, Nan Li and Jeff Offutt, IEEE Transactions on Software Engineering, To appear, 2017 Jeff Offutt 23

Recently Viewed Presentations

  • Muscles

    Muscles

    "Whenever a particular agonist and its synergist are excited and contract, the antagonist is inhibited and relaxed." This occurs all because of the nervous system. Can you contract an agonist and antagonist at the same time? What would happen? Body-lock...
  • Principals of War and Forms of Maneuver

    Principals of War and Forms of Maneuver

    General Military Strategic, Doctrinal, Operational, and Leadership Concepts Agenda Key Theorists Principles of War Facets of the Operational Art Forms of Maneuver METT-TC Strategy Strategic Leadership Key Theorists Clausewitz Jomini Clausewitz Carl von Clausewitz Prussian officer born in 1780 Resigned...
  • Chapter 11 Complexities of Liberalism in Practice To

    Chapter 11 Complexities of Liberalism in Practice To

    Illiberalism. An illiberal democracy, also called a pseudo democracy, partial democracy, low intensity democracy, empty democracy, hybrid regime or delegative democracy, is a governing system in which, although elections take place, citizens are cut off from knowledge about the activities...
  • Recent changes to Pensions - Home | Local Government Association

    Recent changes to Pensions - Home | Local Government Association

    Plus unused AA from the 2012/13, 2013/14, or 2014/15 tax years, which can still be carried forward, as usual. ... "The government is currently minded not to include those individuals with protected TUPE terms in scope of the cap. ...
  • Distributed Systems

    Distributed Systems

    1 Overview. The ubiquitous computing vision (Mark Weiser, 1993) "Ubiquitous computing enhances computer use by making many computers available throughout the physical environment, but making them effectively invisible to the user."
  • April 12, 2013

    April 12, 2013

    Mix-Pair-Share. Using your bell ringer journal: Find a partner: Discuss your situation you wrote about in your journal. Predict how this topic will relate to Animal Farm. Thank your partner, find a new partner, and discuss again. In the News.
  • smartfile.s3.amazonaws.com

    smartfile.s3.amazonaws.com

    Macbeth. and . An Inspector Calls. Paper 2: Great Expectations, Conflict poetry and unseen poetry. English Language. Paper 1: Fiction and Imaginative writing. ... Mind-map themes. Mind-map characters. Breakdown of each exam.
  • Chemical Bonding - cwcboe.org

    Chemical Bonding - cwcboe.org

    Chemical Bonding Chapter 8 Polarity and Dipole moments Review Order the following bonds according to polarity: H-H, O-H, Cl-H, S-H, and F-H. H-H S-H Cl-H O-H F-H Bond Polarity and Dipole Moments Molecules with a charge distribution of a positive...