Requirements elicitation, writing, analysis, and modeling by IIBA Endorsed Education Provider.

Software Testing: The Basics of the Trade

Summary

TOC

Look Inside

Comments

Reviews

Gaia Asher

September 2005, Galiel.Net, Paperback, 175 pages, ISBN 0977036405


How to Test an Application using Business Requirements
How to Plan, Prepare, and Manage Acceptance Testing
How to Find and Build Test Cases from Requirements
	Try our new Virtual Workshops and e-Coaching for today's Business System Analysts (BA's) and Subject Matter Experts (SME's)

In many companies, software testers are simply considered software developers with a lower pay. However, that’s not the case. While it is true that software testing involves a lot of programming, it also requires a significantly different mind set and skills. Let’s illustrate this with a story:

Dark Ages. First, the developer arrives at the site and starts to plan the castle. He thinks, "Let’s put it on top of that hill, it’s the highest and provides a good, defendable position. It will also border with a river and so the defenders will always have water. Walls? Five meters thick and enforced concrete should do the trick. Gates… steel, half a meter thick. Good."

Then the test team arrives on horses in a full armor, and the test lead starts to look at the castle. His thoughts are a bit different, "Ok, the position is strong, cannot take it over with a direct attack. What about the tower? Ladders will not work, too tall… May be walls? Enforced concrete, five meters thick, cannot use a ram. No, no weakness here. Gates… hm-m-m… Steel, half a meter thick, cannot break through. By the way, did they lock them?"

This book is devoted to software testing as a trade, something that people do for a living. And the primary thing that separates the trade and the art is that you cannot limit yourself to a single technique, area, or dimension of your trade. Van Gogh could paint some of his pictures in blue. It was later called a "blue period" and these pictures are in the best museums of the world, because it was art. Software testers have to use all of the spectrum and keep in mind all the dimensions of software testing, because it’s their trade.

A complete up-to-date picture of the dimensions of software testing is what this book is about.

1 PREFACE ____________________________________ 3
2 AT A GLANCE ________________________________ 5
3 TABLE OF CONTENTS ________________________ 9
4 INTRODUCTION _____________________________ 17
5 WHY TEST SOFTWARE? _____________________ 19
5.1 WHY TEST SOFTWARE? ______________________ 19
5.2 LIMITATIONS OF TESTING ____________________ 20
5.2.1 The First Law of Software Programming______ 20
5.2.2 Nothing is ever perfect ____________________ 21
5.2.2.1 Requirements _______________________ 22
5.2.2.2 Design ____________________________ 23
5.2.2.2.1 Requirements context, missed or ignored
requirements ______________________________ 24
5.2.2.2.2 Overusing formal verification tools
without proper qualification __________________ 27
5.2.2.2.3 Late design changes and code fixes ___ 29
5.2.2.3 Code______________________________ 29
5.2.2.4 Platform ___________________________ 30
5.2.2.5 Tests______________________________ 30
5.2.2.6 Users _____________________________ 31
5.2.2.6.1 Monkey_________________________ 32
10
5.2.2.6.2 Farmer__________________________ 33
5.2.2.6.3 Hacker__________________________ 34
5.2.2.7 The Product Team ___________________ 34
5.2.3 Richness of possibilities ___________________ 34
5.2.3.1 Code paths, state transitions____________ 35
5.2.3.2 Inputs (including input editing and timing) 36
5.2.3.3 Data in the environment_______________ 37
5.2.3.4 In essence – you cannot try everything ___ 38
5.3 ALTERNATIVE TO TESTING – PROVING __________ 38
5.4 THREE OBJECTIVES OF TESTING SOFTWARE ______ 39
5.4.1 So, why? _______________________________ 39
5.4.2 Objective #1: Quality Bar__________________ 39
5.4.3 Objective #2: Acceptance __________________ 41
5.4.4 Objective #3: Baseline ____________________ 41
6 WHAT’S SOFTWARE TESTING ABOUT? _______ 43
6.1 THE DIFFERENCE BETWEEN A TESTER AND A
DEVELOPER _____________________________________ 43
6.1.1 A Castle _______________________________ 43
6.1.2 Debugging is not testing___________________ 44
6.1.2.1 Goals _____________________________ 44
6.1.2.2 Actor _____________________________ 45
6.1.2.3 Tools _____________________________ 45
6.1.2.4 Results ____________________________ 45
6.1.3 Super-pessimism – Testing that the glass is half
empty 45
6.2 THE THREE-HEADED DRAGON OF SOFTWARE
DEVELOPMENT ___________________________________ 46
6.2.1 Software developer _______________________ 46
6.2.2 Software architect, analyst, project manager___ 47
6.2.3 Software tester __________________________ 48
6.3 THE LANGUAGE OF SOFTWARE TESTING ________ 48
6.3.1 The bug: what is it? ______________________ 48
6.3.2 Test case _______________________________ 49
6.3.3 Test plan _______________________________ 50
6.3.4 Bug tracking ____________________________ 50
6.3.5 Version control __________________________ 51
6.3.6 Test automation _________________________ 53
6.4 A BUG’S LIFE _____________________________ 54
11
6.4.1 Bug opened _____________________________ 54
6.4.2 Bug approved ___________________________ 55
6.4.3 Bug fixed_______________________________ 55
6.4.4 Bug verified and closed ___________________ 56
6.5 SEVERITY OF BUGS _________________________ 56
6.5.1 Know Thine User!________________________ 57
6.6 THE KINDS OF BUGS ________________________ 59
6.6.1 Code defect _____________________________ 59
6.6.2 Crash _________________________________ 60
6.6.3 A special kind: Regression _________________ 61
6.6.4 Security ________________________________ 61
6.6.5 Performance-related______________________ 62
6.6.5.1 Throughput (performance)_____________ 62
6.6.5.2 Stress _____________________________ 63
6.6.5.3 Volume____________________________ 64
6.6.5.4 Scalability _________________________ 64
6.6.5.5 Resources consumption (leaks) _________ 65
6.6.6 Usability _______________________________ 67
6.6.7 Documentation __________________________ 69
6.6.8 Test “issue” ____________________________ 70
6.6.8.1 Bugs in tests and frameworks __________ 70
6.6.8.2 Mixing up a bug and a feature __________ 70
7 STAGES OF SOFTWARE DEVELOPMENT FOR A
TESTER _________________________________________ 72
7.1 CHALLENGING REQUIREMENTS________________ 72
7.2 CHALLENGING SPECIFICATIONS _______________ 73
7.2.1 Challenging assumptions __________________ 74
7.2.2 Challenging completeness _________________ 75
7.2.3 Challenging behavior _____________________ 77
7.2.4 Challenging usability _____________________ 78
7.2.5 Challenging testability ____________________ 78
7.2.6 That’s not the end ________________________ 79
7.3 CHALLENGING ARCHITECTURE AND TECHNICAL
DESIGN 79
7.4 TESTING CODE_____________________________ 81
7.5 CHALLENGING DOCUMENTATION ______________ 81
7.6 RELEASE _________________________________ 83
8 DIMENSIONS OF SOFTWARE TESTING _______ 85
12
8.1 WHY “DIMENSIONS”? _______________________ 85
8.2 BY OBJECT _______________________________ 86
8.2.1 Positive testing – capabilities _______________ 86
8.2.2 Negative testing – environment _____________ 86
8.3 BY SOURCE CODE VISIBILITY_________________ 86
8.3.1 Black (“opaque”) box testing_______________ 86
8.3.2 White (glass, clear) box testing _____________ 87
8.3.3 “Gray” (“Translucent”) box testing _________ 88
8.4 BY LEVEL ________________________________ 88
8.4.1 Unit testing _____________________________ 88
8.4.2 Component testing _______________________ 89
8.4.3 Integration testing________________________ 90
8.4.4 System testing ___________________________ 91
8.4.4.1 Scenario-based ______________________ 92
8.4.4.2 Free-form __________________________ 92
8.5 BY PLACE ________________________________ 93
8.5.1 Product Team Testing (vendor) _____________ 93
8.5.2 Customer (user) Side Testing _______________ 93
8.5.3 Third Party Testing (certification) ___________ 94
8.6 BY PURPOSE ______________________________ 95
8.6.1 Functional testing________________________ 95
8.6.2 Regression testing________________________ 96
8.6.3 Performance related testing ________________ 97
8.6.3.1 Performance (throughput) testing _______ 97
8.6.3.2 Volume (size) testing _________________ 97
8.6.3.3 Load (stress) testing __________________ 98
8.6.3.4 Scalability testing____________________ 98
8.6.3.5 Resource consumption (including leaks)
testing 98
8.6.3.6 Sizing guidance _____________________ 99
8.6.4 Conformance testing______________________ 99
8.6.4.1 Standards _________________________ 100
8.6.4.2 Legislation ________________________ 100
8.6.4.3 Coding policy testing ________________ 100
8.6.5 Security testing _________________________ 100
8.6.5.1 Functional security testing ____________ 100
8.6.5.2 Security model and STRIDE __________ 101
8.6.5.3 Security penetration testing ___________ 103
8.6.6 Globalization testing_____________________ 103
13
8.6.6.1 Encoding _________________________ 103
8.6.6.2 Watch your language ________________ 104
8.6.6.3 Formats __________________________ 106
8.6.6.4 More calendar fun __________________ 106
8.6.6.5 Units_____________________________ 107
8.6.6.6 Strings ___________________________ 108
8.6.6.7 Length of text______________________ 108
8.6.6.8 Making of an error message___________ 109
8.6.6.9 That’s not all ______________________ 110
8.6.7 Localization testing______________________ 110
8.6.7.1 Cultural differences _________________ 111
8.6.8 Platform and dependencies testing__________ 112
8.6.9 Pseudo-testing _________________________ 114
8.6.9.1 Usability__________________________ 114
8.6.9.2 Reliability_________________________ 115
8.6.9.3 Robustness ________________________ 115
8.7 BY ATTACK SURFACE______________________ 116
8.7.1 What is an attack surface _________________ 116
8.7.2 Input _________________________________ 117
8.7.2.1 Input – GUI _______________________ 117
8.7.2.2 Input – API________________________ 118
8.7.2.3 Input – Files and DB content __________ 119
8.7.3 Environment ___________________________ 119
8.7.3.1 Environment – Resources (storage, memory,
etc.) 119
8.7.3.2 Environment – Access rights __________ 120
8.7.3.3 Environment – Network______________ 120
8.7.3.4 Environment – Files and folders _______ 120
8.7.3.5 Environment – Media _______________ 121
8.7.3.6 Environment – Output devices_________ 121
8.7.3.7 Environment – Platform______________ 122
8.7.4 User _________________________________ 122
8.7.4.1 User – monkey _____________________ 122
8.7.4.2 User – farmer ______________________ 123
8.7.4.3 User – hacker ______________________ 123
8.8 BY FAULT TYPE __________________________ 123
8.8.1 A word on the fault model_________________ 123
8.8.2 Boundary conditions_____________________ 124
8.8.3 Arithmetic overflows_____________________ 125
14
8.8.4 Buffer overflows ________________________ 126
8.8.5 Race conditions_________________________ 127
8.8.6 Uninitialized variables ___________________ 129
8.8.7 Uncaught exceptions_____________________ 130
8.8.8 Resource allocation _____________________ 130
8.8.9 That’s not all again______________________ 130
9 QUALITY OF TESTING ______________________ 131
9.1 HUH? ___________________________________ 131
9.2 CODE COVERAGE__________________________ 131
9.3 SOFTWARE METRICS _______________________ 132
9.4 TEST PLAN REVIEW ________________________ 133
9.5 SEEDING BUGS AND CODE MUTATION__________ 134
9.6 TESTABILITY _____________________________ 134
9.7 BETA RELEASE/BETA TESTING _______________ 136
10 POST-RELEASE FUN ________________________ 138
10.1 DEPLOYMENT TESTING _____________________ 139
10.2 COMPATIBILITY TESTING ___________________ 139
10.3 SYSTEM TESTING__________________________ 139
10.4 STAGING AND PRODUCTION ENVIRONMENT _____ 140
11 TESTING SOFTWARE IN A GLOBAL TEAM ___ 141
11.1 TECHNICALLY SPEAKING ___________________ 141
11.2 HUMAN-ORIENTED ________________________ 142
11.2.1 Time zone differences. _________________ 142
11.2.2 Work relations and cultural differences____ 142
11.2.3 Life in a global org____________________ 143
12 SURVIVING AS A TESTER ___________________ 145
12.1 FUZZY VALUE IN THE SOFTWARE DEVELOPMENT_ 145
12.2 COST/VALUE BALANCE _____________________ 146
12.3 RELIGIONS AND CONFESSIONS _______________ 147
12.4 IEEE ROLE ______________________________ 151
12.5 OFFICE POLITICS __________________________ 151
12.6 OUTSOURCING AND JOB SECURITY ____________ 154
12.7 SEGREGATION ____________________________ 157
12.8 FUTURE OF THE TRADE _____________________ 158
12.8.1 From nobility to peasants_______________ 158
15
12.8.2 Crushing craftsmen ___________________ 161
12.8.3 Assault on software development _________ 162
12.8.4 What’s next?_________________________ 165
13 APPENDIX A. TEST CASE____________________ 168
14 APPENDIX B. BIBLIOGRAPHY _______________ 170
14.1 STANDARDS AND CLASSICS__________________ 170
14.2 EXTREME TESTING & RELATED ______________ 171
14.3 OTHER__________________________________ 172
15 ENDNOTES _________________________________ 174

Review-Date: 2/7/2011 Rating: 1 Summary: Poorly Written, Poor Attitude

I was reviewing this book to see if it would be something worth advising my employees to read. I was looking for something short and to the point that would cover briefly cover all aspects of Test Engineering without offering opinions into which methods work and don‘t work. The only part of the book you really get to read in the 10 page preview is the introduction, the author makes it a point to tell you that he is writing a book free of opinions & gets to the heart of what Test Engineering really is.

This is why I decided to give the book a shot, unfortunately the author is unable to keep to his own goal. Right from the start he is trying to convince you that his book is worth reading, explaining why other books are no good &/or not worth the money. He continues this kind of attitude through at least the end of chapter 2 (for that is as far as I was able to read). He bashed on the programming skills of people from foreign countries and pretty much bashes on anyone who isn‘t him.
This book is a complete waste of time & I would advise anyone wanting to learn about Test Engineering to look elsewhere. There is nothing to learn here except how to have a poor attitude which is not a skill anyone hoping to find a job want to have.