Software Testing Techniques: Finding the Defects that Matter (Programming Series) |
|
|
|
| Scott Loveland, Michael Shannon, Geoffrey Miller, Richard, Jr. Prewitt |
| October 2004, Charles River Media, Paperback, 362 pages, ISBN 1584503467
|
|
|
|
 |
|
| Reliability and continuous availability have become crucial for
computer software in the information economy. Well-publicized failures
from both industry and government have underscored the need for mission-critical
software. Today's software must be thoroughly tested before being
released into the marketplace. When it comes to industrial-strength
reliability, no computer system is more renowned than the mainframe.
But for years the actual approaches used to test mainframe software
have been kept under wraps. What techniques are used? How do the teams
in the elite mainframe development labs test their systems to ensure
they will deliver such extreme levels of robustness? How can these
lessons be applied to other software projects?
Real-World Software Testing answers these questions and provides
practical testing techniques for achieving robust reliability with
large, complex software projects. Written in a friendly, accessible
style, the book highlights areas such as devising attacks to validate
data integrity, wreaking havoc to ensure recoverability, exploiting
virtualized environments to improve test efficiency, the tug of
war between finding defects and meeting schedules, and approaches
for emulating customer environments to find the most critical defects.
Focusing on real life techniques illustrated through examples and
case studies gleaned from the authors' experiences of mainframe
testing and operation, software engineers will know the tried and
true techniques that are used for testing multimillion- dollar commercial
software products.
KEY FEATURES
- Places an emphasis on system level testing, touching on aspects
that are relevant to large-scale projects, but focusing on the
system-wide testing that is critical to establishing the reliability
of enterprise-class software
- Provides practical testing techniques for achieving robust
reliability with large, complex software projects
- Focuses on real-world issues, problems, and solutions, not
abstract theories
- Emphasizes approaches for finding and removing the defects
that matter most to customers
- Highlights techniques for devising attacks to validate data
integrity, wreaking havoc to ensure recoverability, exploiting
virtualized environments to improve test efficiency, finding defects
and meeting schedules, and emulating customer environments to
find the most critical defects
|
 |
|
Notices v
Acknowledgments xix
Preface xxi
Part I Testing: Why Worry? 1
1 The Testing Challenge and Those who Take It On 3
Software Engineering Evolution 4
A Tester's True Goal 6
What Is a User? 7
Testers, What Makes Them Special? 9
Summary 12
2 Industrial-strength Software, It's Not a Science Project 13
Industrial-strength Software 14
Production Environments 15
Mission-critical Software 19
Case Study: A Pension Plan Management Vendor 20
Summary 24
Part II The Testing Ecosystem 25
3 The Development Process 27
Test Process Definitions 28
Traditional Software Development Models 43
Iterative Software Development Models 49
The "Let's Just Skip SVT" Fallacy 59
Summary 60
4 The Test and Development Divide 61
Should Developers Test Their Own Software? 62
Diplomacy: The Tester's Relationship with Development 64
Summary 76
Part III The Best-laid Plans 77
5 Where to Start? Snooping for Information 79
The Importance of Knowing What You Test 80
Viewing All Software as a Solution to a Problem 87
Customer Reconnaissance 87
A Simple Test Preparation Tool 90
Don't Just Take, Give a Little 92
Summary 93
6 Coping with Complexity through Teaming 95
Complex Software Products: You Can't Know It All 96
Reducing Complexity through Component Spies 98
Sharing Expertise Across the Team 104
Summary 106
7 Test Plan Focus Areas 107
The Test Plan Document 108
Unit Test Focus Areas 114
Function Verification Test Focus Areas 115
System Verification Test Focus Areas 122
Integration Test Focus Areas 130
Single-system versus Multisystem Testing 131
Test Plan Reviews 135
Summary 137
8 Testing for Recoverability 139
Function Verification Test 140
System Verification Test 144
Integration Test 146
Case Study: Clustered System Recovery 147
Summary 150
9 Planning for Trouble 151
Scheduling 152
Entry and Exit Criteria 156
Injecting Testability into Development Plans 161
Case Study: The Testability of Errors 163
Summary 166
Part IV Preparing for the Test 167
10 The Magic of Reuse 169
Who Writes More Code: Development or Test? 170
Reuse Opportunities and Techniques 170
Case Study: Testing Real Memory Management 174
Summary 176
11 Developing Good Test Programs 177
Function Verification Test Programs 178
System Verification Test Programs 184
Case Study: Task Termination Processing 185
Summary 188
12 Data Corruption 189
Data Integrity: What Is It? 190
Protecting against Data Corruption 191
The Need for Special Testing 192
Data Integrity Monitors 193
Case Study: Memory and File Thrashers 194
Summary 200
13 Tools-You Can't Build a House without Them 201
The Magic Tool Myth 202
Tool Categories 203
Case Study: A Homegrown Test Tool Framework 208
Buy versus Build 215
Avoid the Tool Merry-Go-Round 216
Summary 217
14 Emulating Customers to Find the Bugs that Matter 219
Determining What Tests to Run 220
It's Not Just What, but How You Test 227
Managing the Test Environments 232
Summary 234
Part V Executing the Test 235
15 Test Execution 237
Test Execution Sequence 238
Artistic Testing 240
An Iterative Approach: Algorithm Verification Testing 243
Catching Problems 248
Problem Diagnosis 250
Testing the Documentation 253
Summary 255
16 Testing with a Virtual Computer 257
Partitioning 258
Virtualization 261
Partitioning at Virtualization Combinations 265
Why Go Virtual? 265
Case Study: A Closer Look at z/VM 268
Summary 276
17 Simulators and Prototypes 277
Simulators and Prototypes: What Are They? 277
The Case of the Missing Hardware 278
Simulation 279
The Perils of Simulation 282
Prototypes 285
Summary 288
18 Managing the Test 289
Testing the Correct Code 290
Run Logs: Keeping Track of What's Going On 291
An End-of-Day Checklist 295
Guarding the Treasury: Problem Management 296
Getting Bugs Fixed 298
Removing Defects versus Meeting Schedules 301
Summary 305
Part VI The Final Stages 307
19 The Customer's Role in Testing 309
Controlled Early Introduction Programs 310
Preproduction Testing 314
Summary 319
20 The Testing End Game 321
Continuous Improvement 321
The Final Lesson-Bugs: You'll Never Find Them All 328
Summary 333
References 335
Glossary 339
Index 349 |
|
 |
|
| Scott Loveland (Wrappinger Falls, NY) is a Senior Technical Staff
Member in the z/OS development laboratory. Geoffrey Miller (Highland,
NY) joined IBM in 1982 and is a Senior Technical Staff Member in the
z/OS development Laboratory. Richard Prewitt (Poughkeepsie, NY) is
a Senior Software Engineer in the z/OS development laboratory. Michael
Shannon (Wrappinger Falls, NY) has worked as both a software debugger
and z/OS system programmer in the z/OS System Test group.
ACCREDITATION
Scott Loveland, Geoffrey Miller, Richard Prewitt, Jr., and Michael
Shannon have collectively spent more than 70 years in the software
testing trenches. They have consulted on best practices in test
and system availability to customers ranging from worldwide corporations
to the U.S. Department of Defense. They currently hunt bugs in the
IBM mainframe development laboratory in Poughkeepsie, New York.
|
|
