1 Copyright (c) 1999, IEEE Computer Society Software Realities Author Contact: Testing without stated requirements Risk and If it is very important to satisfy a requirement, and it is the job of the tester to evaluate the product against that re- quirement, then clearly the tester must be requirements - informed of that requirement. So there are situations where this statement is basi- cally true. The deeper truth is that stated require- Based Testing ments are not the only requirements . Because of incompleteness and ambigu- ity, testing should not be considered merely as an evaluative process. It is also James Bach, Independent Consultant a process of exploring the meaning and implications of requirements . Thus, test- ing is not only possible without stated n Reframing requirements Analysis requirements , it's especially useful when I (Computer, Feb.)
2 1999, pp. 120-122), I proposed that requirements devel- opment is not something that hap- pens all at once at the start of a pro- ject. In real life, requirements are nego- tiated in the course of two simultaneous dialogues throughout the project life cy- they're not stated. I think we need to break out of the mythology that testing is some kind of robotic process. Tre- mendous value comes from testers and developers collaborating. Skilled testers evaluate the product against their under- standing of unstated requirements and cle. Those dialogues entail asking, What use their observations to challenge or do we want? and What can we build? question the project team's shared under- The quality of these dialogues goes a standing of quality. long way to determining the ultimate I think we need to A good tester stays alert for uninten- quality of the product.
3 After reading that break out of the tional gaps in the stated requirements , and column, Jerry Weinberg e-mailed me the mythology that works to resolve them to the degree justi- gentle criticism that the example I used fied by the risks of the situation. testing is some kind (a file conversion tool) was too simplis- tic to illustrate the requirements testing of robotic process. Satisfying stated requirements problem effectively. What about safety- The idea that a Software product must critical or otherwise complex products? satisfy its stated requirements is true if we Since Jerry co-authored, with Don define product quality as the extent to Gause, my favorite book about require- RISK AND requirements TESTING which we can reasonably claim that each ments, Exploring requirements : Quality There are at least four alleged truisms stated requirement is a true statement Before Design (Dorset House, 1989), I about testing a product against require- about the product.
4 But that depends on feel compelled to address that problem ments. Most of the testing textbooks on having a very clear and complete set of and expand upon the role of testing in my bookshelf promote these principles, requirements . Otherwise, you're locked requirements development. and each principle reflects some truth in to a pretty thin idea of quality. I define requirements as the set of ideas about the dynamics of testing. The deeper truth is that while quality that collectively define quality for a par- is defined by requirements , it is not ticular product. I define testing as a 1. Without stated requirements , no test- defined as the mere sum of satisfied . process of developing an assessment of ing is possible. stated requirements . There are many product quality.
5 First, I'll reframe the 2. A Software product must satisfy its ways to satisfy or violate requirements . requirements testing problem in terms of stated requirements . requirements are not all equal in their risk, and then I'll show what happens in 3. All test cases should be traceable to importance, and often they are even in complex or high-risk situations. one or more stated requirements , conflict with each other. It unnecessarily and vice versa. limits us to think about requirements as 4. requirements must be stated in disconnected ideas, subject to a Boolean Editor: James Bach, 1198 South Fork Dr., testable terms. evaluation of true or false. Front Royal, VA 22630; voice (540) 631-0600; A broader way to think about satisfy- fax (540) 631-9264; When we think in terms of risk, however, ing requirements is to turn our thinking I believe a richer set of ideas emerges.
6 Around and consider the risk associated June 1999 113. Software Realities with violating them. Good testers strive trouble of interpreting requirements by of quality. This goes beyond having to answer the question, What impor- simplifying requirement statements to a at least one test for each stated tant problems are there in this product? testable scale may make matters worse. requirement. Here's a real-life example: The screen 4. The test process will be more effec- Tracing test cases to requirements control should respond to user input tive if requirements are specified in To the extent that requirements matter, within 300 milliseconds. I once saw a terms that communicate the essence there should be an association between test designer fret and ponder over this of what is desired, along with an idea testing and requirements .
7 But talk about requirement. She thought she would of risks , benefits, and relative impor- traceability so often boils down to a form need to purchase a special tool to mea- tance of each requirement. Objective of clerical Bingo: For each requirements sure the performance of the product measurability may be necessary, in ID, list the test case IDs that relate; for each down to the millisecond level. She wor- some cases, but is never enough to test ID, list the requirement IDs that ried about how transient processes in foster robust testing. relate. The completeness of testing is then Windows could introduce spurious vari- presumably evaluated by noting that at ation into her measurements. Then she What happens when these principles are least one test is associated with each realized something: With a little prepa- applied to high-risk or complex soft- requirement.
8 This is a pretty idea, yet I've ration, an unaided human can measure ware? First, let me disclaim any and all seen projects where this checkbox trace- time on that scale to a resolution of plus concern about requirements and testing ability was achieved by defining a set of test or minus 50 milliseconds. Maybe that processes performed for reasons other cases consisting of the text of each require- would be accurate enough. It further than creating a quality product. For the ment preceded by the word verify. occurred to her that perhaps this require- purpose of this column, all the require- If the intent of the traceability principle ment was specified in milliseconds not to ments documentation done simply to is to demonstrate that the test strategy has make it more meaningful, but to make it pass a process audit is of no consequence.
9 Validated the product against require- more objectively measurable. When she Don't confuse that with what must be ments, then we have to go deeper than asked the designer, it turned out that the done to produce a quality product. checkbox tracing. We should be ready for real requirement was that the response As risks and complexities increase, par- our clients to ask the question, How do time not be as annoyingly slow as it is ticipation by testing in the requirements you know? We should be able to explain in the current version of this product. dialogue becomes more important if the the relationship between our tests and the Thus we see that the pragmatics of test process is going to achieve its mission. requirements . The fact that a requirement testing are not necessarily served by More testing skill is needed, as is a better is merely associated with a test is not inter- unambiguous specification, though test- rapport with the development and user esting in and of itself.
10 The important thing ing is always served by meaningful com- communities. In the dialogue about what is how it is associated, and that impor- munication. we want, testers should seek multichannel tance grows in pace with product risk. communication: multiple written sources, requirements , TESTING, diagrams, demos, chalk talks, and use Stating requirements AND CHALLENGING Software cases. In the dialogue about what can be in testable terms I reformulate the principles above into built, testers should be familiar with the It's important that requirements be the following, less quotable but more technologies being used, and work with meaningful. However, testable in this robust, guidelines: development to build testability enhanc- context is usually defined as something like ing facilities into the product.