Part one: How to detect defects in non-existent requirements

Software testers specialize in finding things that can be improved in products and processes, including their own work. So they tend to have a lot to complain about. And probably the most popular complaint among testers is of inadequate information on which to base test design very early in the lifecycle.

So we want better requirements. But what does that mean? What would our ideal requirements document be like? Clarity, unambiguity and correctness are all desirable of course, and the entire test process is designed to detect defects in these. Something wrong with a document or other product used as test basis (an “error of commission”) will hopefully be detected by verification, most likely during early test design, or at worst by test execution.

But testing in any form is powerless against the worst defect a test basis can have: incompleteness. As the first product of development, requirements can’t be validated because there’s no earlier document against which to validate them. So something missing (an “error of omission”) is far harder to detect. If it’s not there you can’t review it and no tests will be created relating to it. In a sequential development lifecycle, it will then also be missing from subsequent work products based upon the requirements, for example designs, which will likely be fatally flawed throughout as a result. Because these contain information more detailed than requirements, they are easier to use as a test basis, but it’s now too late; the design is wrong, the tests will be wrong, and the delivered software will have defects - probably severe - testing cannot detect.

Think back to the last few times you tried to get information or carry out transactions on commercial websites, if the memory is not too painful. The majority of users - even proficient IT users - experience failure constantly. There is no way to monitor functional or usability failure in production, and therefore no way to measure its cost - but I believe it is immense. A large proportion of current web applications, even those operated by major organizations, are unfit for purpose because intended change to them is not properly documented and therefore not adequately tested.

So, is it realistic to expect complete requirements? For some software products arguably yes. But for web applications, unfortunately not. Nobody can specify a site, or an addition to an existing site, in sufficient detail, in advance - no person or group is that clever. The design aspects of the requirements will emerge from the development, as understanding of complex functionality, data, and in particular how to make navigation and data entry intuitive, grows. For web (and some other modern development environments) that’s the only sensible way for designers and developers to work.

Trying to guess at designs in advance is futile and damaging. Mocked-up pictures of screens can help business and developers envisage things approximately but have no value as a basis for early test design: they don’t contain nearly enough information and are open to multiple interpretations. Worse, they are invalid; the real product will begin to deviate from them the moment actual design work begins. Web requirements should be written, never pictorial. If a document contains a picture of the product it describes, it’s a design, not requirements, and of no use for VELT. Anyone tasked with capturing requirements should be capable of expressing them in writing. If not, they probably don’t really know what the requirements are.

The kind of requirements written by business analysts are usually closer in form to what testers need. They should describe what the application or change should enable the different user groups to do, and the business procedures and rules that enable test analysis to derive expected outputs. But they don’t address other fundamental properties which are crucial to the business success of a public-facing website - the characteristics site should and should not possess, especially (but not only) in its presentational layer and user interface. That causes shortcomings, inconsistencies and deviations from prevailing standards to be noticed during or after functional test execution, leading to uncontrolled, untested, very risky change too late in the delivery process which is the main cause of rotten websites going into production.

So working within these limitations and being realistic, how can testers get all the important aspects of the site on the test path? The answer is to document them ourselves as part of the test design activity. Most of the design decisions affecting the new functions will already be known to us, or easily predictable. When that is not the case, we can write them generically; specific information can be added when available and if necessary.

The deliverable from this activity can be called “test objectives”. They define what testing will aim to achieve, and can also act as the “title” of a test script or other testing activity during test planning, making traceability and prioritization easier.

More importantly, in the process of creating the test objectives we review the business and other requirements that are available, detecting defects. By considering those requirements systematically with reference to the fundamental functions of any public-facing website [see panel], we add related test objectives which would otherwise be omitted.

The review component of this activity can be made more powerful by enforcing a rule: that test objectives must be achievable. This means absolute language such as “to prove”, “always”, “never” etc cannot be used, because testing cannot show that something will always, or never, happen. Taking care with the wording in this way often reveals ambiguity in the requirements and raises useful and important questions needing clarification. So creating test objectives can detect not only errors of omission in requirements, but other potentially severe defects too, earlier than they can be detected by test design.

Once we have our best effort at what we consider to be a complete and theoretically (not practically - that will be decided during prioritization and later, estimation) achievable set of test objectives for the new functionality, we can use them as a basis for creating and detailing test cases, which should reveal if anything important is still missing.

Part two of this article