Figure 1 below illustrates the process of deciding the importance of any possible test case.

Figure 1.

Provided test cases are approximately equally easy to perform [2], more important ones [1] are those that safeguard against more significant failures [3], while the whole tree below that [3] explains what an “important failure” really depends on.

When failures have approximately equal importance [3] however, you would do best to prioritise test cases [1] starting with those that can be more easily tested [2], rather then those that do not yield reliable results because of execution or observability difficulties [4], or those that are very expensive to perform [4].

The importance of failure [3] is, as in traditional approach, a function of its consequences or price [5] and its probability [6]. How do we know the price of a failure? Well, in principle this is not test stuff, it’s business stuff [9]. If you ask them nicely, they will surely tell you ;-) . But beware, they might retaliate by asking you what failures are possible. Probability of failure [6] is a function of defect probability [8] and usage probability [7] (called more often usage frequency). Defect probability [8] is defined by a number of factors [11], which are diligently listed in many books on testing. By the way, “error guessing” (ISEB) or “defect guessing” (ISTQB) is just this: estimating defect probability [8]. Actual “experience-based techniques” (ISTQB) would mean assessment of the whole tree above involving estimations of all values and the strengths (function shapes) of their interdependencies.

For some reason beyond comprehension, ISTQB chose just one tiny leaf [11] of the whole tree and chose to call it “experience-based testing”!

I cannot resist the temptation to play with words here: ISEB’s “error guessing”, if you treat the meaning of “error” according to its BS 7925-1 definition, would definitely be a part of actual “defect guessing”. You know, like guessing whether all is well with Jane’s marriage (and if it is not, it is more probable she will make an error etc).

Let us get rid of the last bit of the figure not yet mentioned: in order to assess usage probability [7] you need to have some kind of – imagined or experience-based – usage model for this particular little function, the importance of whose failure you calculate.

Note:

the figure above is the only principle you need in order to talk about test case prioritisation. All other oratory on this subject, so prevalent until now, is no longer necessary. Just tell your customer to buy a BBN (Bayesian Belief Net) tool and start gathering data!

Please Google for yourself what BBN is if you are not familiar with it.

The Full Picture

The solution described above has one little catch in it: it should be used for “choosing a hundred or a thousand most important ones from 10¹⁰⁰⁰ theoretically possible cases”. One such tree as on Figure 1 for each of them, quick assessment, and we are soon ready ;-)

Therefore, additional methods of choosing test cases are necessary. Suddenly our proud tree, from being omnipotent and the ultimate test design tool, is reduced due to the sheer weight of numbers to a tool ultimately for a much more limited area, namely to prioritise those test cases already chosen by other means.

By what means, though? There are two approaches: intuitive or systematic. For some reason, both the ISEB and ISTQB describe only systematic test design techniques, sending intuitive into oblivion (in total disregard of the fact that more than 70% of the world’s test case population is designed by intuitive means), or by reducing them to some tiny particle called either “error guessing” or “experience-based testing”. Which in other words means, “assessing defect probability”, which is pathetically inadequate.

For both the intuitive and systematic approaches, there are two basic test design philosophies: black-box and white-box. The black-white scale is certainly equally valid for the intuitive as it is for the systematic approach, contrary to the impression you get from ISEB/ISTQB syllabi that such division is something that exists only for systematic techniques. Note as well that black-white is hardly a two-value discrete scale as is often presented. Rather, it is a scale of two extremes, from 100% pure black (an oracle behind a plaster wall answers your questions and it is irrelevant whether it is a computer or a human) to 100% pure white (you design test cases taking into account electronic gate level), through various shades of grey.

Anyway, the correct (somewhat simplified) test approach/philosophy matrix is set out in Figure 2. below:

Figure 2.

This is not the end of the story, however. Contrary once again to what the syllabi pretend, there is no conceptual canyon between static and dynamic testing. Calling “review” a static technique is the same as calling “test execution” a dynamic technique. Without doubt, test case design is required even in static testing, albeit of course that what constitutes a valid test case will differ. In static testing, the same way as in dynamic, you must first go through test design – for example you must decide in sufficient detail what to review and how - before “executing” the actual review.

Therefore, the final, complete set of test design classes is three-dimensional, composed of eight elements, as shown in the figure below.

Figure 3.

Note that words “test techniques” have not even been uttered yet, because the real issue is test design classes, of which there are approximately eight (approximately, because as mentioned before, in reality there are more than just two values on the black-white scale).

The reason for the emphasis given to test techniques by ISEB/ISTQB is of course didactic: the basic goal of a fundamental test course is to teach testers test design techniques which they will be able to apply where they are likely to work most, i.e. in dynamic testing. However, there is no need to build a whole false conceptual structure to achieve this practical goal! Besides, practice and common sense are anyway offered on the altar of… hard to say of what, because intuitive techniques, with which most testers have to work daily, are totally neglected. Again, it is possible to guess what rationale lies behind: all projects test intuitively anyway, but some sensible use of systematic techniques is badly lacking in many of them. Again, this rationale is dubious: there is no need to build a model that is not true to reality in order to achieve more widespread use of systematic test techniques. Last but not the least, intuitive testing needs improvement as well, so why neglect it altogether?

Finally: An Example

Let us see how the model depicted in Figure 3. works for real test design techniques. Let us take three techniques: (1) state-transition testing; (2) testing based on the experience of similar system’s fault distribution in the past and (3) testing based on instruction or other code coverage.

	state-transition	experience	code coverage
dynamic systematic black-box	Most common usage area	[not applicable]	Checking whether all methods are used (but not whether all code inside methods). OK, this is grey, not pure black
dynamic systematic white-box	E.g. testing object’s behaviour in a number of its states	[not applicable]	Most common usage area
dynamic intuitive black-box	[not applicable]	Similar reports used to be wrong in the past	[not applicable]
dynamic intuitive white-box	[not applicable]	It took us three days to find a bug in a similar sorting algorithm	[not applicable]
static systematic black-box	Designing state machine to verify written requirements specification	[not applicable]	Code inspection to ensure that test environment under preparation really will be able to invoke all states; again, it is grey, not pure black
static systematic white-box	Designing state machine to verify low-level function description	[not applicable]	Code inspection to measure possible coverage of a hard-to-execute piece of code
static intuitive black-box	[not applicable]	In this area user requirements were very vague last time	[not applicable]
static intuitive white-box	[not applicable]	Look, this guy uses jump table instead of proper function calls again!	[not applicable]

It works! And we are proud to make a small discovery, even if it is pretty obvious (but somehow alien to ISEB/ISTQB syllabi): of course there are no “black techniques” or “white techniques”, but there are systematic or intuitive techniques. A technique cannot at the same time be non-systematic and systematic.

Let us hope this article will not be an obstacle for any future accreditation for my company ;-)

www.bbjtest.com, info@bbjtest.com

About the author:

Bogdan Bereza-Jarociński runs the Poland-based company bbjTest, which provides training in testing, QA and SW engineering, as well as consultancy and test tool support services.

To read part one of this article, click here