Is your system good enough? Are the tests good enough? Both of these questions can be answered quantitatively, and the first step is to understand the system. The process starts with listing the main functions of the system, determining how often each function is called, and how severe failures are.
Consider a simple web site that enables event photographers to upload and sell downloads of photographs to customers. This system will need to set up an account for a new photographer, allow the photographer to upload photos, show thumbnails and proofs to customers, and allow the user to purchase and download individual photos.
The next step is to determine the natural units for the system. In this case, the objective is to take orders for pictures. The system does other things – it receives pictures, displays pictures, and registers photographers, but it does all of these things in support of the sale. Because the objective of the site is to sell pictures, the natural units are orders.
With natural units established, we can determine the frequencies of various calls. The easiest to determine is checkout. It will be called once per order. For other functions, we must use the resources at our disposal to make reasonable estimates.
Suppose the business analyst forecasts an average of 3 images per order. Then, we know that there will be 3 downloads per order, and there will be at least 3 occurrences of “add to cart” for each order. We guess 4 occurrences for “add to cart” because the typical user picks an extra picture and later discards it.
Carrying on through the rest of the list in this manner, we make assumptions: the photographer takes 500 pictures per event and sells 6 orders after 30 people browse the pictures. 20 pictures show on a page of thumbnails. A typical pohtographer will sell 1000 orders through the system during his photography stint.
With this list, we can now compute the relative frequency of use of each function. Normalize that column, that is, sum the “calls per order” column and divide each value by that sum.
|Function||calls per order||frequency %|
|Add to cart||4||9|
We have one more round of estimating and arithmetic. How severe is a failure? Again, these will be relative numbers. Suppose a failure to take an order is of severity 1. We estimate the severity of other failures based on that.
We could hem and haw about the mode of failure – taking an order could charge the customer but not deliver the goods, or it might just get stuck leaving the customer unable to complete the purchase. For this purpose, it is better just to say “it fails” and consider the overall consequences of a failure.
A failure to register a photographer is particularly severe because it causes many lost orders over time. Not every photographer who has a problem will abandon the site, though. Still, we will use the higher number to be conservative because photographer registration gives the user a critical first impression.
|Function||Calls per Order||Frequency %||Severity||Freq * Severity||Relative impact %|
|Add to cart||4||9||0.5||0.04||11|
(You may notice that the percent columns don’t add up to exactly 100. That’s rounding error. With this sort of number, we really only have 1 significant figure anyway, so try to relax.)
The next task is to write tests for the functions in proportion to the last column, with at least one test for each function.
Suppose you already have some tests. See how they stack up against this measure. Are you testing the most important parts? Are you testing in proportion to the severity of failures?
This table gives us a start at understanding how much to test, and it will help us to characterize the severity of failures we encounter during testing.
Reference: Musa, John D. Reliability Engineering: More Reliable Software Faster and Cheaper 2nd Edition. Bloomington, Ind.: Authorhouse. 2004.