Tag Archives: system tests

Types of software testing

24 Oct

A physician, a civil engineer, and a computer scientist were arguing about what was the oldest profession in the world.

The physician remarked, “Well, in the Bible, it says that God created Eve from a rib taken out of Adam. This clearly required surgery, and so I can rightly claim that mine is the oldest profession in the world.”

The civil engineer interrupted, and said, “But even earlier in the book of Genesis, it states that God created the order of the heavens and the earth from out of the chaos. This was the first and certainly the most spectacular application of civil engineering. Therefore, fair doctor, you are wrong: mine is the oldest profession in the world.”

The computer scientist leaned back in her chair, smiled, and then said confidently, “Ah, but who do you think created the chaos?” [1]

My wife left me this joke in my lunchbox one day recently.  It drives me to, in equal parts, laugh and cringe.  Writing software is the art of describing, to the most pedantic of machines, exactly what we want done, so we programmers are sure to make errors, but errors are, in fact manageable, if we just know how to look for them before they become problematic.

Tests can help us to validate the code we’ve written.  By writing good tests, automated tests, and by using them appropriately, we can quantify how good our code is and stop testing when the code is good enough.  In this blog, we’ll cover all topics software testing.

Tests come in many different shapes and sizes.  While there are many test strategies, let’s begin by focusing on two with dramatically different aims: unit tests and acceptance tests.

chart showing low-level, white box testing.

Unit testing is low-level white box testing.

Unit tests have been popularized by the excellent library JUnit and variants in several other languages.  Unit tests focus on a single, isolatable portion of code and attempt to verify its functionality.  Unit tests are typically “white box tests” – that is, tests written with knowledge of the inner-workings of the software.  Failures at the unit test level have the power to expose the exact cause of failure.  Unit tests can be cumbersome to maintain because they are so closely related to the code under test, but they can also provide excellent validation of refactoring success.  Unit testing, powerful though it is, is just the beginning of the arsenal of tools to uncover software faults.  Each unit test only adds a little bit to code coverage because they focus on specific small functions.

Acceptance testing is black-box testing and at a high-level.

On the opposite corner of the testing spectrum is acceptance testing.  Acceptance tests are written by the customer to verify that everything works as intended to accept delivery of code. Acceptance tests are black-box tests – written without knowledge of the underlying code.  Acceptance tests are written at a high level, so they offer substantial code coverage per test, but they are not very useful for isolating a problem.  If a customer gives you a failing acceptance test, odds are good that you will want to write a lower level test to help identify the root cause of the failure.

Software developers should never rely on the customer to find their bugs.  Acceptance tests can be thorough, but in practice they are often cursory and not necessarily repeatable.

System testing is the in-house analogue of acceptance testing. The software producer writes system tests to validate that the software is performing adequately for release. For a web application, they might use Selenium to manipulate a web browser in a repeatable fashion.  System tests evaluate the typical flow of the program as well as exceptional flows.  System tests are typically black box tests

A good test suite does not focus on only one kind of testing.  While it would be ideal to have a test suite that has exactly one failure per defect in the code, that goal is, for many purposes, unrealistic.  It is more important to have a suite of tests to detect all the failures than it is to have a suite of tests to precisely count each failure.  The good test suite will use a variety of types of tests at different levels to expose possible problems and to exercise the entire system.

Advertisements