Wednesday, March 12, 2008

Boundary Value Analysis (BVA)

Boundary value analysis is one of the most common and important techniques for test case design. Although not everyone has heard of this technique, almost every one of us uses it when testing code. Of course, if you are a beginning programmer you probably won’t write strict test cases for testing your program, but you will probably test whether or not it works correctly in some “special cases”. Those special cases usually represent boundary/extreme (minimum, maximum) values of some input variable. This is only a common example; the problem is certainly more general than this. BVA is one of the simplest test case design techniques and it is good starting point for someone new to software testing.

Boundary value analysis is basically a black box testing technique. In other words, with boundary-value analysis we can test programs without having their source code with us. Of course, this technique is applicable to white box testing too. Boundary value analysis is almost always used in conjunction with Equivalence partitioning and they are really closely related. Sometimes it is hard to distinguish between test cases generated by those two techniques. Generally speaking boundary values are usually identified as edge (extreme) values of equivalence classes. Therefore, a good practice is to identify equivalence classes (to do equivalence partitioning) prior to searching for boundary values. Equivalence partition boundaries are commonly source of errors. If you are not familiar with equivalence partitioning technique, please refer to the appropriate document. In most books I’ve read, equivalence partitioning is presented before BVA. Thus, this might be the right route for you too.

Programmers make “off-by-one” errors very frequently, and boundary value analysis is the technique that will make the immense pressure on those errors and make them visible. Off-by-one errors are commonly the result of:

  • misplacing < with <=, > with >= and vice versa.
  • treating an 0-indexed array as a 1-indexed array (for example Pascal programmers make this mistake frequently when trying to program in C or C++);
  • incrementing or decrementing variables where not needed and omitting it where needed;
  • errors in loop termination conditions;
  • etc.

OK, now that we have some basic information, we can proceed to some concrete examples of errors that are tested by this technique.

Example of off-by-one error in mathematics

N apples are assembled in a straight line. Each apple is assigned a single number which represents its position in the line. If we are going to take all apples from apple p to q (including them), how many apples will we take?

q-p? No, this is an off-by-one error!

q-p+1? Yes, this is the correct answer.

If you don’t belive, take a look at the following picture. How many apples are there from the 2th position to the 6th positon? 6-2+1 = 5.


Now, let’s take a look at the following C++ code:

Example of off-by-one programming error

Function is given that colors a rectangle with the coordinates of upper left and bottom right points given as arguments. The following picture should describe the process clearer.

void fillRectangle(int top, int left, int bottom, int right) {
for (int i = top; i < bottom; i++)
for (int j = left; j < right; j++)
fillPoint(i, j);
}

This code contains a typical off-by-one error. Instead of < signs we should have put <= signs. As a result the last column and row of the rectangle will not be filled. This is very common type of error and some kind of BVA should be able to uncover this type of errors. This code in particular, gives incorrect result for every input; not only for boundary cases; but nevertheless it is an interesting example. Of course, in most cases you will not be able to see the source code but you should think of and anticipate this kind of errors.

Example of boundary error in programming

The following code fragment is an example of erroneous code with lots of off-by-one errors. It is obvious that for boundary values like 90, 80, 70, 60, and 50 no appropriate grade is given. This bug can be found easily by using BVA technique.

if (points > 90) grade = 'A';
if (points > 80 && points < 90) grade = 'B';
if (points > 70 && points < 80) grade = 'C';
if (points > 60 && points < 70) grade = 'D';
if (points > 50 && points < 60) grade = 'E';
if (points < 50) grade = 'F';

Now that we have seen some basic examples of bugs that we are looking for lets try to define the process of finding them, for the beginning rather informally.

For example if our program takes an integer variable in range (1-10) and prints its square value. Then, using previously described logic good test cases are 1 and 10. Also 0 and 11 (illegal test cases though) and maybe even 2 and 9 represent good test cases, because they are close to the given boundaries. If we are working with double/float/real numbers in the same range it is worth trying with numbers very close to the edges for example 0.9999, 1.0001, 9.9999 and 10.0001.

Values likely to be problematic

Sometimes boundary values are not so obvious. Take for an example a program that calculates the following math function:


By using elementary math we can conclude that the function is defined for:


Therefore good test cases are:

-4.0001, -4, - 3.9999, 2.9999, 3, 3.0001, 4.9999, 5, 5.0001 since these are the values on and close to the edges. Also since we usually don’t have -∞ and +∞ in programming languages we can try with MinDouble and MaxDouble values. Zero is almost always good boundary case, even in this function.

Further, since numbers in computers are stored in binary representation, sometimes good boundary cases are powers of two like …4, 8, 16, …, 1024,… and corresponding negative numbers …-4, -8, -16, …, -1024m,... Maximal and minimal values of numerical variables are usually close to (smaller/greater by one) or equal to powers of two. This is one more reason for using powers of two as boundary values. 1 (Also power of two) and 0 might represent good boundary values in many examples.

Now it is time to define these rules a bit more formally.

  • If we have an input condition which states that some variable is in the given range of values, then we should try testing with these values and the values directly beyond the boundaries.
  • If we have an input condition which states that some variable represents quantity/number of values, then we should try testing with the given boundary values (minimum and maximum number of values) and the values directly beneath and beyond these values. For example if we have a list with 1-10 elements we should try testing the list with 0, 1, 10 and 11 elements. We can also test a list with 2 or 9 elements but these are less likely to be such effective test cases, at least in this specific case.
  • Another good feature of the BVA technique that it looks for boundary values not only in the input space, but also in the result (output) space. Therefore, we should locate these values and try to produce them by using suitable input values. This is not always possible, but it is worth trying. Result space should be analyzed in the same way as input space (the previous two rules).
  • Human ingenuity is far more powerful than simple rules and should be used to the utmost extend in this technique.

We can also, define overall process more formally:

  • Use equivalence partitioning to find equivalence classes;
  • For each equivalence class consider the boundary values and the values directly above and below them.
  • Apply the same technique to output if possible.
  • Use your own imagination to find interesting boundary values and equivalence classes. Try to think what are the special/boundary values and what makes them special. Why this program would treat them differently?

This process can give duplicate test inputs. Of course, duplicates should be ignored. It is OK to test multiple boundary values in a single test case, but only if those values are legal. It is not correct practice to use a test case testing more than one illegal boundary value.

Other problems that are especially vulnerable to boundary values are:

  • Working with years, months, weeks, days…
  • Array indexing

Some good ideas and guidelines:

  • When working with files pay attention to operations with first / last record.
  • When working with arrays pay attention to operations with first / last element.
  • When working with lists pay attention to operations with first / last element.
  • When working with trees pay attention to root and leaf nodes and empty tree.
  • When working with queues and stacks try underflow / overflow scenarios.
  • When working with characters pay attention to special characters like ‘/0’, ‘ ‘, ‘A’, ‘Z’, ‘a’, ‘z’, ‘0’, ‘9’, control characters, etc…
  • When working with strings pay attention to empty strings, strings with all blank characters and strings of maximal length (if the length of sting is limited by programming language).
  • NULL values in databases represent huge source of errors.

It is good idea to find the measure type of your variable and according to that find special values:

  • Distance: nearest, farthest
  • Length: shortest, longest
  • Order: first, last
  • Size: biggest, smallest
  • Speed: slowest, fastest
  • etc.

Testing a simple program with BVA technique

Simple program for adding two integer numbers:

Although this is an extremely simple example, many interesting test cases can be found. Here we will concentrate only on boundary value analysis technique. Apart from that there are many interesting examples.

Some interesting BVA test cases might be:

  • One of the numbers is 0. (This gives us two test cases, if we test some feature on the first number, we must do the same on the second because we don't know whether the application handle both numbers in the same way. Therefore we treat both numbers symmetrically. This applies to all similar following cases.)
  • One of the numbers is MaxInt (Maximal integer value), other is <= 0.
  • One of the numbers is MinInt (Minimal integer value), other is >= 0.
  • Spaces or tabs are present in one of the numbers, for example ' 12 '.
  • The sum of two given numbers is MaxInt.
  • The sum of two given numbers is MinInt.
  • Enter one of the numbers as +0.
  • Enter one of the numbers as -0.
  • Enter one of the numbers as positive number with explicit + sign. For example +4.
  • One or both fields are empty.
  • If you find something interesting, please let me know!

The essential component of a test case is its expected result. If you have a good test case but you don’t know expected result, then your test case is certainly useless because it can’t be used for checking program's correctness. In examples I presented in this lesson test cases are given rather informally without explicitly specifying expected result. This is OK since our focus is on logic not on creating formal real world test cases, but in your practice you should explicitly write down everything.

Please don’t let the name of technique (BVA) and common examples mislead you. You should not test only values that are on the “edges” you should test all values that you suspect are special in any way, so maybe the more appropriate name for this technique would be Special Value Analysis.

Now I will tell you about one very interesting example from my own practice, which is somehow related to boundary value analysis.

I tested a part of application for generating HTML reports based on value of some fields in database. It looked as not error prone place but I tried and succeeded in finding one special case. Since reports are generated in HTML format I supposed that there is some kind of report template in HTML which is filled with actual data. That led me to an interesting test case. I tried to enter some HTML tags in database table fields that will populate reports. If the values of these fields are simply inserted into reports than it would be a serious problem. My guess was right, after doing this reports were incorrect and even browser reported erroneous code. Even using <> signs (which are used for tags) represented error.

By the way, in the time of publishing this post I noticed the same error in this blog publishing software! I lost some considerable amount of text at the places where I used < and > signs, because software confused them with parts of HTML tags. In most cases I had to write HTML equivalent of this signs. Also, multiple space characters are not interpreted correctly.

And at the end, you should use BVA whenever you find it appropriate, in any kind of application and in any testing level (unit, integration, system, acceptance). Have a fun learning this and try to avoid boundaries in thinking. Try to find as many as possible interesting boundary values!

Thank you for patience. Any kind of opinion, advice, suggestion or critic is welcome!

Useful links:

References:

  • The Art of Software Testing, by Glenford Myers
  • Software Testing (Second Edition), by Ron Patton

No comments: