Combinatorial testing with jenny

Saturday 1 May 2004

As a developer, I’ve often puzzled over how to do good feature coverage testing. You have to take a complex program with many dimensions of features, and design a finite set of tests that will verify that any possible use of the program behaves correctly. How can the infinite variety of uses of the software be whittled down to a manageable yet complete set of test cases? When I am asked to interview candidates for testing positions, I ask them this question, precisely because I don’t know the answer. If a candidate says something that enlightens me even a little, I recommend hiring them.

Kim was also thinking about combinatorial testing, and found jenny as an aid in planning combinatorial tests. Jenny is a simple program that doesn’t understand anything about your particular software. All it does is tell you which combinations of features to test to ensure that every possible pair (or triple, or quadruple, or ..) of features has been tested together. Then you take its feature tuples and write tests from them.

I haven’t used jenny, so I don’t know how much of a help it is. Clearly, it takes more than a combinatorial tuple generator to create good test scripts. Even deciding what dimensions are important in your software seems like an art. Any real product would probably have hundreds of dimensions, meaning they couldn’t all be fed into jenny at once. I guess you’d have to make some decisions about how to partition the dimensions into manageable arenas. Even with these limitations, jenny seems like it could be a useful tool to reduce the number of test cases while keeping the same coverage.

Comments

[gravatar]
Ian Bicking 12:27 AM on 2 May 2004

Sounds like an automated brainstorm.

[gravatar]
Tom Morris 3:16 PM on 3 May 2004

A Microsoft tester wrote about pairwise testing recently:

http://blogs.msdn.com/micahel/archive/2004/04/28/122702.aspx

[gravatar]
Bob Jenkins 12:50 AM on 11 May 2004

Actually, jenny can handle hundreds of dimension at once. But not thousands.

Add a comment:

Ignore this:
Leave this empty:
Name is required. Either email or web are required. Email won't be displayed and I won't spam you. Your web site won't be indexed by search engines.
Don't put anything here:
Leave this empty:
URLs auto-link and some tags are allowed: <a><b><i><p><br><pre>.