The Pairwise Independent Combinatorial Testing tool (PICT) can help you efficiently design test cases and test configurations for software systems. With PICT, you can generate tests that are more effective than manually generated tests and create them in a fraction of the time required by hands-on test case design. PICT generates a compact set of parameter value choices that represent the test cases you should use to get comprehensive combinatorial coverage of your parameters.
Using PICT to Combine Test Case Parameters
PICT runs as a command line tool. You prepare a model file detailing the parameters of the interface (or set of configurations, or data) you want to test. PICT generates a compact set of parameter value choices that represent the test cases you should use to get comprehensive combinatorial coverage of your parameters.
For instance, if you wish to create a test suite for partition and volume creation, the domain can be described by the following parameters: Type, Size, File system, Format method, Cluster size, and Compression. Each parameter has a limited number of possible values, each of which is determined by its nature (for example, Compression can only be On or Off) or as an equivalence partition (such as Size).
Type: Primary, Logical, Single, Span, Stripe, Mirror, RAID-5
There are over 4,700 possible combinations of these values. It would be very difficult to test all of them in a reasonable amount of time. Research shows that testing all pairs of possible values provides very good coverage and the number of test cases will remain manageable. For example, {Primary, FAT} is one pair and {10, slow} is another; a single test case can cover many pairs.
For the set of parameters shown above, PICT will produce 60 test cases. The last section of this guide contains a complete model for this volume/partition example.
PICT is a command-line tool that accepts a plain-text model file as an input and outputs a set of test cases.
Usage: pict model [options]
Options:
/o:N- Order of combinations (default: 2)
/d:C- Separator for values(default: ,)
/a:C- Separator for aliases (default: |)
/n:C- Negative value prefix (default: ~)
/e:file - File with seeding rows
/r[:N]- Randomize generation, N - seed
/c- Case-sensitive model evaluation
/s- Show model statistics
The PICT installer adds PICT to the path, so you can run it from any folder on your system. To see the PICT output for the example in the previous section, create the model file with any plain text editor, such as Windows® Notepad®, and run PICT with this command:
C:\YourFolder> pict ModelFile.txt
To create a tab-delimited output file, use command line output redirection.
You can use Microsoft® Excel® to sort or filter PICT’s output. You either import the output file into Excel, or use an .xls extension when you create the file and open it directly in Excel.
A model consists of at least one, and at most, three sections:
parameter definitions
[sub-model definitions]
[constraint definitions]
Model sections should always be specified in the order shown above and cannot overlap. The parameters definition section comes first, followed by the optional sub-model and constraints sections, if you use them. Sections do not require any special separators between them. Empty lines can appear anywhere. You can include comments by prefixing lines with the “#”character.
Simple Model
To produce a very basic model file, list parameter names–each on a separate line–with their possible values delimited by commas:
<ParamName> : <Value1>, <Value2>, <Value3>, ...
Example:
#
# This is a sample model for testing volume create/delete functions
#
Type: Primary, Logical, Single, Span, Stripe, Mirror, RAID-5
By default, PICT generates a pairwise, or order two, suite of test cases–all pairs covered. You can set the order to a value larger than two using the option /o:. For example, if you specify /o:3, the resultant test cases will cover all triplets of values,producing a larger number of tests than the pairwise option, but potentially giving the test suite more coverage. The maximum order for a simple model is equal to the number of parameters, which will result in an exhaustive, all possible combinations, test suite. Following the same principle, specifying /o:1 will produce a test suite that covers all values only once (combinations of 1).
Sub-Models
Sub-models allow you to bundle certain parameters into groups that get their own combinatory orders. This can be useful if combinations of certain parameters need to be tested more thoroughly or must be combined in separation from the other parameters in the model. The sub-model definition has the following format:
For example, sub-modeling is useful when hardware and software parameters are combined together. Without sub-models, each test case would produce a new, unique hardware configuration. Placing all hardware parameters into one sub-model produces fewer distinct hardware configurations and potentially lowers the cost of testing. The order of combinations you can assign to each sub-model allows for additional flexibility.
PLATFORM: x86, ia64, amd64
CPUS: Single, Dual, Quad
RAM: 128MB, 1GB, 4GB, 64GB
HDD: SCSI, IDE
OS: NT4, Win2K, WinXP, Win2K3
IE: 4.0, 5.0, 5.5, 6.0
APP: SQLServer, Exchange, Office
{ PLATFORM, CPUS, RAM, HDD } @ 3
{ OS, IE } @ 2
The diagram below depicts how the generation would look for the above model:
1.You can define as many sub-models as you want; any parameter can belong to any number of sub-models. Model hierarchy can be just one level deep.
2.The combinatory order of a sub-model cannot exceed the number of its parameters. In the example above, an order of the first sub-model can be any value between one and four.
3.If you do not specify the order for a sub-model, PICT uses the default order or the order specified by /o option.
Constraints allow you to specify limitations on the domain. In the earlier partitions example, the pair {FAT, 5000} will occur in at least one test case. However, the FAT file system cannot create volumes larger than 4096 MB. Note that you should not simply remove such violating test cases from the test suite, because an offending test case may cover other, possibly valid, pairs that don’t appear in any of the valid tests. To avoid the risk of losing valid pairs, you should eliminate disallowed combinations during the generation process. You do that in PICT by specifying constraints, as in this example:
Type: Primary, Logical, Single, Span, Stripe, Mirror, RAID-5
A term [parameter] relation value is an atomic part of a predicate. The following relations can be used: =, <>, >, >=, <, <=, and LIKE. LIKE is a wildcard-matching operator (* - any character, ?– one character).
[Size] < 10000
[Compression] = "OFF"
[File system] like "FAT*"
Operator IN allows specifying a set of values that satisfy the relation explicitly:
IF [Cluster size] in {512, 1024, 2048} THEN [Compression] = "Off";
IF [File system] in {"FAT", "FAT32"} THEN [Compression] = "Off";
The IF,THEN, and ELSE parts of a predicate may contain multiple terms joined by logical operators: NOT, AND, and OR. Parentheses are also allowed in order to change the default operator priority:
PICT uses the concept of parameter types. There are two types of parameters: string and numeric. A parameter is considered numeric only when all its values are numeric. If a value has multiple names, only the first one counts. Types are only important when evaluating constraints. You can only compare a numeric parameter to a number, and a string parameter to another string. For example:
Size: 1, 2, 3, 4, 5
Value: a, b, c, d
IF [Size] > 3 THEN [Value] > "b";
String comparison is lexicographical and case-insensitive by default. You can make comparisons case-sensitive if you specify the /c option. Numeric values are compared as numbers.
Using aliasing you can specify multiple names for one value. Multiple names do not change the combinatorial complexity of the model. No matter how many names a value has, it is treated as one entity. The only difference will be in the output; any test case that would normally have that one value will have one of its names instead. Names are rotated among the test cases. The following example shows a practical application of aliases. In most of the cases, you can ignore the differences between Microsoft® Windows 2000 Server and Microsoft Windows 2000 Advanced Server for testing purposes. So, a tester may decide to test just one version, thereby reducing the number of test cases. However, for increased confidence in the test suite, you may decide to test both versions interchangeably instead. Specifying one value with two names will result in having either Server or Advanced Server in the output without adding extra test cases.
By default, names should be separated by the “|” (pipe) character, but you can change to a different character using the /a: option.
if [OS_1] = "WinXP" then [SKU_1] = "Professional";
if [OS_2] = "WinXP" then [SKU_2] = "Professional";
#
# No German WinPowered
#
if [SKU_1] = "WinPowered" then [LANG_1] = "EN";
#
# Let’s not test the same OS on both sides
#
[OS_1] <> [OS_2];
Note: When evaluating constraints, PICT uses only the first name. For instance, [SKU_1] = "Server" will match the second value but [SKU_1] = "AdvServer" will match nothing. Also, PICT uses only the first name to determine whether a value is a numeric type or negative.
In addition to testing valid combinations, referred to as “positive testing,” you usually need to test using values outside the allowable range to make sure the program handles errors properly. Each “negative testing” test case should have only one invalid value because most applications take some failure action when they detect the first error. For this reason, a problem known as input masking can occur with negative testing. one invalid input prevents another invalid input from being tested.
Consider the following function which takes two arguments:
floatSumSquareRoots( float a, float b )
{
if ( a < 0 ) throw error; // [1]
if ( b < 0 ) throw error; // [2]
return ( sqrt( a ) + sqrt( b ));
};
Although the function can be called with any values for numbers a andb, it only makes sense to do the calculation on non-negative numbers. For that reason the function performs verifications [1] and [2] on the arguments. Now, assume a test ( a= -1, b = -1 ) was used to test a negative case. Here, a = -1 actually masks b = -1 because check [2] never gets executed and the failure if check [2] didn’t exist would go untested.
To prevent input masking, it is important that two invalid values (of two different parameters) are not used in the same test case. Prefixing any value with “~” (tilde) marks it invalid. Option /n: allows you to specify a different prefix.
#
# Trivial model for SumSquareRoots
#
A: ~-1, 0, 1, 2
B: ~-1, 0, 1, 2
In this example, PICT guarantees that all possible pairs of valid values will be covered and all possible combinations of any invalid value will be paired with all positive values at least once.
A B
2 0
2 2
0 0
0 1
1 0
2 1
0 2
1 2
1 1
~-1 2
1 ~-1
0 ~-1
~-1 1
2 ~-1
~-1 0
Notes:
1.PICT does not include the prefix as part of a value during comparisons. In constraints, use the value without the prefix. A valid constraint (although artificial in this example) would be:if [A] = -1 then [B] = 0. Also the prefix does not affect the type of a value. For example, both parameters in the above example are numeric despite having the non-numeric char“~” in some of the values. The prefix will appear in the output.
2.If a value has multiple names, prefixing only the first name will make the value negative.
Using weights, you can force PICT to prefer certain values. A weight can be any positive integer. PICT uses weight “1” if you don’t specify a different weight explicitly:
#
# Let’s focus on primary partitions formatted with NTFS
#
Type: Primary (10), Logical, Single, Span, Stripe, Mirror, RAID-5
Format method: quick, slow
File system: FAT, FAT32, NTFS (10)
Important note: Weight values have no absolute meaning. For example, when a parameter is defined like this:
File system: FAT, FAT32, NTFS (10)
This does not mean that NTFS will appear in the output ten times more often than FAT or FAT32. Moreover, you cannot be sure that the weights you specified will be honored at all because PICT must handle two potentially contradictory requirements:
1.To cover all combinations in the smallest number of test cases.
2.To choose values according to their weights.
(1) will always take precedence over (2) and weights will only be honored when the choice of a value is not determined by a need to satisfy (1). More specifically, during the test case creation, PICT evaluates candidate values and it always chooses one that covers most of the still unused combinations. Sometimes there is a tie among candidate values and no one choice is better than any other. In those cases, PICT uses weights to make the final choice. You can use weights to attempt to shift the bias towards some values, but PICT decides whether or not to honor that request, and to what extent, using several factors, not just the weights alone.
1.It allows you to specify important combinations that should always appear in any generated test suite. PICT will initializes the output with combinations you provide and then builds up the rest of the suite, while still making sure that all n-order combinations get covered.
2.It lets you minimize changes to the output when you need to modify your model. You can provide PICT with the results of a previous generation and PICT will reuse the old test cases as much as possible.
Seeding rows must be defined in a separate file (a seeding file). Use option /e: to provide the tool with the file location:
pict.exe model.txt /e:seedrows.txt
Seeding files use the same format as PICT output. The first line contains parameter names separated by tab characters and each subsequent line contains one seeding row with values also separated by tabs. You can create this format can easily using Windows Notepad or in Microsoft® Excel. Using the PICT output format for the seeding file also allows you to directly any prior PICT result outputs to seed new test suite generation.
VerSKULangArch
Win2KProENx86
Win2KDEx86
WinXPProENia64
Any seeding row may be complete, with values specified for all parameters. Or, you may omit some values, as in the second seeding row above which does not have a value for the SKU parameter. In this case, PICT will choose the best value for SKU.
Important notes:
There are a few rules of matching seeding rows with the current model:
1.If a seeding file contains a parameter that is not in the current model, PICT discards the entire column representing the parameter in the seeding file.
2.If a seeding row contains a value that is not in the current model, PICT removes the value from the seeding row. PICT will use only valid values from that seeding row, and a row with invalid values becomes an incomplete or partial row.
3.If a seeding row violates any of the current constraints, PICT ignores the row.
PICT will issue warnings if (1) or (2) occurs.
Some formats normally allowed in PICT models may lead to ambiguities when you use seeding:
1.Blank parameter and value names.
2.Parameter and value names containing tab characters.
3.Value names and all their aliases not unique within a parameter.
PICT will warn you if any of those problems exist in a model where you’ve also used seeding.
Output Randomization
If you use the same model and options several times, every run will result in the same identical test suite. However, you can randomize the output using the /r option. A randomized generation prints out the seed used for that particular test suite to the error output stream. You can use that seed value to recreate a test suite using the /r:<seed> option.
The efficiency of the generation process depends greatly on starting conditions. Every test suite produced is guaranteed to cover all needed combinations but some starting points may cause the tool to pack combinations into test cases more efficiently than others. If you need to minimize the number of test cases, try running PICT with the /r option several times and use the output with the fewest tests.
Case Sensitivity
By default, PICT does all of its comparisons and checks case-insensitively. For instance, if there are two parameters defined as OS and os, PICT considers them duplicate names (parameter names must be unique). PICT also resolves constraints case-insensitively by default: IF [OS] = "Win2K" THEN ... will match both Win2K and win2k values (values of a parameter are not required to be unique). You can use option /c to make the model evaluation fully case-sensitive.
PICT prints all errors, warning messages, and the randomization seed to the error stream. It prints the test cases to the standard output stream. The first line of the output contains names of the parameters. Each of the following lines represents one generated test case. Values in each line are separated by tab characters.
Type Size Format method File system Cluster size Compression
Primary 40000 quick NTFS 1024 off
Span 100 slow FAT32 16384 on
RAID-5 500 quick FAT 4096 on
Primary 5000 slow FAT32 4096 off
RAID-5 1000 slow FAT 32K on
Stripe 500 slow NTFS 2048 on
RAID-5 10 quick FAT32 2048 on
Primary 10 slow FAT 1024 on
Logical 100 quick NTFS 32768 on
Primary 5000 quick NTFS 512 off
Span 500 slow FAT32 32768 off
Mirror 5000 slow NTFS 64096 on
RAID-5 100 slow NTFS 4096 off
RAID-5 10 slow FAT 512 on
Logical 1000 quick FAT32 8192 on
Logical 100 quick NTFS 4096 off
Single 10000 quick FAT 65536 off
Warning Messages
There are two warnings that you should pay close attention to, as they indicate that some constraint or constraints might be defined incorrectly. PICT does not stop test suite generation because, even though something is wrong with the semantics of the model, PICT can still generate some meaningful results. However, if you see either of these warnings, re-examine and fix your parameters and constraints.
This warning gives you an early notice that you might have defined a relation incorrectly. All relations, either on the condition or on the term side of a constraint, should resolve to a subset of values to be meaningful. If no value or all values satisfy the relation, then something is not right. If you see this warning, review your value relationships and fix them.
Restrictive constraints. Output will not contain following values…
Sometimes a set of constraints becomes so intertwined that some values always violate one or more constraints. One example of such situation is a dependency loop:
if [A] = "a1" then [B] = "b1";
if [B] = "b1" then [C] = "c1";
if [C] = "c1" then [A] = "a2";
In this example, choosing a1 implies choosing b1, which makes the tool choose c1, which in turn implies a2. So, if you have chosen a1, you have to choose a2. All those constraints when evaluated separately are perfectly valid. It’s only their interaction that leads to the ambiguity. PICT will avoid this situation by not using a value of a1 at all. a1will never be used in an output and PICT will display the “Restrictive constraints…” warning.
