Test dimensions

Tests are written to check the behaviour of a given product. They may have multiple purposes, in the sense of what they are trying to check, but there are other properties that those tests have which place them within the test space. In general, there may be multiple purposes for a test, but for other dimensions the test usually only falls into one of the variants of the dimension.

Dimension	Meaning	Variants	Variant meaning
Purpose	What the test is used for	Smoke	Initial testing to shake out common issues
		Regression	Ensure that the we do not vary in known behaviour
		Acceptance	Ensure that the feature requirements are met
		Security	Ensure that the system is not able to be exploited for undesirable behaviour
		Robustness	Ensure that when exceptional events happen we don't break
		Measurement	Report on properties of the product which are measurable (usually continuous values)
Scope †	How much of the product and its environment the tests cover	Unit	Tests individual parts of the tests
		Integration †	Tests a module works internally, without reference to other parts of the product; sometimes split into 'module' and 'integration' tests to distinguish between tests of a module, and tests of interactions between modules
		System	Tests that the product works in the form that it will be delivered
		System integration	Tests that when used within a product environment the product works
		Customer	Tests that when used by the customer in their environment, it works.
Scale	How well the product works when given different load levels †	Minimal	Tested on the simplest environment
		Stress testing †	That the product functions when given lots of input
		Performance	How well the product functions
Technique	How the results are obtained	Static	By examination of the product structure
		Dynamic	By examination of the product's behaviour
		Passive	By examination of the effect of the product †
Method	How the tests are defined	Scripted	Checks for defined criteria
		Exploratory	Searches for areas of problems (sub-variants: Fuzz testing, injection testing) †
		Exhaustive	Checks the entire test space
Approach	Invasiveness of the testing †	Black box	Tests only know about inputs and outputs.
		White box	Tests have knowledge of the implementation and may directly access the internals of the product. †
		Grey box	Tests know about the implementation in order to exercise parts of the internals without direct access. †
Strategy	How the code is exercised	Positive	Checks that the prescribed inputs has desirable behaviour
Strategy	How the code is exercised	Negative	Checks that deviation from prescribed inputs has desirable behaviour (usually error reports)
Gate	When the code is tested	Interactive	Tested by developers whilst coding
		Commit	Tested on developer changes
		Mainline †	Tested on visibility to other users during development
		Release	Tested on visibility to the release process (usually for generation of candidates)
		Regular	Tested on a regular schedule †
Compatibility	Interactions with other products and environment	Baseline	Generic product/environment testing.
		Platform	How changes in platform affect the product (might be OS, CPU Architecture, libraries on system)
		Upgrade †	Tests that upgrades of the product versions function together (and possibly backwards compatibly if Rollback is expected)
Execution	How the tests are executed	Manual	Human interaction with the product
Execution	How the tests are executed	Automatic	Computer drives the product
Compliance	What criteria the tests must meet	Confidence †	No requirements, intended to increase confidence in areas †
		Functional	Meets our requirements
		License	Meets third party licensing requirements
		Regulatory	Meets legal requirements
Audience	How widely the tests are performed	Pre-alpha	Tests isolated to individual developers before they are ready to go to the group.
		Alpha	Tests internally within the product group
		Beta	Limited release to an external audience
		Pre-release	Limited release, usually to specific customers who have an interest in new features.
		Release	Available to all customers.

† Additional notes in cell tooltip / float overs.