- Home
- Insights
Verification
Verification strategies for complex software
Coverage targets do not tell you whether a system works. How to allocate verification effort by risk, across levels, and turn test results into release evidence.
- Published
- Reading time
- 3 min read
Most organisations that build complex software have a test suite. Far fewer have a verification strategy: an explicit answer to the question: which risks are we reducing, at which level, with which technique, and how do we know it worked?
Without that answer, test effort follows convenience. Code that is easy to unit-test gets unit-tested thoroughly. The interactions that actually fail in the field (timing, concurrency, configuration, degraded networks) are left to a manual system test near the end, if they are tested at all.
Start from risks, not from levels
The classic test pyramid is a useful heuristic for cost, but it says nothing about what to test. A better starting point is a short list of the ways the system is most likely to fail or most expensive to fail:
| Risk | Where it is best caught | Typical technique |
|---|---|---|
| Algorithmic or logic errors | Unit level, on the host | Example-based and property-based tests |
| Contract violations between components | Component / contract level | Contract tests against shared interface definitions |
| Timing and resource exhaustion | Integration level, on target | Load tests, execution-time measurement |
| Behaviour under faults | Integration / system level | Fault injection, network impairment |
| Misconfiguration and deployment errors | System level, production-like setup | Automated deployment and smoke tests |
Each row is a deliberate decision. The gaps, risks with no row, are equally deliberate and should be written down rather than discovered.
Make the lower levels genuinely cheap
Verification at higher levels is expensive: it needs hardware, environments and time. It pays to push every check that can run lower down as far down as it can go.
For embedded and distributed software, this mostly comes down to architecture. Logic that is separated from hardware and from the communication layer can be tested on a developer’s machine in milliseconds. Logic that is entangled with drivers and middleware can only be tested on a bench. The test strategy and the architecture are the same conversation.
Treat test infrastructure as a product
Higher-level tests fail for two reasons: the system is wrong, or the test environment is. When the second happens often enough, engineers learn to re-run failures rather than investigate them, and the suite stops providing information.
Test infrastructure therefore deserves the same engineering discipline as the product:
- Environments are reproducible. Test rigs, simulators and containers are defined in version control and can be rebuilt.
- Flaky tests are defects. They are tracked, quarantined with an owner and a deadline, and fixed, not silently retried.
- Failures are diagnosable. Every failing test produces logs, timing information and artefacts sufficient to understand the failure without reproducing it.
Measure what coverage cannot
Code coverage answers one narrow question: which lines executed during tests. It does not say whether the tests checked anything meaningful, and it says nothing about combinations, timing or configuration.
Coverage is useful as a negative signal (uncovered code is certainly unverified) and as a gate for new code. It is a poor target in itself. More informative measures for complex systems include:
- the proportion of identified risks with an automated test at the appropriate level;
- requirements with at least one passing, traceable test on the release candidate;
- mean time from a defect being introduced to it being detected;
- the rate of test failures caused by infrastructure rather than product defects.
Turn results into evidence
For regulated or safety-relevant products, verification must eventually produce evidence: a record that a specific requirement was verified, by a specific test, on a specific build. Assembling that record by hand before each release is slow and error-prone.
The alternative is to make traceability a by-product of normal work. Tests carry the identifiers of the requirements they verify; the CI system records results against the commit and build; a report is generated, not written. When the release decision comes, the evidence already exists.
Summary
A verification strategy is a short, living document that connects risks to levels and techniques, backed by infrastructure that makes running those checks cheap and trustworthy. It is less glamorous than a new test framework, and considerably more effective.