Coverage strategies
Coverage strategies
What is "coverage" in combinatorial testing?
When testing an AI system, multiple independent factors influence its behaviour simultaneously. A user might be:
- Expressing a Complaint (Intent dimension)
- With a Frustrated tone (Emotional Tone dimension)
- As a First-Time User (User Expertise dimension)
- Using Ambiguous phrasing (Scenario Clarity dimension)
A "coverage strategy" determines how many combinations of these factors your test suite will exercise.
Why Full Cartesian is usually impractical
For a system with 8 dimensions and an average of 3 values each, the full Cartesian product is:
3^8 = 6,561 tuplesEach tuple drives at least one test case conversation. At 6,561 conversations, the cost of generation, evaluation, and review becomes prohibitive for most teams. Pairwise testing addresses this directly.
Worked example: 5-dimension system
Suppose your system has these 5 dimensions:
| Dimension | Values |
|---|---|
| Intent | Complaint, Query, Booking |
| Emotional Tone | Neutral, Frustrated, Anxious |
| Scenario Clarity | Clear, Ambiguous, Contradictory |
| User Expertise | First-Time, Returning, Power |
| Task Complexity | Single-Step, Multi-Step |
Full Cartesian: 3 × 3 × 3 × 3 × 2 = 162 tuples
Pairwise result: ~18–25 tuples — every dimension-value pair appears together with every other dimension-value pair at least once.
Research shows this still catches 70–90% of defects from factor interactions.
Choosing your strategy
| Strategy | When to use | Typical tuple count |
|---|---|---|
| Pairwise | Default for all projects | 15–100 |
| N-wise (N=3) | High-risk domains (healthcare, finance, safety-critical) | 50–500 |
| N-wise (N=4) | When 3-factor interactions are known failure points | 200–2000 |
| Full Cartesian | Regulatory exhaustive evidence, ≤ 4 dims × ≤ 3 values | Product of all value counts |
What is a "partial tuple" vs. a "full tuple"?
Partial tuple (pairwise and n-wise): covers a subset of your dimensions.
{
"tuple_type": "partial",
"dimension_values": [
{"dimension_id": "...", "value": "Complaint"},
{"dimension_id": "...", "value": "Frustrated"}
],
"dimension_count": 2
}Multiple partial tuples together achieve the desired coverage level. Any given partial tuple does not constrain the test generator on uncovered dimensions — they are free to vary.
Full tuple (full Cartesian): covers all approved dimensions in a single combination.
{
"tuple_type": "full",
"dimension_values": [
{"dimension_id": "...", "value": "Complaint"},
{"dimension_id": "...", "value": "Frustrated"},
{"dimension_id": "...", "value": "First-Time"},
{"dimension_id": "...", "value": "Ambiguous"},
{"dimension_id": "...", "value": "Multi-Step"}
],
"dimension_count": 5
}How coverage is tracked in the Coverage Dashboard
The Coverage Dashboard (Phase 6) tracks:
- Pairwise coverage %: fraction of all possible dimension-value pairs that have at least one approved tuple.
- Tuple status breakdown: PENDING_REVIEW / APPROVED / REJECTED / AUTO_REJECTED counts.
- Stale coverage: tuples flagged when a dimension value they reference is deprecated.
- Gap suggestions: dimension combinations with zero approved tuples.
When to use Full Cartesian despite the count
- Regulatory compliance requiring explicit evidence of every combination being tested.
- Very small spaces (≤ 4 dimensions × ≤ 3 values) where the count is manageable.
- Known interaction failures where any missing combination represents an unacceptable risk.
- Audit trail requirements where every combination must appear by name in the test record.
In all other cases, pairwise is the correct default.
Estimated Count Preview
Before triggering generation, the strategy selector shows a live estimated tuple count for your chosen strategy:
| Strategy | Estimation method |
|---|---|
| Pairwise | Upper bound from required pair coverage set |
| N-wise | Upper bound from required N-combination coverage set |
| Full Cartesian | Exact product count |
The estimate updates immediately as you change strategy or the N value for n_wise. Use it to understand what you're committing to before clicking "Done Reviewing".
⚠️ If Full Cartesian estimated count is > 500, a warning is shown. Consider pairwise unless you specifically need exhaustive coverage.
Max Tuples Cap
The strategy selector includes an optional Max tuples field. When set:
- The system generates the full covering array first, then randomly samples down to your limit
- The sampling seed is deterministic:
project_id + run_id. The same run always produces the same subset. - This gives you cost and time predictability without losing the structural integrity of the covering array
- Leaving it blank means fully autonomous generation (no cap)
When to use a cap:
- Full Cartesian on a moderate-sized space (100–1000 tuples) but you only want 200 for a quick smoke test
- N-wise on a large space where you want representative coverage but not all combinations
- Any time you're in a cost-sensitive context and want to limit LLM assessment calls