QAVE Studio Docs
QAVE Studio Docs
Sign inGet started — the Studio loop
Getting StartedFirst assurance runQuick assure
ConceptsLifecycleGlossaryHow QAVE Studio worksComing soon
Account & workspaceSignup and loginPeopleDemo creditsRolesSettings
Generate simulation suitesContext templatePII and field enhanceDimensionsAgents and plannerDraft review and Save
ExportExport formatsExport formats (detail)
Assurance SDKSDK overviewQAVE Studio suite fetch APIAgent and JSONL queueLangfuse bridge
Execute and tracesRecipe chatbot (reference SUT)TracingFetch tracing
AssureEvaluate APIRubrics and evidenceReading scores
History
WorkbenchGolden data
CoverageCoverage dashboardCoverage strategiesCoverage advisorTest case quality
RAG AssuranceRAG runsRAG datasetsRAG analytics
Project evalsError analysisUnderstanding scores
Studio guidesDimensionsTuplesConversation modesMulti-lingual testingWorkflowsHuman reviewBest practices
ReferenceAPI and packagesTroubleshooting
Coverage

Coverage strategies

Coverage strategies

What is "coverage" in combinatorial testing?

When testing an AI system, multiple independent factors influence its behaviour simultaneously. A user might be:

  • Expressing a Complaint (Intent dimension)
  • With a Frustrated tone (Emotional Tone dimension)
  • As a First-Time User (User Expertise dimension)
  • Using Ambiguous phrasing (Scenario Clarity dimension)

A "coverage strategy" determines how many combinations of these factors your test suite will exercise.


Why Full Cartesian is usually impractical

For a system with 8 dimensions and an average of 3 values each, the full Cartesian product is:

3^8 = 6,561 tuples

Each tuple drives at least one test case conversation. At 6,561 conversations, the cost of generation, evaluation, and review becomes prohibitive for most teams. Pairwise testing addresses this directly.


Worked example: 5-dimension system

Suppose your system has these 5 dimensions:

DimensionValues
IntentComplaint, Query, Booking
Emotional ToneNeutral, Frustrated, Anxious
Scenario ClarityClear, Ambiguous, Contradictory
User ExpertiseFirst-Time, Returning, Power
Task ComplexitySingle-Step, Multi-Step

Full Cartesian: 3 × 3 × 3 × 3 × 2 = 162 tuples

Pairwise result: ~18–25 tuples — every dimension-value pair appears together with every other dimension-value pair at least once.

Research shows this still catches 70–90% of defects from factor interactions.


Choosing your strategy

StrategyWhen to useTypical tuple count
PairwiseDefault for all projects15–100
N-wise (N=3)High-risk domains (healthcare, finance, safety-critical)50–500
N-wise (N=4)When 3-factor interactions are known failure points200–2000
Full CartesianRegulatory exhaustive evidence, ≤ 4 dims × ≤ 3 valuesProduct of all value counts

What is a "partial tuple" vs. a "full tuple"?

Partial tuple (pairwise and n-wise): covers a subset of your dimensions.

{
  "tuple_type": "partial",
  "dimension_values": [
    {"dimension_id": "...", "value": "Complaint"},
    {"dimension_id": "...", "value": "Frustrated"}
  ],
  "dimension_count": 2
}

Multiple partial tuples together achieve the desired coverage level. Any given partial tuple does not constrain the test generator on uncovered dimensions — they are free to vary.

Full tuple (full Cartesian): covers all approved dimensions in a single combination.

{
  "tuple_type": "full",
  "dimension_values": [
    {"dimension_id": "...", "value": "Complaint"},
    {"dimension_id": "...", "value": "Frustrated"},
    {"dimension_id": "...", "value": "First-Time"},
    {"dimension_id": "...", "value": "Ambiguous"},
    {"dimension_id": "...", "value": "Multi-Step"}
  ],
  "dimension_count": 5
}

How coverage is tracked in the Coverage Dashboard

The Coverage Dashboard (Phase 6) tracks:

  • Pairwise coverage %: fraction of all possible dimension-value pairs that have at least one approved tuple.
  • Tuple status breakdown: PENDING_REVIEW / APPROVED / REJECTED / AUTO_REJECTED counts.
  • Stale coverage: tuples flagged when a dimension value they reference is deprecated.
  • Gap suggestions: dimension combinations with zero approved tuples.

When to use Full Cartesian despite the count

  1. Regulatory compliance requiring explicit evidence of every combination being tested.
  2. Very small spaces (≤ 4 dimensions × ≤ 3 values) where the count is manageable.
  3. Known interaction failures where any missing combination represents an unacceptable risk.
  4. Audit trail requirements where every combination must appear by name in the test record.

In all other cases, pairwise is the correct default.


Estimated Count Preview

Before triggering generation, the strategy selector shows a live estimated tuple count for your chosen strategy:

StrategyEstimation method
PairwiseUpper bound from required pair coverage set
N-wiseUpper bound from required N-combination coverage set
Full CartesianExact product count

The estimate updates immediately as you change strategy or the N value for n_wise. Use it to understand what you're committing to before clicking "Done Reviewing".

⚠️ If Full Cartesian estimated count is > 500, a warning is shown. Consider pairwise unless you specifically need exhaustive coverage.


Max Tuples Cap

The strategy selector includes an optional Max tuples field. When set:

  • The system generates the full covering array first, then randomly samples down to your limit
  • The sampling seed is deterministic: project_id + run_id. The same run always produces the same subset.
  • This gives you cost and time predictability without losing the structural integrity of the covering array
  • Leaving it blank means fully autonomous generation (no cap)

When to use a cap:

  • Full Cartesian on a moderate-sized space (100–1000 tuples) but you only want 200 for a quick smoke test
  • N-wise on a large space where you want representative coverage but not all combinations
  • Any time you're in a cost-sensitive context and want to limit LLM assessment calls

Coverage dashboard

Coverage dashboard

Coverage advisor

Coverage advisor

On this page

What is "coverage" in combinatorial testing?Why Full Cartesian is usually impracticalWorked example: 5-dimension systemChoosing your strategyWhat is a "partial tuple" vs. a "full tuple"?How coverage is tracked in the Coverage DashboardWhen to use Full Cartesian despite the countEstimated Count PreviewMax Tuples Cap