QAVE Studio Docs
QAVE Studio Docs
Sign inGet started — the Studio loop
Getting StartedFirst assurance runQuick assure
ConceptsLifecycleGlossaryHow QAVE Studio worksComing soon
Account & workspaceSignup and loginPeopleDemo creditsRolesSettings
Generate simulation suitesContext templatePII and field enhanceDimensionsAgents and plannerDraft review and Save
ExportExport formatsExport formats (detail)
Assurance SDKSDK overviewQAVE Studio suite fetch APIAgent and JSONL queueLangfuse bridge
Execute and tracesRecipe chatbot (reference SUT)TracingFetch tracing
AssureEvaluate APIRubrics and evidenceReading scores
History
WorkbenchGolden data
CoverageCoverage dashboardCoverage strategiesCoverage advisorTest case quality
RAG AssuranceRAG runsRAG datasetsRAG analytics
Project evalsError analysisUnderstanding scores
Studio guidesDimensionsTuplesConversation modesMulti-lingual testingWorkflowsHuman reviewBest practices
ReferenceAPI and packagesTroubleshooting
Coverage

Coverage dashboard

Coverage dashboard

What is Coverage?

In QAVE Studio, coverage answers one question: "Have we tested everything that matters for this chatbot or agent system?"

Coverage is computed from your approved dimensions, tuples, and test cases — not from documents or uploaded files. This makes it measurable, explainable, and actionable.


The Two-Tab Layout

The Coverage Dashboard has two views:

Scorecard Tab

Your executive summary. Shows:

  • Release Readiness Score — a single percentage (0–100%) that combines all coverage signals
  • 6 Signal Cards — individual health indicators for pairwise coverage, approval rate, language coverage, and more
  • Top Actions — a prioritised list of the most impactful things you can do right now
  • Pipeline Funnel — shows how many dimensions, tuples, cases, and exports you have
  • Focus Area Breakdown — per-focus-area coverage at a glance
  • Coverage Attestation — formal sign-off section for release reviewers

New users: Start here. The Top Actions list tells you exactly what to do next.

Dashboard Tab

Your detailed breakdown. Shows the pairwise heatmap, gap analysis, remediation plan, per-tuple table, language bars, focus area cards, and quality audit panel. Use this when you want to drill into specifics.


The Release Readiness Score

The readiness score is a weighted composite of 5 signals:

SignalWeightWhat it measures
Effective Pairwise Coverage30%Pairs covered by tuples with ≥1 approved test case
Test Case Approval Rate25%Approved cases / total generated cases
Suite Approval Rate20%Approved suites / total suites
Language Coverage15%Best language coverage across all configured languages
Focus Area Completeness10%Dimensions grouped into focus areas / total dimensions

Traffic lights: Green ≥ 95%, Amber ≥ 70%, Red < 70%


Coverage Strategies Explained

Pairwise Coverage

Tests every possible pair of dimension values at least once. This is the most common strategy.

Example: If you have Intent (3 values) and Persona (2 values), pairwise coverage ensures all 6 combinations appear in at least one tuple.

N-wise Coverage

Like pairwise but for N-way combinations. N=3 means every combination of 3 dimension values is covered. More thorough but requires more tuples.

Full Cartesian

Every single combination of all dimension values. Used only for small, high-risk test spaces. Can produce thousands of tuples.


Pipeline Stability

The coverage banner at the top of the page shows the pipeline state:

  • Green (Stable): Coverage is up to date. All runs are idle.
  • Amber (In Flight): A pipeline stage is running (dimensions, tuples, or test suite generation). Metrics will update automatically when it completes.
  • Red (Incomplete): Setup is not complete. You need to generate and approve dimensions before coverage is meaningful.

Important: Coverage metrics can be misleading if shown while the pipeline is running. The amber banner is your signal to wait.


Gap Analysis

A gap is a dimension-value combination that is not covered by any approved tuple.

Reading the Gap List

Each gap shows:

  • The uncovered combination (e.g., Intent=Complaint × Persona=NewUser)
  • A priority (high/medium/low based on dimension importance)
  • An action button — click it to navigate directly to the right page with the context pre-filled

Gap Action Types

ButtonWhat it does
Create TupleOpens Pipeline → Tuples tab with the missing dimension values pre-filled
Generate Test CasesOpens Generate page for that specific tuple
Review CasesOpens Test Suites filtered to pending cases
Assign to Focus AreaOpens Pipeline → Focus Areas tab

Remediation Plan

The Remediation Plan (available in the Dashboard tab under Gap Analysis) uses a greedy set-cover algorithm to recommend the minimum set of new tuples needed to maximise your pairwise coverage.

  • Click the Remediation Plan accordion to expand it
  • It shows how many new tuples to add, what pairs each tuple covers, and the projected coverage gain
  • Click Add next to any tuple in the plan to create it immediately

Example: "Add 3 tuples to go from 62% → 89% pairwise coverage."

This is especially useful when you have many gaps and aren't sure where to start.


Effective Coverage vs. Pairwise Coverage

Pairwise coverage counts tuples regardless of whether test cases have been reviewed.

Effective coverage counts only tuples that have at least one approved test case.

The gap between these two numbers shows your work-in-progress: tuples exist but cases haven't been reviewed yet.

For release readiness, use effective coverage — not pairwise coverage.


Approval Metrics

  • Test Case Approval Rate: What percentage of generated cases have been reviewed and approved?
  • Suite Approval Rate: What percentage of test suites have been formally approved?
  • Per-Tuple Approval: Each tuple shows a color dot (green=all approved, amber=some, red=none, gray=no cases)

Focus Areas and Coverage

Focus areas group related dimensions. Coverage is computed per focus area, giving you a more focused view of coverage for each capability or topic area.

Focus Area Completeness measures how many of your approved dimensions are assigned to at least one focus area. Ungrouped dimensions are not included in focus area coverage.

To improve focus area completeness: Go to Pipeline → Focus Areas and assign ungrouped dimensions.


Language Coverage

If your chatbot supports multiple languages, language coverage shows what percentage of tuples have generated test cases in each language.

Click "Generate in [LANG]" to open the Generate page with that language pre-selected.


Coverage Attestation

Attestation is a formal sign-off that certifies your current coverage is sufficient for release. It creates a tamper-evident, timestamped record of the full coverage snapshot.

How to Attest

  1. Go to the Scorecard tab
  2. Scroll to Coverage Attestation at the bottom
  3. Click Attest Coverage (available to organisation Admins and other roles your workspace allows)
  4. Review the readiness score in the confirmation dialog
  5. Optionally add release notes (e.g. "Ready for v2.4 — all critical intents covered")
  6. Click Confirm Attestation

Attestation States

BadgeMeaning
ActiveCurrent sign-off, pipeline unchanged since attestation
StaleCoverage has changed since this attestation (new tuples, cases, or approvals). Re-attest to confirm.
SupersededA newer attestation replaced this one

Downloading Attestation Records

Each row in the attestation history has a download button. This exports the full coverage snapshot as a JSON file for audit purposes.

Who can attest: Typically organisation Admin (and other roles if your workspace enables them).
Who can view: Project members with access to Coverage.


Ask Coverage (AI Chat)

The Ask Coverage button (bottom-right of the page) opens a conversational chat with an AI agent that has live access to your coverage data.

Example questions:

  • "What is my readiness score and what's holding it back?"
  • "Which focus areas have the lowest coverage?"
  • "What are the top 5 gaps I should fix first?"
  • "How many new tuples do I need to reach 80% pairwise coverage?"

How it works: When you send a question, it is processed asynchronously in the background. The answer streams back token-by-token as it is generated — you will see text appear progressively rather than waiting for the full response.

Conversation history is preserved within your session. Use the + button in the chat header to start a fresh conversation.

The assistant uses your live coverage data — it never makes up numbers. It is read-only and cannot create or modify anything.

Note: This is the Ask Coverage conversational chat. For the Domain Coverage Advisor (taxonomy gap analysis card), see the Coverage Advisor Guide.


Frequently Asked Questions

Q: My readiness score is 0%. What do I do? Look at the Top Actions list on the Scorecard tab. It will guide you step by step. Usually the first step is approving dimensions.

Q: Why does pairwise coverage show 100% but effective coverage is lower? Your tuples cover all pairs, but not all test cases have been reviewed and approved yet. Go to Test Suites and review pending cases.

Q: Coverage data seems stale. How do I refresh? Coverage auto-refreshes when pipeline events are received via WebSocket. If you've made changes and don't see them reflected, check the pipeline stability banner. If it shows "in flight", wait for the run to complete.

Q: What does "gap" mean exactly? A gap is a combination of dimension values (one value from each of two dimensions) that no approved tuple covers. Gaps reduce your pairwise coverage percentage.

Q: I approved more test cases. Why hasn't the score updated? Approval events trigger an automatic cache refresh. If you still don't see the update after 30 seconds, use the browser refresh button on the page.

Q: My attestation shows "Stale". Does that mean it's invalid? Stale means the pipeline has changed since you attested (new approvals, tuples, or dimensions). The previous sign-off is still on record, but you should re-attest to confirm the new state is also acceptable.

Q: How is the Remediation Plan computed? It uses a greedy set-cover algorithm: at each step it picks the candidate tuple that covers the most remaining uncovered pairs. The result is not guaranteed to be optimal but is deterministic and runs in milliseconds.

Coverage

Measure release readiness from dimensions, tuples, and approved suites.

Coverage strategies

Coverage strategies

On this page

What is Coverage?The Two-Tab LayoutScorecard TabDashboard TabThe Release Readiness ScoreCoverage Strategies ExplainedPairwise CoverageN-wise CoverageFull CartesianPipeline StabilityGap AnalysisReading the Gap ListGap Action TypesRemediation PlanEffective Coverage vs. Pairwise CoverageApproval MetricsFocus Areas and CoverageLanguage CoverageCoverage AttestationHow to AttestAttestation StatesDownloading Attestation RecordsAsk Coverage (AI Chat)Frequently Asked Questions