Coverage dashboard
Coverage dashboard
What is Coverage?
In QAVE Studio, coverage answers one question: "Have we tested everything that matters for this chatbot or agent system?"
Coverage is computed from your approved dimensions, tuples, and test cases — not from documents or uploaded files. This makes it measurable, explainable, and actionable.
The Two-Tab Layout
The Coverage Dashboard has two views:
Scorecard Tab
Your executive summary. Shows:
- Release Readiness Score — a single percentage (0–100%) that combines all coverage signals
- 6 Signal Cards — individual health indicators for pairwise coverage, approval rate, language coverage, and more
- Top Actions — a prioritised list of the most impactful things you can do right now
- Pipeline Funnel — shows how many dimensions, tuples, cases, and exports you have
- Focus Area Breakdown — per-focus-area coverage at a glance
- Coverage Attestation — formal sign-off section for release reviewers
New users: Start here. The Top Actions list tells you exactly what to do next.
Dashboard Tab
Your detailed breakdown. Shows the pairwise heatmap, gap analysis, remediation plan, per-tuple table, language bars, focus area cards, and quality audit panel. Use this when you want to drill into specifics.
The Release Readiness Score
The readiness score is a weighted composite of 5 signals:
| Signal | Weight | What it measures |
|---|---|---|
| Effective Pairwise Coverage | 30% | Pairs covered by tuples with ≥1 approved test case |
| Test Case Approval Rate | 25% | Approved cases / total generated cases |
| Suite Approval Rate | 20% | Approved suites / total suites |
| Language Coverage | 15% | Best language coverage across all configured languages |
| Focus Area Completeness | 10% | Dimensions grouped into focus areas / total dimensions |
Traffic lights: Green ≥ 95%, Amber ≥ 70%, Red < 70%
Coverage Strategies Explained
Pairwise Coverage
Tests every possible pair of dimension values at least once. This is the most common strategy.
Example: If you have Intent (3 values) and Persona (2 values), pairwise coverage ensures all 6 combinations appear in at least one tuple.
N-wise Coverage
Like pairwise but for N-way combinations. N=3 means every combination of 3 dimension values is covered. More thorough but requires more tuples.
Full Cartesian
Every single combination of all dimension values. Used only for small, high-risk test spaces. Can produce thousands of tuples.
Pipeline Stability
The coverage banner at the top of the page shows the pipeline state:
- Green (Stable): Coverage is up to date. All runs are idle.
- Amber (In Flight): A pipeline stage is running (dimensions, tuples, or test suite generation). Metrics will update automatically when it completes.
- Red (Incomplete): Setup is not complete. You need to generate and approve dimensions before coverage is meaningful.
Important: Coverage metrics can be misleading if shown while the pipeline is running. The amber banner is your signal to wait.
Gap Analysis
A gap is a dimension-value combination that is not covered by any approved tuple.
Reading the Gap List
Each gap shows:
- The uncovered combination (e.g., Intent=Complaint × Persona=NewUser)
- A priority (high/medium/low based on dimension importance)
- An action button — click it to navigate directly to the right page with the context pre-filled
Gap Action Types
| Button | What it does |
|---|---|
| Create Tuple | Opens Pipeline → Tuples tab with the missing dimension values pre-filled |
| Generate Test Cases | Opens Generate page for that specific tuple |
| Review Cases | Opens Test Suites filtered to pending cases |
| Assign to Focus Area | Opens Pipeline → Focus Areas tab |
Remediation Plan
The Remediation Plan (available in the Dashboard tab under Gap Analysis) uses a greedy set-cover algorithm to recommend the minimum set of new tuples needed to maximise your pairwise coverage.
- Click the Remediation Plan accordion to expand it
- It shows how many new tuples to add, what pairs each tuple covers, and the projected coverage gain
- Click Add next to any tuple in the plan to create it immediately
Example: "Add 3 tuples to go from 62% → 89% pairwise coverage."
This is especially useful when you have many gaps and aren't sure where to start.
Effective Coverage vs. Pairwise Coverage
Pairwise coverage counts tuples regardless of whether test cases have been reviewed.
Effective coverage counts only tuples that have at least one approved test case.
The gap between these two numbers shows your work-in-progress: tuples exist but cases haven't been reviewed yet.
For release readiness, use effective coverage — not pairwise coverage.
Approval Metrics
- Test Case Approval Rate: What percentage of generated cases have been reviewed and approved?
- Suite Approval Rate: What percentage of test suites have been formally approved?
- Per-Tuple Approval: Each tuple shows a color dot (green=all approved, amber=some, red=none, gray=no cases)
Focus Areas and Coverage
Focus areas group related dimensions. Coverage is computed per focus area, giving you a more focused view of coverage for each capability or topic area.
Focus Area Completeness measures how many of your approved dimensions are assigned to at least one focus area. Ungrouped dimensions are not included in focus area coverage.
To improve focus area completeness: Go to Pipeline → Focus Areas and assign ungrouped dimensions.
Language Coverage
If your chatbot supports multiple languages, language coverage shows what percentage of tuples have generated test cases in each language.
Click "Generate in [LANG]" to open the Generate page with that language pre-selected.
Coverage Attestation
Attestation is a formal sign-off that certifies your current coverage is sufficient for release. It creates a tamper-evident, timestamped record of the full coverage snapshot.
How to Attest
- Go to the Scorecard tab
- Scroll to Coverage Attestation at the bottom
- Click Attest Coverage (available to organisation Admins and other roles your workspace allows)
- Review the readiness score in the confirmation dialog
- Optionally add release notes (e.g. "Ready for v2.4 — all critical intents covered")
- Click Confirm Attestation
Attestation States
| Badge | Meaning |
|---|---|
| Active | Current sign-off, pipeline unchanged since attestation |
| Stale | Coverage has changed since this attestation (new tuples, cases, or approvals). Re-attest to confirm. |
| Superseded | A newer attestation replaced this one |
Downloading Attestation Records
Each row in the attestation history has a download button. This exports the full coverage snapshot as a JSON file for audit purposes.
Who can attest: Typically organisation Admin (and other roles if your workspace enables them).
Who can view: Project members with access to Coverage.
Ask Coverage (AI Chat)
The Ask Coverage button (bottom-right of the page) opens a conversational chat with an AI agent that has live access to your coverage data.
Example questions:
- "What is my readiness score and what's holding it back?"
- "Which focus areas have the lowest coverage?"
- "What are the top 5 gaps I should fix first?"
- "How many new tuples do I need to reach 80% pairwise coverage?"
How it works: When you send a question, it is processed asynchronously in the background. The answer streams back token-by-token as it is generated — you will see text appear progressively rather than waiting for the full response.
Conversation history is preserved within your session. Use the + button in the chat header to start a fresh conversation.
The assistant uses your live coverage data — it never makes up numbers. It is read-only and cannot create or modify anything.
Note: This is the Ask Coverage conversational chat. For the Domain Coverage Advisor (taxonomy gap analysis card), see the Coverage Advisor Guide.
Frequently Asked Questions
Q: My readiness score is 0%. What do I do? Look at the Top Actions list on the Scorecard tab. It will guide you step by step. Usually the first step is approving dimensions.
Q: Why does pairwise coverage show 100% but effective coverage is lower? Your tuples cover all pairs, but not all test cases have been reviewed and approved yet. Go to Test Suites and review pending cases.
Q: Coverage data seems stale. How do I refresh? Coverage auto-refreshes when pipeline events are received via WebSocket. If you've made changes and don't see them reflected, check the pipeline stability banner. If it shows "in flight", wait for the run to complete.
Q: What does "gap" mean exactly? A gap is a combination of dimension values (one value from each of two dimensions) that no approved tuple covers. Gaps reduce your pairwise coverage percentage.
Q: I approved more test cases. Why hasn't the score updated? Approval events trigger an automatic cache refresh. If you still don't see the update after 30 seconds, use the browser refresh button on the page.
Q: My attestation shows "Stale". Does that mean it's invalid? Stale means the pipeline has changed since you attested (new approvals, tuples, or dimensions). The previous sign-off is still on record, but you should re-attest to confirm the new state is also acceptable.
Q: How is the Remediation Plan computed? It uses a greedy set-cover algorithm: at each step it picks the candidate tuple that covers the most remaining uncovered pairs. The result is not guaranteed to be optimal but is deterministic and runs in milliseconds.