Test case quality
Test case quality
What is Quality in QAVE Studio?
Quality measures whether your generated test cases are actually useful — not just whether they exist.
You can have 95% pairwise coverage with poor-quality test cases. Quality is a separate signal from coverage on purpose: it prevents teams from gaming the system by generating low-effort, repetitive cases just to fill coverage gaps.
Why Quality Matters for Chatbot Testing
Every project in QAVE Studio represents a chatbot, agent, or multi-agent system. Test cases must:
- Reflect real user interactions — not artificial or generic conversations
- Test the right things — aligned with your system's declared use cases and failure scenarios
- Be diverse — not just rephrased versions of the same scenario
- Be structurally correct — right format, right number of turns, right language
Low-quality test cases will pass generation but fail to catch real issues in your chatbot.
Quality Rubrics
Quality is measured across 5 rubrics:
R1: Structural Validity (20%)
Does the conversation format match what was requested?
- Single-turn cases should have exactly 1 user message
- Multi-turn cases should have the correct number of turns
- Role sequence should be correct (user/assistant alternating)
R2: Language Match (10%)
Is the test case written in the requested language?
- For non-English languages, are the right characters/scripts used?
- Is the language consistent throughout the conversation?
R3: Tuple Fidelity (35%)
Does the test case actually test what its tuple says it should?
- A case assigned to Intent=Complaint + Persona=NewUser should contain a complaint from a new user
- This is measured using semantic similarity — not exact keyword matching
- This is the most important rubric — a case that doesn't match its tuple is testing the wrong thing
R4: System Alignment (25%)
Is the test case consistent with your declared system description?
- Does the scenario fit within the declared use cases?
- Are the user requests plausible for your system's declared target users?
- This rubric uses AI to evaluate against your Context Template
R5: Diversity (10%)
Are the test cases in the suite diverse enough?
- High similarity between cases means you're testing the same thing multiple times
- Low diversity wastes generation capacity
- Measured using embedding distance between cases
Quality Score Interpretation
| Score | Meaning | Recommended Action |
|---|---|---|
| ≥ 90% | Excellent | Ready for release |
| 70–89% | Good | Minor improvements possible |
| 50–69% | Fair | Review low-scoring cases and regenerate weak suites |
| < 50% | Poor | Consider regenerating the suite with updated generation guidance |
How to Trigger a Quality Audit
Quality audits are available in two places:
- From the Coverage Dashboard: In the Dashboard tab, scroll to the bottom. A Quality Audit panel appears for each test suite. Click Run Audit to start one.
- Suite selector: If you have multiple suites, use the suite buttons above the panel to switch between them.
Audits run asynchronously. The panel polls automatically every 4 seconds while an audit is running and updates when complete. Results include:
- An overall score and per-rubric breakdown
- A bar chart for each rubric
- An Issues list of specific cases that scored low, with the rubric and explanation
Quality vs. Coverage
These are independent signals:
| Signal | What it measures | Where shown |
|---|---|---|
| Pairwise Coverage | Are all dimension pairs tested? | Coverage Scorecard — Signal Card |
| Effective Coverage | Are all pairs backed by approved cases? | Coverage Scorecard — Signal Card |
| Quality Score | Are the test cases actually good? | Coverage Scorecard — Quality Signal Card |
Release readiness requires both:
- Good coverage ensures breadth (we tested all the things)
- Good quality ensures depth (we tested them properly)
Improving Quality
If R1 (Structural Validity) is low:
- Check the conversation format setting in your generation runs
- If using multi-turn, ensure
num_turnsis set correctly
If R3 (Tuple Fidelity) is low:
- Review the dimension values in low-scoring tuples — are they clear and specific enough?
- Add generation guidance to steer the model toward more specific scenarios
- Consider splitting vague dimension values into more precise ones
If R4 (System Alignment) is low:
- Review your Context Template — especially the
ai_system_descriptionandcore_use_casesfields - Are these specific enough? Vague descriptions produce misaligned test cases
- Update the template and regenerate if necessary
If R5 (Diversity) is low:
- Try increasing the number of tuples before regenerating
- Use different tuples for different test scenarios
- Consider using N-wise coverage strategy instead of pairwise to create more diverse combinations
Frequently Asked Questions
Q: Do I need to run quality audits before I can see coverage? No. Coverage and quality are independent. You can see coverage immediately after test cases are generated. Quality audits are optional but recommended before release.
Q: Will poor quality block me from exporting? No. Quality scores are advisory. You can export test cases regardless of quality score. However, we recommend reviewing low-quality cases before export.
Q: How often should I run quality audits? After each generation run. Also run before attestation if the suite has changed since the last audit. Only one audit can run at a time per suite — if you try to run a second while one is active you'll see a "already running" message.
Q: Can I run audits on multiple suites at once? Yes. Each suite runs its own independent audit. There is no global lock — only one audit per suite at a time.
Q: My quality score dropped after I added more dimensions. Why? More dimensions create more tuples, and new tuples may have less specific generation guidance. Add generation guidance to new tuples and regenerate to improve quality.