QAVE Studio Docs
QAVE Studio Docs
Sign inGet started — the Studio loop
Getting StartedFirst assurance runQuick assure
ConceptsLifecycleGlossaryHow QAVE Studio worksComing soon
Account & workspaceSignup and loginPeopleDemo creditsRolesSettings
Generate simulation suitesContext templatePII and field enhanceDimensionsAgents and plannerDraft review and Save
ExportExport formatsExport formats (detail)
Assurance SDKSDK overviewQAVE Studio suite fetch APIAgent and JSONL queueLangfuse bridge
Execute and tracesRecipe chatbot (reference SUT)TracingFetch tracing
AssureEvaluate APIRubrics and evidenceReading scores
History
WorkbenchGolden data
CoverageCoverage dashboardCoverage strategiesCoverage advisorTest case quality
RAG AssuranceRAG runsRAG datasetsRAG analytics
Project evalsError analysisUnderstanding scores
Studio guidesDimensionsTuplesConversation modesMulti-lingual testingWorkflowsHuman reviewBest practices
ReferenceAPI and packagesTroubleshooting
Coverage

Test case quality

Test case quality

What is Quality in QAVE Studio?

Quality measures whether your generated test cases are actually useful — not just whether they exist.

You can have 95% pairwise coverage with poor-quality test cases. Quality is a separate signal from coverage on purpose: it prevents teams from gaming the system by generating low-effort, repetitive cases just to fill coverage gaps.


Why Quality Matters for Chatbot Testing

Every project in QAVE Studio represents a chatbot, agent, or multi-agent system. Test cases must:

  1. Reflect real user interactions — not artificial or generic conversations
  2. Test the right things — aligned with your system's declared use cases and failure scenarios
  3. Be diverse — not just rephrased versions of the same scenario
  4. Be structurally correct — right format, right number of turns, right language

Low-quality test cases will pass generation but fail to catch real issues in your chatbot.


Quality Rubrics

Quality is measured across 5 rubrics:

R1: Structural Validity (20%)

Does the conversation format match what was requested?

  • Single-turn cases should have exactly 1 user message
  • Multi-turn cases should have the correct number of turns
  • Role sequence should be correct (user/assistant alternating)

R2: Language Match (10%)

Is the test case written in the requested language?

  • For non-English languages, are the right characters/scripts used?
  • Is the language consistent throughout the conversation?

R3: Tuple Fidelity (35%)

Does the test case actually test what its tuple says it should?

  • A case assigned to Intent=Complaint + Persona=NewUser should contain a complaint from a new user
  • This is measured using semantic similarity — not exact keyword matching
  • This is the most important rubric — a case that doesn't match its tuple is testing the wrong thing

R4: System Alignment (25%)

Is the test case consistent with your declared system description?

  • Does the scenario fit within the declared use cases?
  • Are the user requests plausible for your system's declared target users?
  • This rubric uses AI to evaluate against your Context Template

R5: Diversity (10%)

Are the test cases in the suite diverse enough?

  • High similarity between cases means you're testing the same thing multiple times
  • Low diversity wastes generation capacity
  • Measured using embedding distance between cases

Quality Score Interpretation

ScoreMeaningRecommended Action
≥ 90%ExcellentReady for release
70–89%GoodMinor improvements possible
50–69%FairReview low-scoring cases and regenerate weak suites
< 50%PoorConsider regenerating the suite with updated generation guidance

How to Trigger a Quality Audit

Quality audits are available in two places:

  1. From the Coverage Dashboard: In the Dashboard tab, scroll to the bottom. A Quality Audit panel appears for each test suite. Click Run Audit to start one.
  2. Suite selector: If you have multiple suites, use the suite buttons above the panel to switch between them.

Audits run asynchronously. The panel polls automatically every 4 seconds while an audit is running and updates when complete. Results include:

  • An overall score and per-rubric breakdown
  • A bar chart for each rubric
  • An Issues list of specific cases that scored low, with the rubric and explanation

Quality vs. Coverage

These are independent signals:

SignalWhat it measuresWhere shown
Pairwise CoverageAre all dimension pairs tested?Coverage Scorecard — Signal Card
Effective CoverageAre all pairs backed by approved cases?Coverage Scorecard — Signal Card
Quality ScoreAre the test cases actually good?Coverage Scorecard — Quality Signal Card

Release readiness requires both:

  • Good coverage ensures breadth (we tested all the things)
  • Good quality ensures depth (we tested them properly)

Improving Quality

If R1 (Structural Validity) is low:

  • Check the conversation format setting in your generation runs
  • If using multi-turn, ensure num_turns is set correctly

If R3 (Tuple Fidelity) is low:

  • Review the dimension values in low-scoring tuples — are they clear and specific enough?
  • Add generation guidance to steer the model toward more specific scenarios
  • Consider splitting vague dimension values into more precise ones

If R4 (System Alignment) is low:

  • Review your Context Template — especially the ai_system_description and core_use_cases fields
  • Are these specific enough? Vague descriptions produce misaligned test cases
  • Update the template and regenerate if necessary

If R5 (Diversity) is low:

  • Try increasing the number of tuples before regenerating
  • Use different tuples for different test scenarios
  • Consider using N-wise coverage strategy instead of pairwise to create more diverse combinations

Frequently Asked Questions

Q: Do I need to run quality audits before I can see coverage? No. Coverage and quality are independent. You can see coverage immediately after test cases are generated. Quality audits are optional but recommended before release.

Q: Will poor quality block me from exporting? No. Quality scores are advisory. You can export test cases regardless of quality score. However, we recommend reviewing low-quality cases before export.

Q: How often should I run quality audits? After each generation run. Also run before attestation if the suite has changed since the last audit. Only one audit can run at a time per suite — if you try to run a second while one is active you'll see a "already running" message.

Q: Can I run audits on multiple suites at once? Yes. Each suite runs its own independent audit. There is no global lock — only one audit per suite at a time.

Q: My quality score dropped after I added more dimensions. Why? More dimensions create more tuples, and new tuples may have less specific generation guidance. Add generation guidance to new tuples and regenerate to improve quality.

Coverage advisor

Coverage advisor

RAG Assurance

Score retrieval quality with runs, datasets, and analytics.

On this page

What is Quality in QAVE Studio?Why Quality Matters for Chatbot TestingQuality RubricsR1: Structural Validity (20%)R2: Language Match (10%)R3: Tuple Fidelity (35%)R4: System Alignment (25%)R5: Diversity (10%)Quality Score InterpretationHow to Trigger a Quality AuditQuality vs. CoverageImproving QualityIf R1 (Structural Validity) is low:If R3 (Tuple Fidelity) is low:If R4 (System Alignment) is low:If R5 (Diversity) is low:Frequently Asked Questions