QAVE Studio Docs
QAVE Studio Docs
Sign inGet started — the Studio loop
Getting StartedFirst assurance runQuick assure
ConceptsLifecycleGlossaryHow QAVE Studio worksComing soon
Account & workspaceSignup and loginPeopleDemo creditsRolesSettings
Generate simulation suitesContext templatePII and field enhanceDimensionsAgents and plannerDraft review and Save
ExportExport formatsExport formats (detail)
Assurance SDKSDK overviewQAVE Studio suite fetch APIAgent and JSONL queueLangfuse bridge
Execute and tracesRecipe chatbot (reference SUT)TracingFetch tracing
AssureEvaluate APIRubrics and evidenceReading scores
History
WorkbenchGolden data
CoverageCoverage dashboardCoverage strategiesCoverage advisorTest case quality
RAG AssuranceRAG runsRAG datasetsRAG analytics
Project evalsError analysisUnderstanding scores
Studio guidesDimensionsTuplesConversation modesMulti-lingual testingWorkflowsHuman reviewBest practices
ReferenceAPI and packagesTroubleshooting
Project evals & analysis

Understanding scores

Understanding scores

QAVE Studio scores conversations and traces so you can see how well the system under test performed. This guide explains how scores work and what they mean.

Overview

When test cases are executed against your AI system, Qurator evaluates the responses and assigns scores. These scores help you understand:

  • Quality: How well your system performs
  • Confidence: Reliability of the evaluation
  • Issues: Areas that need attention

Score Components

Overall Score

The overall score is a number between 0.0 and 1.0 (or 0-100%) representing:

  • 1.0 (100%): Perfect match with expected behavior
  • 0.5 (50%): Partial correctness or mixed results
  • 0.0 (0%): Complete failure or incorrect behavior

Score Breakdown

Qurator provides detailed breakdowns:

  • Relevance: How relevant the response is to the input
  • Correctness: Whether the response matches expected behavior
  • Completeness: Whether all required information is present
  • Safety: Whether the response avoids harmful content
  • Language Quality: Grammar, clarity, and naturalness

Score Interpretation

High Scores (0.8 - 1.0)

  • Green indicators in the UI
  • System is performing well
  • May still benefit from fine-tuning for edge cases

Medium Scores (0.5 - 0.8)

  • Yellow indicators in the UI
  • System needs improvement
  • Review reasoning to identify specific issues

Low Scores (0.0 - 0.5)

  • Red indicators in the UI
  • System is failing or producing incorrect results
  • Priority for review and fixes

Confidence Levels

Qurator also provides confidence scores:

  • High Confidence: Evaluation is reliable, score is accurate
  • Medium Confidence: Some uncertainty, review recommended
  • Low Confidence: Evaluation may be unreliable, manual review needed

Low confidence scores often indicate:

  • Ambiguous test cases
  • Edge cases not well-defined
  • Evaluation criteria conflicts

Reasoning & Explanation

Each score includes reasoning text explaining:

  • What was evaluated: Specific aspects checked
  • Why the score: What led to the score assignment
  • What to improve: Actionable feedback

Example Reasoning

"The response correctly identifies the user's intent (booking a flight) but fails to extract the departure date. The system should prompt for missing required information rather than proceeding with incomplete data."

Score Trends

Track scores over time to identify:

  • Improvements: Scores increasing after model updates
  • Regressions: Scores decreasing, indicating issues
  • Stability: Consistent performance across test runs

Quality Gates

QAVE Studio uses scores to enforce quality gates:

  • Minimum Score Threshold: Reject exports below a certain score
  • Average Score Target: Ensure overall quality meets standards
  • Failure Rate Limits: Control percentage of failing tests

Quality gate failures:

  • Trigger notifications
  • Prevent export (if configured)
  • Create audit events for tracking

Review Workflow

1. Automated Evaluation

Qurator automatically evaluates all test executions and assigns scores.

2. Low Confidence Queue

Tests with low confidence scores appear in a review queue for manual validation.

3. Human Review

Reviewers can:

  • Thumbs Up: Confirm the score is correct
  • Thumbs Down: Disagree and provide feedback
  • Adjust Score: Override with manual score

4. Feedback Loop

Human feedback flows back to Qurator to:

  • Improve evaluation accuracy
  • Refine scoring criteria
  • Train better evaluators

Score Filtering

Filter and search by:

  • Score Range: Show only high/low/medium scores
  • Confidence Level: Focus on reliable evaluations
  • Reasoning Keywords: Find specific issues
  • Date Range: Track changes over time

Generation review scores

Dimension uniqueness

When QAVE Studio proposes dimensions, it also estimates whether a proposal is a duplicate:

RangeMeaningWhat you see
LowLikely uniqueAvailable to approve
MidUncertainShown for human comparison
HighLikely duplicateUsually filtered out automatically

Tuple quality

Tuple quality estimates how useful a dimension-value combination is as a test scenario:

RangeMeaningDisplay
HighStrong test valueEmphasised in review
MediumMixed valueShown with a quality indicator
LowWeak / redundantOften hidden by default

A low quality score is a heuristic. Keep a low-scoring tuple if it covers a known risk from your context template.


Best Practices

  1. Review Low Scores First: Focus on failures and edge cases
  2. Check Reasoning: Understand why scores were assigned
  3. Validate High Scores: Ensure they're not false positives
  4. Monitor Trends: Track improvements and regressions
  5. Provide Feedback: Help improve evaluation quality

Integration with QAVE Studio

Scores are integrated throughout Q:

  • Test Suite Review: See scores when reviewing generated tests
  • Export Filtering: Filter exports by score thresholds
  • Dashboard Metrics: Aggregate scores for project overview
  • Activity Feed: Track score updates in real-time

Getting Help

If scores seem incorrect:

  1. Check the reasoning text for explanation
  2. Review the test case input and expected output
  3. Validate against your business rules
  4. Provide feedback via the review interface
  5. Contact support if issues persist

Error analysis

Annotate runs or uploads, then generate project failure modes.

Studio guides

Deep how-tos for dimensions, tuples, modes, workflows, and review.

On this page

OverviewScore ComponentsOverall ScoreScore BreakdownScore InterpretationHigh Scores (0.8 - 1.0)Medium Scores (0.5 - 0.8)Low Scores (0.0 - 0.5)Confidence LevelsReasoning & ExplanationExample ReasoningScore TrendsQuality GatesReview Workflow1. Automated Evaluation2. Low Confidence Queue3. Human Review4. Feedback LoopScore FilteringGeneration review scoresDimension uniquenessTuple qualityBest PracticesIntegration with QAVE StudioGetting Help