Understanding scores
Understanding scores
QAVE Studio scores conversations and traces so you can see how well the system under test performed. This guide explains how scores work and what they mean.
Overview
When test cases are executed against your AI system, Qurator evaluates the responses and assigns scores. These scores help you understand:
- Quality: How well your system performs
- Confidence: Reliability of the evaluation
- Issues: Areas that need attention
Score Components
Overall Score
The overall score is a number between 0.0 and 1.0 (or 0-100%) representing:
- 1.0 (100%): Perfect match with expected behavior
- 0.5 (50%): Partial correctness or mixed results
- 0.0 (0%): Complete failure or incorrect behavior
Score Breakdown
Qurator provides detailed breakdowns:
- Relevance: How relevant the response is to the input
- Correctness: Whether the response matches expected behavior
- Completeness: Whether all required information is present
- Safety: Whether the response avoids harmful content
- Language Quality: Grammar, clarity, and naturalness
Score Interpretation
High Scores (0.8 - 1.0)
- Green indicators in the UI
- System is performing well
- May still benefit from fine-tuning for edge cases
Medium Scores (0.5 - 0.8)
- Yellow indicators in the UI
- System needs improvement
- Review reasoning to identify specific issues
Low Scores (0.0 - 0.5)
- Red indicators in the UI
- System is failing or producing incorrect results
- Priority for review and fixes
Confidence Levels
Qurator also provides confidence scores:
- High Confidence: Evaluation is reliable, score is accurate
- Medium Confidence: Some uncertainty, review recommended
- Low Confidence: Evaluation may be unreliable, manual review needed
Low confidence scores often indicate:
- Ambiguous test cases
- Edge cases not well-defined
- Evaluation criteria conflicts
Reasoning & Explanation
Each score includes reasoning text explaining:
- What was evaluated: Specific aspects checked
- Why the score: What led to the score assignment
- What to improve: Actionable feedback
Example Reasoning
"The response correctly identifies the user's intent (booking a flight) but fails to extract the departure date. The system should prompt for missing required information rather than proceeding with incomplete data."
Score Trends
Track scores over time to identify:
- Improvements: Scores increasing after model updates
- Regressions: Scores decreasing, indicating issues
- Stability: Consistent performance across test runs
Quality Gates
QAVE Studio uses scores to enforce quality gates:
- Minimum Score Threshold: Reject exports below a certain score
- Average Score Target: Ensure overall quality meets standards
- Failure Rate Limits: Control percentage of failing tests
Quality gate failures:
- Trigger notifications
- Prevent export (if configured)
- Create audit events for tracking
Review Workflow
1. Automated Evaluation
Qurator automatically evaluates all test executions and assigns scores.
2. Low Confidence Queue
Tests with low confidence scores appear in a review queue for manual validation.
3. Human Review
Reviewers can:
- Thumbs Up: Confirm the score is correct
- Thumbs Down: Disagree and provide feedback
- Adjust Score: Override with manual score
4. Feedback Loop
Human feedback flows back to Qurator to:
- Improve evaluation accuracy
- Refine scoring criteria
- Train better evaluators
Score Filtering
Filter and search by:
- Score Range: Show only high/low/medium scores
- Confidence Level: Focus on reliable evaluations
- Reasoning Keywords: Find specific issues
- Date Range: Track changes over time
Generation review scores
Dimension uniqueness
When QAVE Studio proposes dimensions, it also estimates whether a proposal is a duplicate:
| Range | Meaning | What you see |
|---|---|---|
| Low | Likely unique | Available to approve |
| Mid | Uncertain | Shown for human comparison |
| High | Likely duplicate | Usually filtered out automatically |
Tuple quality
Tuple quality estimates how useful a dimension-value combination is as a test scenario:
| Range | Meaning | Display |
|---|---|---|
| High | Strong test value | Emphasised in review |
| Medium | Mixed value | Shown with a quality indicator |
| Low | Weak / redundant | Often hidden by default |
A low quality score is a heuristic. Keep a low-scoring tuple if it covers a known risk from your context template.
Best Practices
- Review Low Scores First: Focus on failures and edge cases
- Check Reasoning: Understand why scores were assigned
- Validate High Scores: Ensure they're not false positives
- Monitor Trends: Track improvements and regressions
- Provide Feedback: Help improve evaluation quality
Integration with QAVE Studio
Scores are integrated throughout Q:
- Test Suite Review: See scores when reviewing generated tests
- Export Filtering: Filter exports by score thresholds
- Dashboard Metrics: Aggregate scores for project overview
- Activity Feed: Track score updates in real-time
Getting Help
If scores seem incorrect:
- Check the reasoning text for explanation
- Review the test case input and expected output
- Validate against your business rules
- Provide feedback via the review interface
- Contact support if issues persist