Studio guides
Best practices
Best practices
Follow these best practices to get the most out of QAVE Studio and ensure high-quality test generation.
Context Template Best Practices
AI System Description
- Be specific. Include the deployment channel (chat, voice, email), the business domain (retail, healthcare, insurance), and what the bot can and cannot do.
- Good: "A voice-based appointment scheduling assistant for a dental clinic, handling booking, rescheduling, and cancellation requests. Cannot handle medical questions."
- Avoid: "A customer service chatbot."
Target Users / Personas
- Describe 2–4 distinct user types. Include their technical literacy level and context (mobile vs. desktop, new vs. returning customer).
- Good: "New patient (first visit, unfamiliar with the booking process), Returning patient (existing appointment, wants to reschedule), Elderly patient (low tech literacy, prefers voice guidance)"
Core Use Cases
- List 5–8 primary use cases, not a comprehensive list. Focus on the most tested scenarios.
- Good: Booking a new appointment, rescheduling an existing appointment, cancelling an appointment, checking appointment status, requesting a reminder
Known Limitations / Risks
- This is critical for generating adversarial test cases. Be honest about what the system struggles with.
- Example: "Cannot handle multi-patient scheduling in one call. Sometimes misunderstands accented speech for date references."
Failure Sensitivity
- Use
criticalfor safety-critical outcomes (medical advice, financial transactions) - Use
highfor significant UX failures (booking the wrong date, double-booking) - Use
mediumfor recoverable errors (slow response, incorrect confirmation message)
PII Allowlist
- Add your brand name, product names, and internal terminology here to prevent them from being falsely redacted during PII scrubbing.
- Example: "DentaCare", "SmileFirst Clinic", "DentBot"
Dimension Review Best Practices
- Approve dimensions that represent genuinely different test behaviors. A good dimension creates meaningfully different conversations when its value changes.
- Reject dimensions that are too granular (e.g., "User's Mood at 9am" vs. "Emotional Tone"), too narrow (applies only to one test case), or don't produce different bot responses.
- Use Add Dimension to cover gaps not addressed by generated dimensions. Common gaps: regulatory compliance scenarios, edge-case user behaviors, technical failure modes.
- If dimensions look too similar to each other, check the dedup confidence score — dimensions with high similarity (0.5–0.85) are shown for review; above 0.85 are auto-rejected.
- Promoting to system default (admin only): If a dimension is valuable across multiple projects (e.g., "Emotional Tone" in all customer service bots), promote it to the system default library.
Tuple Selection Best Practices
- Start with pairwise strategy (default) — it covers most defect-revealing combinations without combinatorial explosion. Mathematically, most bugs involve interactions between exactly 2 factors.
- Select high and medium quality tuples for initial generation. Low quality tuples (score < 0.3) are auto-rejected; medium quality (0.3–0.7) can be generated in a follow-up run.
- Generate for English first, then expand to other languages using the language selector on the Generate page.
- Interpret quality scores carefully — a "low quality" tuple might still represent a valid (but less important) test scenario. Review before bulk-rejecting.
- If tuple counts are very high (> 100), consider switching to pairwise strategy. If using n-wise (N=3), reduce the number of dimensions first.
Test Generation Best Practices
Samples Per Tuple
- Start with 2 samples per tuple. Increase to 3–5 for critical tuples that need edge-case coverage.
- Avoid generating too many samples for low-value tuples — review coverage data first.
Generation Focus
- Use generation focus for session-specific guidance that doesn't need to be a permanent dimension.
- Examples:
- "Focus on edge cases where the patient has multiple upcoming appointments"
- "Emphasize scenarios where the bot needs to escalate to a human agent"
- "Generate adversarial cases where the user provides ambiguous date references"
- Keep focus under 500 characters and be specific about scenarios or edge cases.
Conversation Format
- Multi-turn (default): Best for realistic conversational testing (2+ turns)
- Single-turn: Use for imperative command testing or simple Q&A systems
- Q&A: Use for direct question-answer evaluation
Language
- Specify the target language using ISO 639-1 codes (e.g.,
en,hi,ar,es) - When generating in multiple languages, run separate generation rounds and compare coverage in the Coverage Dashboard
Test Suite Review Best Practices
- Group by tuple — review test cases that share a tuple together for consistency
- Check tuple metadata — each test case should reference the correct tuple and dimension values
- Approve high-quality tests — only approve tests where the scenario, expected behavior, and criteria are all correct
- Reject tests with hallucinated context — if the test assumes something not in the context template, reject it
- Bulk actions — use "Approve All" / "Reject All" for suites where quality is clearly high or low
Export Strategy
Format Selection
- JSONL — for Phoenix evaluation, fine-tuning pipelines, general ML use
- Simulator JSONL — for voice simulators and Qurator integration (includes full tuple metadata)
- Pytest — for CI/CD pipelines and regression testing
Suite Selection
- Latest Suite — quick export for immediate use
- All Approved Suites — comprehensive dataset (only APPROVED test cases)
- Specific Suite — when you need a particular generation run
Export After Approval
- Only export suites after reviewing and approving test cases
- Include tuple metadata in downstream pipelines for traceability
Security & Compliance
PII Handling
- The 3-layer pipeline (Presidio → Local LLM → Auditor) ensures no raw PII reaches cloud LLMs
- Add company/product names to the
pii_allowlist_termsfield to prevent false positives - Review scrubbed fields carefully — the auditor layer catches most issues, but human review is the final gate
Project Isolation
- Templates, dimensions, tuples, and test cases are scoped to your project
- Projects are isolated from other organisations
- Qdrant collections are namespaced per project