QAVE Studio Docs
QAVE Studio Docs
Sign inGet started — the Studio loop
Getting StartedFirst assurance runQuick assure
ConceptsLifecycleGlossaryHow QAVE Studio worksComing soon
Account & workspaceSignup and loginPeopleDemo creditsRolesSettings
Generate simulation suitesContext templatePII and field enhanceDimensionsAgents and plannerDraft review and Save
ExportExport formatsExport formats (detail)
Assurance SDKSDK overviewQAVE Studio suite fetch APIAgent and JSONL queueLangfuse bridge
Execute and tracesRecipe chatbot (reference SUT)TracingFetch tracing
AssureEvaluate APIRubrics and evidenceReading scores
History
WorkbenchGolden data
CoverageCoverage dashboardCoverage strategiesCoverage advisorTest case quality
RAG AssuranceRAG runsRAG datasetsRAG analytics
Project evalsError analysisUnderstanding scores
Studio guidesDimensionsTuplesConversation modesMulti-lingual testingWorkflowsHuman reviewBest practices
ReferenceAPI and packagesTroubleshooting
Studio guides

Best practices

Best practices

Follow these best practices to get the most out of QAVE Studio and ensure high-quality test generation.


Context Template Best Practices

AI System Description

  • Be specific. Include the deployment channel (chat, voice, email), the business domain (retail, healthcare, insurance), and what the bot can and cannot do.
  • Good: "A voice-based appointment scheduling assistant for a dental clinic, handling booking, rescheduling, and cancellation requests. Cannot handle medical questions."
  • Avoid: "A customer service chatbot."

Target Users / Personas

  • Describe 2–4 distinct user types. Include their technical literacy level and context (mobile vs. desktop, new vs. returning customer).
  • Good: "New patient (first visit, unfamiliar with the booking process), Returning patient (existing appointment, wants to reschedule), Elderly patient (low tech literacy, prefers voice guidance)"

Core Use Cases

  • List 5–8 primary use cases, not a comprehensive list. Focus on the most tested scenarios.
  • Good: Booking a new appointment, rescheduling an existing appointment, cancelling an appointment, checking appointment status, requesting a reminder

Known Limitations / Risks

  • This is critical for generating adversarial test cases. Be honest about what the system struggles with.
  • Example: "Cannot handle multi-patient scheduling in one call. Sometimes misunderstands accented speech for date references."

Failure Sensitivity

  • Use critical for safety-critical outcomes (medical advice, financial transactions)
  • Use high for significant UX failures (booking the wrong date, double-booking)
  • Use medium for recoverable errors (slow response, incorrect confirmation message)

PII Allowlist

  • Add your brand name, product names, and internal terminology here to prevent them from being falsely redacted during PII scrubbing.
  • Example: "DentaCare", "SmileFirst Clinic", "DentBot"

Dimension Review Best Practices

  • Approve dimensions that represent genuinely different test behaviors. A good dimension creates meaningfully different conversations when its value changes.
  • Reject dimensions that are too granular (e.g., "User's Mood at 9am" vs. "Emotional Tone"), too narrow (applies only to one test case), or don't produce different bot responses.
  • Use Add Dimension to cover gaps not addressed by generated dimensions. Common gaps: regulatory compliance scenarios, edge-case user behaviors, technical failure modes.
  • If dimensions look too similar to each other, check the dedup confidence score — dimensions with high similarity (0.5–0.85) are shown for review; above 0.85 are auto-rejected.
  • Promoting to system default (admin only): If a dimension is valuable across multiple projects (e.g., "Emotional Tone" in all customer service bots), promote it to the system default library.

Tuple Selection Best Practices

  • Start with pairwise strategy (default) — it covers most defect-revealing combinations without combinatorial explosion. Mathematically, most bugs involve interactions between exactly 2 factors.
  • Select high and medium quality tuples for initial generation. Low quality tuples (score < 0.3) are auto-rejected; medium quality (0.3–0.7) can be generated in a follow-up run.
  • Generate for English first, then expand to other languages using the language selector on the Generate page.
  • Interpret quality scores carefully — a "low quality" tuple might still represent a valid (but less important) test scenario. Review before bulk-rejecting.
  • If tuple counts are very high (> 100), consider switching to pairwise strategy. If using n-wise (N=3), reduce the number of dimensions first.

Test Generation Best Practices

Samples Per Tuple

  • Start with 2 samples per tuple. Increase to 3–5 for critical tuples that need edge-case coverage.
  • Avoid generating too many samples for low-value tuples — review coverage data first.

Generation Focus

  • Use generation focus for session-specific guidance that doesn't need to be a permanent dimension.
  • Examples:
    • "Focus on edge cases where the patient has multiple upcoming appointments"
    • "Emphasize scenarios where the bot needs to escalate to a human agent"
    • "Generate adversarial cases where the user provides ambiguous date references"
  • Keep focus under 500 characters and be specific about scenarios or edge cases.

Conversation Format

  • Multi-turn (default): Best for realistic conversational testing (2+ turns)
  • Single-turn: Use for imperative command testing or simple Q&A systems
  • Q&A: Use for direct question-answer evaluation

Language

  • Specify the target language using ISO 639-1 codes (e.g., en, hi, ar, es)
  • When generating in multiple languages, run separate generation rounds and compare coverage in the Coverage Dashboard

Test Suite Review Best Practices

  • Group by tuple — review test cases that share a tuple together for consistency
  • Check tuple metadata — each test case should reference the correct tuple and dimension values
  • Approve high-quality tests — only approve tests where the scenario, expected behavior, and criteria are all correct
  • Reject tests with hallucinated context — if the test assumes something not in the context template, reject it
  • Bulk actions — use "Approve All" / "Reject All" for suites where quality is clearly high or low

Export Strategy

Format Selection

  • JSONL — for Phoenix evaluation, fine-tuning pipelines, general ML use
  • Simulator JSONL — for voice simulators and Qurator integration (includes full tuple metadata)
  • Pytest — for CI/CD pipelines and regression testing

Suite Selection

  • Latest Suite — quick export for immediate use
  • All Approved Suites — comprehensive dataset (only APPROVED test cases)
  • Specific Suite — when you need a particular generation run

Export After Approval

  • Only export suites after reviewing and approving test cases
  • Include tuple metadata in downstream pipelines for traceability

Security & Compliance

PII Handling

  • The 3-layer pipeline (Presidio → Local LLM → Auditor) ensures no raw PII reaches cloud LLMs
  • Add company/product names to the pii_allowlist_terms field to prevent false positives
  • Review scrubbed fields carefully — the auditor layer catches most issues, but human review is the final gate

Project Isolation

  • Templates, dimensions, tuples, and test cases are scoped to your project
  • Projects are isolated from other organisations
  • Qdrant collections are namespaced per project

Human review

Review gates in Setup Wizard and when to use Workbench for SME judgment.

Reference

API names, packages, troubleshooting, and pointers.

On this page

Context Template Best PracticesAI System DescriptionTarget Users / PersonasCore Use CasesKnown Limitations / RisksFailure SensitivityPII AllowlistDimension Review Best PracticesTuple Selection Best PracticesTest Generation Best PracticesSamples Per TupleGeneration FocusConversation FormatLanguageTest Suite Review Best PracticesExport StrategyFormat SelectionSuite SelectionExport After ApprovalSecurity & CompliancePII HandlingProject Isolation