Multi-lingual testing
Multi-lingual testing
QAVE Studio supports generating test cases in multiple languages. This guide explains how to set up multi-lingual testing and interpret language coverage data.
Setting Primary Languages
In the Context Template, the Primary Languages field accepts a comma-separated list of ISO 639-1 language codes.
Example:
en, hi, ar, es, frThese become the target languages for test generation and are tracked in the Coverage Dashboard's language coverage section.
Supported ISO 639-1 codes (examples):
| Code | Language |
|---|---|
en | English |
hi | Hindi |
ar | Arabic |
es | Spanish |
fr | French |
pt | Portuguese |
de | German |
zh | Chinese (Mandarin) |
ja | Japanese |
ko | Korean |
ta | Tamil |
te | Telugu |
mr | Marathi |
bn | Bengali |
Dimension Value Translations
For multi-lingual projects, you can add translated values to each dimension. These translations are used to constrain the generated conversations in the target language.
Adding a translation:
- Open a dimension in the Dimensions page
- Click on a value (e.g., "Frustrated")
- Click Add Translation
- Select the language code and enter the translated value (e.g.,
hi→ "निराश")
When QAVE Studio generates a test case for the Hindi language, it uses "निराश" instead of "Frustrated" in the dimension constraint, helping the LLM generate more natural conversations.
Language Selector on the Generate Page
On the Generate page, use the Language dropdown to select the output language for this generation run.
- If you select
hi(Hindi), all generated conversations will be in Hindi - The dimension value translations for Hindi will be used in the generation prompt
- The generated test cases will have
language: "hi"in their metadata
Recommended workflow for multi-lingual coverage:
- Generate English test cases first (baseline coverage)
- Check the Coverage Dashboard → Language Coverage section
- Use "Generate in [language]" shortcuts for each uncovered language
- Repeat until target language coverage is achieved
How Language Affects Generated Conversations
When a non-English language is selected:
- The generation prompt includes a language instruction: "Generate this conversation entirely in [Language Name]."
- If dimension value translations are available for that language, they are included in the tuple constraint
- The LLM generates the full conversation (user and bot messages) in the target language
- The test case's
languagefield is set to the ISO 639-1 code
Note: The expected_behavior and criteria fields are always generated in English for consistency in evaluation, regardless of the conversation language.
Coverage Across Languages
The Coverage Dashboard tracks language coverage:
- A progress bar shows test case counts per configured language
- Languages with 0 test cases are highlighted as gaps
- Each language bar has a "Generate in [language]" shortcut that takes you to the Generate page with that language pre-selected
Target: All languages listed in the Context Template's primary_languages field should have test cases.
Language Coverage vs. Pairwise Coverage
Language coverage and pairwise coverage are tracked separately:
- Pairwise coverage is calculated across all tuples regardless of language
- Language coverage tracks which languages have test cases
To achieve both full pairwise coverage AND language coverage, you need to generate test cases for each language across your approved tuples. The Coverage Dashboard shows both metrics independently.
Troubleshooting Multi-Lingual Generation
Issue: Generated conversations mix languages (code-switching)
- Ensure dimension value translations are added for the target language
- Increase the specificity of the generation focus: "Generate this conversation entirely in [language]. Do not use any English."
Issue: Test cases for a language have poor quality
- Check that the configured model in
DIMENSION_GENERATOR_MODELsupports the target language - For Indic languages (Hindi, Tamil, etc.), prefer models with strong Indic support
- For Arabic, prefer models with Arabic language support
Issue: Language not showing up in Coverage Dashboard
- Verify the language code is listed in the Context Template's
primary_languagesfield - Only languages explicitly listed in
primary_languagesappear in the language coverage section