BOUQuET

Links and references

• An interface for contributing new volunteer translations: https://bouquet.metademolab.com

• A paper by Omnilingual Team, 2025 with the original design on the dataset: https://aclanthology.org/2025.emnlp-main.1400,

• A paper by Omnilingual Team, 2026 with its extension to more languages: https://arxiv.org/abs/2603.16309

• A leaderboard of machine translation systems based on BOUQuET: https://huggingface.co/spaces/facebook/bouquet

• Code for evaluating new models : https://github.com/facebookresearch/bouquet

• A mirror version of the dataset, with additional configurations and Met-BOUQuET data (human judgements of machine translation quality): https://huggingface.co/datasets/facebook/bouquet

• A dataset of human annotations of sentence translation difficulty for the Spanish BOUQuET: see here

Uses

The dataset is intended for evaluation of machine translation quality. By purpose, it is similar to FLORES+ or WMT24++. Unlike these datasets, BOUQuET focuses more on linguistic diversity, both across languages (including some extremely low-resourced languages) and within a language (covering different registers).

The base BOUQuET dataset is not intended as a training dataset, but the dev subset may be used for validation during model development.

As default evaluation metrics, we recommend ChrF++ and MetricX. For difficult target languages, mode-based metrics like MetricX should be adjusted with LID scores (e.g. by GlotLID) to penalize off-target translations. Our evaluation codebase contains the recommended implementations of the evaluation metrics.

Dataset composition

BOUQuET consists of short paragraphs, fully parallel in all languages at the sentence level. The dataset is distributed both at the sentence level and at the paragraph level. By default, data with both levels is loaded; the paragraph_level and sentence_level configs may be used to load the levels separately.

The public portion of the dataset contains two splits:

• dev: 504 unique sentences, 120 paragraphs

• test: 854 unique sentences, 198 paragraphs

An additional split made up of 632 unique sentences and 144 paragraphs is being held out for quality assurance purposes and is not distributed here.

Mozilla contributions

A part of the BOUQuET dataset has been contributed by the Mozilla Foundation. This parts consist of translations into the following 16 language varieties: azz_Latn, bas_Latn, bft_Arab, brh_Arab, bsh_Arab, cak_Latn, chv_Cyrl, cux_Latn, dua_Latn, eto_Latn, kls_Arab, ksf_Latn, kxp_Arab, skr_Arab, tui_Latn, ydg_Arab.

We are extremely grateful to our Mozilla Foundation partners for this collaboration!

Data columns

The BOUQuET dataset contains the following fields:

• level (str): "sentence_level" (paragraph-level texts can be reconstructed by joining the sentences of each paragraph with a whitespace or a newline, as per the newline_next column)

• split (str): "dev" or "test"

• uniq_id (str): identifier of the dataset item (e.g. P464-S1 for sentence-level, P464 for paragraph-level data)

• src_lang (str): NLLB-compatible non-English language code (such as hin_Deva)

• tgt_lang (str): always eng_Latn (but because the data is multiway parallel, any two of languages can be paired as a source and target instead)

• src_text (str): non-English text

• tgt_text (str): English text

• orig_text (str): the original text (sentence or paragraph), which sometimes corresponds to src_text

• par_comment (str): comment to the whole paragraph

• newline_next (bool): whether the sentence should be followed by a newline in the paragraph

• par_id (str): paragraph id (e.g. P464)

• domain (str): one of the 8 domains (see their list below)

• register (str): three-letter identifier of the register of the original sentence (see the paper for the explanation of their meaning)

• tags (str): comma-separated linguistic tags of a sentence (see the paper for more details)

Languages

The language codes used in the dataset always include consist the three-letter ISO 639-3 language code (e.g. "eng") and the four-letter ISO 15924 writing system code (e.g. "Latn"). Optionally, they may also include a 8-letter Glottolog code to define a more fine-grained dialect (as of now, this is applied only for Brazilian Portuguese).

Currently, BOUQuET covers 275 language varieties (more will be added later): 8 source ("pivot") languages + English + 266 added languoids (including two donated by community).

List of languages

Dataset Creation

The base dataset has been created manually from scratch by professional linguists, by composing the source sentences that cover a variety of domains and registers in 8 diverse non-English languages: Egyptian Arabic (alternating with Modern Standard Arabic when appropriate), French, German, Hindi, Indonesian, Mandarin Chinese, Russian, and Spanish.

For each of the source languages, the sentences have been created in the following 8 domains:

1. How-to, written tutorials or instructions

2. Conversations (dialogues)

3. Narration (creative writing that doesn’t include dialogues)

4. Social media posts

5. Social media comments (reactive)

6. Other web content

7. Reflective piece

8. Miscellaneous (address to a nation, disaster response, etc.)

Apart from the domains, a variety of registers (contextual styles) were used. Each sentence is annotated with the register characterized by three features: connectedness, preparedness, and social differential.

The linguists who were creating the dataset were instructed to maintain the diversity of sentence lengths, word orders, sentence structures, and other linguistic characteristics.

Subsequently, the source sentences were translated from the 8 source languages into English, and then, into the other languages.

See the paper for more details.

Contribution

To contribute to the dataset (adding translations for a new language, or verifying some of the existing translations), please use the web annotation tool at https://bouquet.metademolab.com. Please contact us in case of any questions!

Citation

If you are referring to this dataset, please cite the BOUQuET paper and the Omnilingual MT paper.

@inproceedings{andrews-etal-2025-bouquet,
    title = "{BOUQ}u{ET} : dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translation",
    author = "Andrews, Pierre  and
      Artetxe, Mikel  and
      Meglioli, Mariano Coria  and
      Costa-juss{\`a}, Marta R.  and
      Chuang, Joe  and
      Dale, David  and
      Duppenthaler, Mark  and
      Ekberg, Nathanial Paul  and
      Gao, Cynthia  and
      Licht, Daniel Edward  and
      Maillard, Jean  and
      Mourachko, Alexandre  and
      Ropers, Christophe  and
      Saleem, Safiyyah  and
      S{\'a}nchez, Eduardo  and
      Tsiamas, Ioannis  and
      Turkatenko, Arina  and
      Ventayol-Boada, Albert  and
      Yates, Shireen",
    editor = "Christodoulopoulos, Christos  and
      Chakraborty, Tanmoy  and
      Rose, Carolyn  and
      Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    m nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.1400/",
    doi = "10.18653/v1/2025.emnlp-main.1400",
    pages = "27515--27535",
    ISBN = "979-8-89176-332-6",
}

@misc{omnilingual2026,
    title={Omnilingual {MT}: Machine Translation for 1,600 Languages},
    author={The Omnilingual MT Team and Belen Alastruey and Niyati Bafna and Andrea Caciolai and Kevin Heffernan and Artyom Kozhevnikov and Christophe Ropers and Eduardo S{\'a}nchez and Charles-Eric Saint-James and Ioannis Tsiamas and Chierh Cheng and Joe Chuang and Paul-Ambroise Duquenne and Mark Duppenthaler and Nate Ekberg and Cynthia Gao and Pere Llu{\'i}s Huguet Cabot and Jo{\~a}o Maria Janeiro and Jean Maillard and Gabriel Mejia Gonzalez and Holger Schwenk and Edan Toledo and Arina Turkatenko and Albert Ventayol-Boada and Rashel Moritz and Alexandre Mourachko and Surya Parimi and Mary Williamson and Shireen Yates and David Dale and Marta R. Costa-juss{\`a}},
    year={2026},
    eprint={},
    archivePrefix={arXiv},
    primaryClass={cs.CL},
    url={https://arxiv.org/abs/2603.16309},
}

Glossary

• Domain. By the term domain, we mean different spaces in which language is produced in speech, sign, or writing (e.g., books, social media, news, Wikipedia, organization websites, official documents, direct messaging, texting). In this dataset, we focus solely on the written modality.

• Register. We understand the term register as a functional variety of language that includes socio-semiotic properties, as expressed in [Halliday and Matthiessen (2004)], or more simply as a "contextual style", as presented in [Labov (1991), pp.79–99]. In that regard, a register is a specific variety of language used to best fit a specific communicative purpose in a specific situation.