LocEssentials logo
LocEssentials

Open Source · Standards-Based · Human-Centered

Translation Quality
Management System

A methodology for producing and using annotated translations, grounded in ASTM F43 international standards, such as the MQM error typology. Annotations are produced in Label Studio and distributed to translators as dynamic feedback pages. Inter-annotator agreement is measured to guide training and ensure consistent, actionable feedback.

Inter-Annotator Agreement MQM · H-Quest Label Studio GNU GPL-2.0

How it works

Annotation Workflow

01

Source Text

Produce a source text for translation in Markdown (.md) format.

02

MD → JSON

Markdown translations are converted to Label Studio-compatible JSON.

03

Annotation

Evaluators mark errors (MQM) and rate holistic quality (H-Quest) in Label Studio.

04

Distribution

Processed JSON annotations display dynamically on a feedback webpage for translators.

05

Agreement

Where multiple evaluators assessed the same text, inter-annotator agreement is measured.

06

Improvement

Analysis guides targeted training to harmonize translator and evaluator performance, and high quality data can be used to train translators.

Sample outputs

Explore the Examples

📄
Feedback Webpage · Folder 5

Translation Evaluations

Dynamic feedback page that displays MQM error annotations as highlighted spans and holistic H-Quest quality scores. Translators use this page to study their errors and track improvement over time.

5_evaluations_for_distribution/translation-evaluations.html Open example
📊
Agreement Report · Folder 6

Quality Reviews - Inter-Annotator Agreement

HTML export of Jupyter notebook analysis showing exact matching, F1 scores for partial matching, and Cohen's Kappa for category agreement across multiple evaluators of the same translation.

6_annotator_agreement/reports/QUALITY REVIEWS_report_2026-03-19.html Open report

Background

Standards & Approach

Standards Foundation

This system is grounded in three ASTM F43 standards:

  • F2575 — Standard Practice for Language Translation
  • WK46396 — MQM Analytic Evaluation (draft)
  • WK54884 — H-Quest Holistic Evaluation (draft)

Why Whole-Text Annotation?

Unlike TEnT-based sentence-by-sentence evaluation, this workflow presents complete texts in Label Studio, enabling contextualized quality assessment. See Brandt (2025) and Post & Junczys-Dowmunt (2023).

Label Studio Setup

GAI Disclaimer

Code in this repo was developed with assistance from Claude GAI models. Please review thoroughly before integrating into your implementation.

Continue Learning

Learn More

LOC401

Translation Quality Management Based on MQM

A professional course covering translation quality management systems grounded in the Multidimensional Quality Metrics (MQM) framework. The course walks through the full evaluation workflow, from annotations completed in Label Studio to distributing structured feedback to translators, while building the skills needed to produce quality, standards-based quality assessments. Learn how to produce consistent annotated data for translator feedback and for training human and machine translation systems

Go to Loc401