How Good is this Translation?: Automated Measurement of AI Translation Accuracy with Skill Scores for Multilingual Weather Reports

The National Weather Service (NWS) Office of Central Processing is automating the translation of key NWS text products into multiple languages using Artificial Intelligence (AI) language models. These translations are critical to protect life and property of those with low English proficiency.

The NWS translation workflow relies on AI translations with a human-in-the-loop framework to ensure accuracy of meaning and technical jargon. However, how do we ensure translations are accurate when human resources are limited? As the volume and complexity of translated weather data grows, so does the need for automating accuracy checks of the AI to identify problem translations and strategically deploying human resources.

In our work, the accuracy of the AI is measured using various skill scores that are calculated by comparing the original English content to its "reverse translation" (i.e. translating English text into the target language and then back into English for comparison). Cloud-based tools such as AWS S3 and DynamoDB are employed to process the data from logs that are generated as translations occur. These metrics are reported and visualized in a user-friendly dashboard to pinpoint areas where further AI training is needed. Project translators can then focus their efforts only on specific segments that have low skill scores, therefore optimizing the efficiency of continuously training the AI language models. In this talk, we will demonstrate the metric dashboard and discuss strengths and weaknesses of each skill score.