The default advice on Neural Machine Translation (NMT) has become “throw it at GPT-4 and hope for the best.” That works well enough for French to English. It falls apart on a low-resource language like Dongxiang. Fourteen years of wrestling with complex logic has taught me that a general-purpose LLM is not a substitute for touching the architecture yourself.
Some languages are barely acknowledged by mainstream models, and building a translator for one is closer to preservation work than product work. Below is how we fine-tuned Meta’s NLLB-200 (No Language Left Behind) for a minority language, and the Neural Machine Translation bottlenecks that actually cost us time.
Why the standard NMT recipe fails here
The common mistake is assuming more data always means better results. Working through those senior dev insights on applied statistics made the trade-off obvious: in a low-resource setting, noise hurts you more than volume helps. If 30% of your training set is hallucinated text or misaligned Chinese-Dongxiang pairs, the model does not come out slightly less accurate. It comes out broken.
Step 1: cleaning the bilingual dataset
Normalization comes first. Raw scraped text fed into a transformer gives you nothing usable, so the pipeline has to strip extra whitespace and standardize punctuation before anything else. This is the Python preprocessing I use to keep the two scripts apart and cut the noise.
import re
import pandas as pd
def clean_dxg(s: str) -> str:
# Restrict to Latin characters for Dongxiang
s = re.sub(r"[^A-Za-z\s,\.?]", " ", s)
s = re.sub(r"\s+", " ", s).strip()
return s
def clean_zh(s: str) -> str:
# Restrict to Chinese characters for Mandarin
s = re.sub(r"[^\u4e00-\u9fff，。？]", "", s)
return s.strip()
# Naive approach: Just splitting lines.
# Fix: Ensure sentence-level alignment before training.Neural machine translation: the tokenizer trap
Most people assume a new language means retraining the tokenizer from scratch. Don’t. NLLB’s Unigram tokenizer holds up better than you would expect. Before you spend days on SentencePiece, measure subword fertility, the average number of tokens per word. A rate around 1.9 to 2.2 on a new language means the default tokenizer is coping. If it spikes past 10, you have a fragmentation problem.
Step 3: registering the language ID
NLLB wants explicit language tags in src_lang and tgt_lang. A language missing from Meta’s predefined list has no way to be encoded, so you add the token and resize the embedding matrix by hand. Get careless with the index and this is where it breaks.
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
import torch
def bbioon_register_language(model_name, new_lang_code):
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
# Add special token
tokenizer.add_special_tokens({"additional_special_tokens": [new_lang_code]})
model.resize_token_embeddings(len(tokenizer))
# Initialize new embedding with small variance
new_id = tokenizer.convert_tokens_to_ids(new_lang_code)
embed_dim = model.model.shared.weight.size(1)
model.model.shared.weight.data[new_id] = torch.randn(embed_dim) * 0.02
return model, tokenizerStep 4: training with Adafactor
For fine-tuning a transformer on a single GPU, an A100 in our case, I reach for the Adafactor optimizer every time. It skips the full momentum vectors that Adam keeps, so it uses less memory and you can push the batch size higher before CUDA runs out.
A war story: on a similar NMT task I used standard AdamW and spent six hours chasing what I was certain were race conditions. They were memory leaks from the optimizer state. Adafactor would have saved me the afternoon.
Evaluation: why BLEU scores lie
We hit a BLEU-4 of 44.00 on Dongxiang, which reads well on a slide. Automatic metrics are only a proxy, though, and with a small corpus you also have to watch for drift detection. The model overfits to the sentence structures it was shown, so the score holds up right until a user writes something the corpus never contained.
If this kind of work is eating your billable hours, I can take it on. I have been building WordPress and backend integrations since the 4.x days.
What actually decides the outcome
Model size is not what makes a Dongxiang translation system work. A precise data pipeline and a stable fine-tuning run are. Meta’s NLLB-200 documentation is a reasonable starting point, but most of the real work sits in the requirements.txt and the preprocessing scripts. Ship something small and get native speakers to read the output before you trust a benchmark.
