If you run a multi-language WordPress site or a translation pipeline of any size, you have probably met the translation hallucinations problem. The model takes a clean French sentence and the output mentions a “wife” who appears nowhere in the source. The usual responses are throwing more compute at it or hoping it stays rare. Neither one tells you what the model was doing at the moment it went off.
The usual measure of uncertainty is output entropy: how spread out the model’s probabilities are across the candidate tokens. It gives you a number and nothing else. Wide distributions can mean the model is torn between two synonyms, which is harmless, or that it has stopped tracking the source at all. Attention misalignment is what separates those two cases.
What entropy does not tell you
The standard way to catch translation hallucinations is to read the probability distribution over each output token and treat high entropy as doubt. That holds up on easy sentences and gets fragile on the rest. Two acceptable synonyms produce a wide distribution and get flagged as an error, and even when the flag is right, the number still says nothing about what kind of doubt you are looking at.
Metrics like xCOMET score well, but xCOMET means fine-tuning 3.5 billion parameters, which is more machinery than most of us want sitting in a QA step. A glassbox alternative is a bidirectional cross-check: run the pair through the forward model (source to target) and the backward model (target to source), then compare where each one is looking. When they disagree about the alignment, you are probably looking at a hallucination.
If you want the background on how these models work underneath, I wrote a guide to neural machine translation for low-resource languages.
Measuring bidirectional attention misalignment
The mechanism needs two models rather than one. Once the forward model has produced a translation, you feed the pair into the backward model with teacher forcing, so it is not generating anything new. You are only asking whether its attention lands back on the source tokens the forward pass came from. When it cannot find that path, the reciprocal attention map comes out blurred instead of sharp.
def bbioon_get_bidirectional_attention(dual_model, src_tensor, tgt_tensor):
"""
Extract forward/backward cross-attention and calculate the reciprocal map.
This is where we detect if the model's 'eyes' are misaligned.
"""
dual_model.eval()
with torch.no_grad():
# Extract attention weights from both directions
fwd_attn, bwd_attn = dual_model.get_cross_attention(src_tensor, tgt_tensor)
# Align matrices for element-wise comparison
B, T = tgt_tensor.shape
S = src_tensor.shape[1]
fwd_aligned = torch.zeros(B, T, S, device=src_tensor.device)
bwd_aligned = torch.zeros(B, T, S, device=src_tensor.device)
if T > 1:
fwd_aligned[:, 1:T, :] = fwd_attn
if S > 1:
bwd_aligned[:, :, 1:S] = bwd_attn.transpose(1, 2)
# The magic happens here: element-wise multiplication
# High value = Agreement. Low value = Potential Hallucination.
reciprocal = fwd_aligned * bwd_aligned
return fwd_aligned, bwd_aligned, reciprocalWhich signals to pull out of the maps
The raw matrix is too big to feed anything directly, so you reduce it to a handful of properties that move when translation hallucinations show up:
- Focus. Sharp attention on one or two source positions is a good sign. Attention spread thinly across the whole sentence means the model is guessing.
- Reciprocity. If token A attends to token B, does B attend back to A? Spurious alignments fail that round trip.
- Sinks. A lost transformer tends to dump attention onto safe tokens like
SOSorPAD, so the mass sitting on those positions is worth tracking by itself.
I made a related argument about where these layers now sit in the stack in why AI is a WordPress fundamental layer.
Training a lightweight QE head
None of this requires retraining the NMT model. Freeze the main weights and train a small MLP classifier, the QE head, on the 75 extracted features. In my testing, combining the attention features with entropy raised ROC-AUC noticeably, and the gain was largest on linguistically distant pairs like Chinese to English.
In a production WordPress setup the practical shape is a small Python service you call over the REST API, either from a save_post hook or from a background queue worker.
<?php
/**
* Example: Triggering a Quality Estimation check from WordPress
* This is how you prevent bad translations from hitting your live site.
*/
function bbioon_verify_translation_quality( $target_text, $source_text ) {
$api_url = 'https://your-qe-service.local/v1/check';
$response = wp_remote_post( $api_url, [
'body' => json_encode([
'source' => $source_text,
'target' => $target_text,
]),
'headers' => [ 'Content-Type' => 'application/json' ],
]);
if ( is_wp_error( $response ) ) {
return false;
}
$data = json_decode( wp_remote_retrieve_body( $response ), true );
// If the 'BAD' probability is too high, flag it for human review
return ( $data['prob_bad'] < 0.4 );
}If this kind of work is eating your dev hours, I can take it on. I have been working with WordPress since the 4.x days.
What this buys you
Machine translation is ordinary infrastructure now, which is exactly why a broken sentence costs more than it used to. Swapping a single entropy number for attention misalignment gives you something you can inspect and argue with, and it flags the bad tokens before a reader sees them.
The original research on semantic entropy goes deeper into the theory, and the COMET framework is on GitHub if you want to compare approaches.
