I thought I had seen every way a database search can break, and then I picked up a sanctions screening project for a global financial client. The site worked fine in London. The moment someone typed “Владимир” into a system indexed on “Vladimir,” it returned nothing. No error, just an expensive silence. Cross-Script Name Retrieval tends to stay invisible until it takes down a production pipeline.
Where multilingual name matching fails silently
Most of us reach for classical fuzzy matching: Levenshtein distance, Soundex. Both fall apart across script boundaries. If two strings share no Unicode block, the edit distance is basically the length of the string. Phonetic codes like Double Metaphone are worse, since they assume Latin pronunciation and break on Arabic, Cyrillic, or Hindi.
Romanization is not fixed either. The Chinese character “张” maps to Zhang, Chang, or Cheung depending on regional convention. Build your architecture around one canonical Latin form and you are betting on where your users happen to come from. The way out is to stop comparing characters and start comparing bytes.
Why bytes beat heavyweight LLMs
The common move is to throw a large multilingual LLM at it. That works, and it also costs far more than the problem does. A 7B parameter model is overkill for comparing two-word names, and it adds latency you feel on every query. Compact transformer encoders trained on raw UTF-8 bytes do the same job.
A byte-level encoder treats every Unicode character as a sequence of 1 to 4 bytes drawn from a 256-symbol alphabet, so the vocabulary is universal by construction. Train it contrastively, teaching the model that “Владимир” and “Vladimir” belong near each other in vector space, and the script gap closes without a tokenizer or script detection. My post on Agentic RAG and hybrid search covers how vector similarity can still fail in harder cases.
Implementing cross-script name retrieval
In a real WordPress or WooCommerce deployment you would not run a 4-million parameter transformer inside the PHP process. That is how you get a 504 Gateway Timeout. Push embedding generation into a microservice and query a vector database like FAISS. The integration I usually write looks like this:
<?php
/**
* Naive Approach: The SQL mistake most devs make.
*/
function bbioon_bad_search($query) {
global $wpdb;
// This will NEVER find \"Владимир\" if the DB is in Latin \"Vladimir\"
return $wpdb->get_results($wpdb->prepare(
\"SELECT * FROM wp_users WHERE display_name LIKE %s\",
'%' . $wpdb->esc_like($query) . '%'
));
}
/**
* Better Approach: Using a Vector Search API for Cross-Script Name Retrieval.
*/
function bbioon_vector_name_search($query) {
$api_url = 'https://api.your-ml-service.com/v1/search';
$response = wp_remote_post($api_url, [
'body' => json_encode(['query' => $query, 'k' => 10]),
'headers' => ['Content-Type' => 'application/json'],
'timeout' => 2, // Keep it fast or fail gracefully
]);
if (is_wp_error($response)) {
return bbioon_bad_search($query); // Fallback to SQL
}
return json_decode(wp_remote_retrieve_body($response), true);
}
Scaling with ANCE hard negative mining
Training is what makes Cross-Script Name Retrieval work. Random in-batch negatives are too easy, and the model learns to tell “Catherine” from “Zhao Wei” almost immediately. The hard case is “Katarina” against “Katherine.” That is where ANCE (Approximate Nearest Neighbour Contrastive Estimation) comes in. Rebuild a FAISS index periodically during training and mine hard negatives, meaning the names the model currently thinks are similar and are not, and the embedding space gets much sharper.
Retrieval gets noticeably more precise after that. In our tests the gap between Latin and non-Latin queries shrank by 10x compared with classical baselines. For the scaling side of this, I wrote about vector search optimization and cost reduction.
Gotchas: CJK and synthetic data
Chinese and Korean stay difficult even with a byte-level model. Romanization ambiguity there is severe enough that one character can map to very different phonetics. The other risk is training entirely on LLM-generated data: if your data engine, say Llama 3, transliterates with a bias, your retrieval model inherits it. Check against ground-truth data from Wikidata or the documentation for Sentence Transformers.
If this cross-script name retrieval work is eating your dev hours, I can take it on. I have been working with WordPress since the 4.x days.
Where this leaves multilingual search
Language-specific tokenizers and large LLMs make multilingual search more complicated than it has to be. Byte-level models are faster, smaller, and handle disjoint scripts natively. If your site has to find “Владимир” as readily as “Vladimir,” a phonetic vector index will get you there and SQL LIKE will not.
